Repository navigation
Add GQA Causal Attention Backward challenge (hard #163) - #390
claude[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 269 lines changed.
- ID 163 is unused, and the challenge is not a duplicate: #111 is single-head and unmasked, while #80, #153, #53 and #141 are forward-only.
- challenge.py defines the 5 required methods, has no init override, and sets name, atol, rtol, num_gpus and access_tier. The extra private helpers (_case, _rand, _rand_case) only build test data.
- The reference uses only standard torch ops, with shape and dtype asserts and no device checks. All generate_* allocations use self.device, and imports are limited to ctypes, math, typing, torch and the challenge base.
- Tests cover single token, zeros, MHA, GQA with distinct KV heads, MQA, odd group size, non-power-of-two tails, overflow-prone scores and a realistic size. Output buffers are prefilled with 123.0, and atol/rtol of 1e-4 is sensible for float32.
- The performance case (32/8 heads, seq 2048, head_dim 64) totals about 84MB, far under the memory limit.
- The starters are empty stubs with only the parameter comment, and the Mojo starter uses max.gpu imports and MutAnyOrigin. The HTML signature and the worked example match challenge.py; the lse values I spot-checked (0.7071, 0.0, 1.1079) are correct.
- No network, file or system access, no subprocess, eval or exec, no import-time side effects, and no embedded reviewer-directed text. The diff appears complete.
shxjames
left a comment
There was a problem hiding this comment.
Reviewed 2ce0ae7. Spec and reference agree; no correctness blocker.
Spec / reference. FA2-style backward: O and lse from a float32 causal GQA forward are inputs (solvers may recompute them), τ = 1/√head_dim, causal mask is j ≤ i (diagonal included), D_i = dO_i·O_i (= rowsum(P∘dP)), and query head h uses KV head h / G. dQ stays per query head; dK[g] and dV[g] sum the group's query heads. The only outputs are dQ, dK, dV. All six starters match the signature; JAX returns the three outputs.
An independent fp64 backward, a float64 autograd implementation (expanded K/V, plain forward), and the closed forms for tests 2 and 3 all match reference_impl. The HTML example (including O and lse) matches both at the printed 4 decimals.
Headroom (share of atol + rtol·max|ref| per output tensor). Official fp32 reference vs fp64 is ≤ 0.0071 on perf dK (≥140×). Independent fp32 and a flash-style recompute of O/lse stay at ≤ 0.0020 vs the reference on f10 and perf (≥500×). The alternative D = rowsum(P∘dP) stays at ≤ 0.011 on the sharp-softmax case and ≤ 0.0035 on perf (≥87×), so accumulation order / atomics / split-K should be comfortable.
Unlike #387, TF32 / FP16 / BF16 are all rejected at 1e-4: worst shares are TF32 53 / FP16 57 / BF16 568 on the sharp case, and 18 / 16 / 97 on perf. No need to tighten the tolerance.
Injected bugs. Missing mask, strict (j < i) mask, missing 1/√d scale, missing D, last- or first-head-only dK/dV, and overwriting instead of summing are all caught, usually by a large margin (the S=1, Q=K=0, all-zero, and G=1 cases miss some by construction). The wrong h % Hkv mapping is caught on the designed GQA cases (f6, f8, f10, perf). An unstabilized exp(s) softmax is caught only by the sharp scale-100 case, and only in fp32 (it overflows); every other case stays finite and close. That one test is doing that job, and it works.
dK / dV are not degenerate under the max-scaled grader: on perf, max|dK| = 4.72 and max|dV| = 9.81 (means ~0.08). Every test dict includes all three output buffers, pre-filled with 123; an unwritten buffer fails by 10^5×.
Perf. 32/8/2048/64: I/O 84 MB; materializing P for every query head would be 537 MB, still fine on a T4. Causal backward is ~43 GFLOP, theoretical T4 fp32 floor ~5 ms, compute-bound as claimed. I have not approved or triggered a T4 run; the author reports a fused CUDA solution already passed --action submit on a T4.
CI is green. I have not approved.
Summary
Adds GQA Causal Attention Backward (hard #163): the backward pass of causal grouped-query attention, written the FlashAttention-2 way. The forward pass's saved
Oand per-rowlseare inputs, and the solver writesdQ,dKanddV. Several query heads share each K/V head, sodK[g]anddV[g]must add up the contributions of every query head in the group. This is the attention backward that runs in every layer when training Llama-, Mistral- or Qwen-style models.Signature:
solve(Q, K, V, O, dO, lse, dQ, dK, dV, num_q_heads, num_kv_heads, seq_len, head_dim)What a solver learns
P = exp(s - lse)tile by tile from the saved log-sum-exp, so neither the S×S score matrix nor P is ever stored, and use the cheap softmax correctionD_i = dO_i · O_i(one dot product per row) instead ofrowsum(P ∘ dP), which needs a full pass over the row's keys.dKanddVare sums over queries, whiledQis a sum over keys. Parallelizing over K/V tiles keepsdKanddVin registers or shared memory, and leavesdQto atomics or a second pass.G = num_q_heads / num_kv_headsquery heads. The efficient design has one block own a (K/V head, key tile) pair and loop over all G query heads, so the sum happens in on-chip storage with no atomics and no G-times-larger buffer. It also loads each K/V tile once instead of G times.Novelty compared with the closest existing challenges
rowsum(P∘dP). There are no saved statistics, no head sharing and no causal structure. In Challenge/topk kmoe #163 the core difficulty is summing dK/dV over query heads that share a K/V head, plus causal tile skipping and reusinglseandO. Could we update mojo to a newer version. #111 has none of these. A correct Could we update mojo to a newer version. #111 kernel run once per query head would give wrong dK/dV (it would overwrite or race instead of summing).Naive approach and what an optimized version changes
num_q_heads-sized buffer that is then reduced, or summed with global atomics. Masked tiles are computed and then zeroed.D = rowsum(dO∘O). Then one block per (K/V head, key tile) keeps K_j, V_j, dK_j and dV_j on chip. It loops over the G query heads and over query tilesi ≥ jonly, recomputes P fromlse, and accumulates dK_j and dV_j in place. dQ goes out through atomics or a separate dQ-parallel pass. dK and dV are each written exactly once.How the benchmark exposes it
The performance case is
num_q_heads = 32, num_kv_heads = 8, seq_len = 2,048, head_dim = 64(the Llama 3.2 1B attention shape, G = 4), with float32 randn data.Correctness tests and the mistakes they catch
Every output buffer starts filled with
123.0, so a kernel that skips entries, or accumulates into the output without zeroing it first, fails. O and lse in each test are produced by a float32 causal GQA forward pass on that test's Q, K and V.j < i) mask, which gives dV = 0; a missing D term; keeping one head (answer isdV = dO[0] + dO[1],dQ = dK = 0)dV[j] = Σ_h Σ_{i≥j} dO[h,i]/(i+1),dQ = dK = 0. Catches a missing or off-by-one mask and an unsummed dVh % Hkvinstead ofh / G; a race or overwrite between headsexp(s)/Σexp(s)without subtracting lse or a running max gives inf/NaNI ran these bugs as mutated implementations on CPU against the tests. Each one fails at least one test:
h % Hkvhead mappingIndependent checks: the float32 reference matches a float64 PyTorch autograd implementation, which uses expanded K/V heads and a plain forward pass, on every functional case and on the performance case. The largest absolute error is ≤ 4.4e-6 with outputs up to about 10. Tests 2 and 3 were also checked against hand-derived closed forms.
Tolerance (
atol = rtol = 1e-4, same as #111): a second, independent float32 formulation (softmax recomputed from scratch,D = rowsum(P∘dP)) agrees with the reference to ≤ 3.7e-6. That leaves a margin of more than 25× for different accumulation orders and atomics. Every bug in the table above misses by ≥ 1e-2.Validation
--action runpassed and--action submitpassed all functional and performance tests. The solution file is not committed.O→out) in the test generator, and Black reformatted the file. I did not re-submit after this change, which is rename and formatting only; the CPU accuracy and bug checks were re-run afterwards and still pass.pre-commit run --all-filespasses, andscripts/validate_challenges.pyreports no errors.<p>, uses<h2>sections, has an SVG on a#222background, the example matchesgenerate_example_test, uses LaTeX matrices, and has the performance bullet. challenge.py has the 5 attributes, shape/dtype-only assertions, XLA-safe ops anddevice=self.device. All 6 starters have exactly one parameter comment (JAX adds the return comment), and the CUDA/Mojo comments use "device pointers" without the parenthetical. The performance test is about 70 MB of inputs, well inside 16 GB / 5.🤖 Generated with Claude Code