Skip to content

Add GQA Causal Attention Backward challenge (hard #163) - #390

Open
claude[bot] wants to merge 1 commit into
mainfrom
add-challenge-163-gqa-causal-attention-backward
Open

claude[bot] wants to merge 1 commit into
mainfrom
add-challenge-163-gqa-causal-attention-backward

Conversation

@claude

@claude claude Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds GQA Causal Attention Backward (hard #163): the backward pass of causal grouped-query attention, written the FlashAttention-2 way. The forward pass's saved O and per-row lse are inputs, and the solver writes dQ, dK and dV. Several query heads share each K/V head, so dK[g] and dV[g] must add up the contributions of every query head in the group. This is the attention backward that runs in every layer when training Llama-, Mistral- or Qwen-style models.

Signature: solve(Q, K, V, O, dO, lse, dQ, dK, dV, num_q_heads, num_kv_heads, seq_len, head_dim)

What a solver learns

  • FA2 backward structure: recompute P = exp(s - lse) tile by tile from the saved log-sum-exp, so neither the S×S score matrix nor P is ever stored, and use the cheap softmax correction D_i = dO_i · O_i (one dot product per row) instead of rowsum(P ∘ dP), which needs a full pass over the row's keys.
  • Splitting the work for a reduction with two outputs: dK and dV are sums over queries, while dQ is a sum over keys. Parallelizing over K/V tiles keeps dK and dV in registers or shared memory, and leaves dQ to atomics or a second pass.
  • Summing gradients across GQA heads without extra synchronization: a K/V tile gets gradient from all G = num_q_heads / num_kv_heads query heads. The efficient design has one block own a (K/V head, key tile) pair and loop over all G query heads, so the sum happens in on-chip storage with no atomics and no G-times-larger buffer. It also loads each K/V tile once instead of G times.
  • Causal tile skipping: a key tile only needs query tiles at or below the diagonal, which roughly halves the work.

Novelty compared with the closest existing challenges

Naive approach and what an optimized version changes

  • Naive (correct): for each query head, build the full S×S scores and P in global memory, then run separate kernels for dP, dS, dQ, dK and dV. dK and dV are computed per query head into a num_q_heads-sized buffer that is then reduced, or summed with global atomics. Masked tiles are computed and then zeroed.
  • Optimized (same results up to float32 rounding): first a small kernel computes D = rowsum(dO∘O). Then one block per (K/V head, key tile) keeps K_j, V_j, dK_j and dV_j on chip. It loops over the G query heads and over query tiles i ≥ j only, recomputes P from lse, and accumulates dK_j and dV_j in place. dQ goes out through atomics or a separate dQ-parallel pass. dK and dV are each written exactly once.

How the benchmark exposes it

The performance case is num_q_heads = 32, num_kv_heads = 8, seq_len = 2,048, head_dim = 64 (the Llama 3.2 1B attention shape, G = 4), with float32 randn data.

  • Memory traffic: the naive version writes and reads several 32 × 2048² float32 matrices: S/P, dP and dS, about 512 MB each and over 2 GB of traffic in total. Q, K, V, O and dO together are only about 70 MB. The tiled version never stores these matrices, which turns a bandwidth-bound pipeline into a compute-bound one.
  • Head-sharing reduction: with G = 4, the per-query-head approach writes 4× the dK/dV data and then reduces it, or sends 4-way-contended atomics to every dK/dV element. It also loads each K/V tile from global memory 4 times. The group-looping block loads K/V once and writes dK/dV once.
  • Causality: at seq_len = 2,048, skipping the masked upper-triangle tiles removes about half the FLOPs. That is around 43 GFLOP of the 86 GFLOP non-causal total.
  • Load balance: the first key tiles see the most query tiles, so solvers have to think about ordering the grid or scheduling work persistently.

Correctness tests and the mistakes they catch

Every output buffer starts filled with 123.0, so a kernel that skips entries, or accumulates into the output without zeroing it first, fails. O and lse in each test are produced by a float32 causal GQA forward pass on that test's Q, K and V.

# Case Catches
1 Example (Hq=2, Hkv=1, S=3, D=2) Shown in the HTML; also catches a missing mask, a missing scale, a missing D term, and keeping only one head's dK/dV
2 S=1, two query heads, one K/V head Strict (j < i) mask, which gives dV = 0; a missing D term; keeping one head (answer is dV = dO[0] + dO[1], dQ = dK = 0)
3 Q = K = 0 Checked against a closed form: dV[j] = Σ_h Σ_{i≥j} dO[h,i]/(i+1), dQ = dK = 0. Catches a missing or off-by-one mask and an unsummed dV
4 All zeros Exact zeros (lse = log(i+1))
5 MHA (G=1), mixed signs Mistakes in the softmax backward itself: the 1/√d scale on dQ and dK, the D term
6 Hq=8, Hkv=2 Head mapping with h % Hkv instead of h / G; a race or overwrite between heads
7 MQA (Hkv=1), S=100, D=50 Tails along key, query and head_dim tiles
8 G=3, S=255, D=34 Odd group size and tails in every dimension
9 Q, K = ±5, D=16: diagonal scores = 100 Computing exp(s)/Σexp(s) without subtracting lse or a running max gives inf/NaN
10 Hq=32, Hkv=8, S=1024, D=64 Realistic size at the performance-test head layout

I ran these bugs as mutated implementations on CPU against the tests. Each one fails at least one test:

Bug Failing tests
No mask 1, 3, 5–10
Strict mask 1–3, 5–10
No scale 1, 5–10
No D term 1, 2, 5–10
h % Hkv head mapping 6, 8, 10
Last-head-only dK/dV 1–3, 6–10
Unstabilized softmax 9

Independent checks: the float32 reference matches a float64 PyTorch autograd implementation, which uses expanded K/V heads and a plain forward pass, on every functional case and on the performance case. The largest absolute error is ≤ 4.4e-6 with outputs up to about 10. Tests 2 and 3 were also checked against hand-derived closed forms.

Tolerance (atol = rtol = 1e-4, same as #111): a second, independent float32 formulation (softmax recomputed from scratch, D = rowsum(P∘dP)) agrees with the reference to ≤ 3.7e-6. That leaves a margin of more than 25× for different accumulation orders and atomics. Every bug in the table above misses by ≥ 1e-2.

Validation

  • I wrote a CUDA solution on the FA2 pattern (delta kernel, then one block per (K/V head, key tile) looping over the group's query heads with dK/dV in shared memory, and atomics for dQ). On a T4, --action run passed and --action submit passed all functional and performance tests. The solution file is not committed.
  • After those runs, flake8 E741 forced me to rename a local variable (O → out) in the test generator, and Black reformatted the file. I did not re-submit after this change, which is rename and formatting only; the CPU accuracy and bug checks were re-run afterwards and still pass.
  • pre-commit run --all-files passes, and scripts/validate_challenges.py reports no errors.
  • I went through every item in the CLAUDE.md checklist: HTML starts with <p>, uses <h2> sections, has an SVG on a #222 background, the example matches generate_example_test, uses LaTeX matrices, and has the performance bullet. challenge.py has the 5 attributes, shape/dtype-only assertions, XLA-safe ops and device=self.device. All 6 starters have exactly one parameter comment (JAX adds the return comment), and the CUDA/Mojo comments use "device pointers" without the parenthetical. The performance test is about 70 MB of inputs, well inside 16 GB / 5.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 269 lines changed.

  • ID 163 is unused, and the challenge is not a duplicate: #111 is single-head and unmasked, while #80, #153, #53 and #141 are forward-only.
  • challenge.py defines the 5 required methods, has no init override, and sets name, atol, rtol, num_gpus and access_tier. The extra private helpers (_case, _rand, _rand_case) only build test data.
  • The reference uses only standard torch ops, with shape and dtype asserts and no device checks. All generate_* allocations use self.device, and imports are limited to ctypes, math, typing, torch and the challenge base.
  • Tests cover single token, zeros, MHA, GQA with distinct KV heads, MQA, odd group size, non-power-of-two tails, overflow-prone scores and a realistic size. Output buffers are prefilled with 123.0, and atol/rtol of 1e-4 is sensible for float32.
  • The performance case (32/8 heads, seq 2048, head_dim 64) totals about 84MB, far under the memory limit.
  • The starters are empty stubs with only the parameter comment, and the Mojo starter uses max.gpu imports and MutAnyOrigin. The HTML signature and the worked example match challenge.py; the lse values I spot-checked (0.7071, 0.0, 1.1079) are correct.
  • No network, file or system access, no subprocess, eval or exec, no import-time side effects, and no embedded reviewer-directed text. The diff appears complete.

@shxjames shxjames left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2ce0ae7. Spec and reference agree; no correctness blocker.

Spec / reference. FA2-style backward: O and lse from a float32 causal GQA forward are inputs (solvers may recompute them), τ = 1/√head_dim, causal mask is j ≤ i (diagonal included), D_i = dO_i·O_i (= rowsum(P∘dP)), and query head h uses KV head h / G. dQ stays per query head; dK[g] and dV[g] sum the group's query heads. The only outputs are dQ, dK, dV. All six starters match the signature; JAX returns the three outputs.

An independent fp64 backward, a float64 autograd implementation (expanded K/V, plain forward), and the closed forms for tests 2 and 3 all match reference_impl. The HTML example (including O and lse) matches both at the printed 4 decimals.

Headroom (share of atol + rtol·max|ref| per output tensor). Official fp32 reference vs fp64 is ≤ 0.0071 on perf dK (≥140×). Independent fp32 and a flash-style recompute of O/lse stay at ≤ 0.0020 vs the reference on f10 and perf (≥500×). The alternative D = rowsum(P∘dP) stays at ≤ 0.011 on the sharp-softmax case and ≤ 0.0035 on perf (≥87×), so accumulation order / atomics / split-K should be comfortable.

Unlike #387, TF32 / FP16 / BF16 are all rejected at 1e-4: worst shares are TF32 53 / FP16 57 / BF16 568 on the sharp case, and 18 / 16 / 97 on perf. No need to tighten the tolerance.

Injected bugs. Missing mask, strict (j < i) mask, missing 1/√d scale, missing D, last- or first-head-only dK/dV, and overwriting instead of summing are all caught, usually by a large margin (the S=1, Q=K=0, all-zero, and G=1 cases miss some by construction). The wrong h % Hkv mapping is caught on the designed GQA cases (f6, f8, f10, perf). An unstabilized exp(s) softmax is caught only by the sharp scale-100 case, and only in fp32 (it overflows); every other case stays finite and close. That one test is doing that job, and it works.

dK / dV are not degenerate under the max-scaled grader: on perf, max|dK| = 4.72 and max|dV| = 9.81 (means ~0.08). Every test dict includes all three output buffers, pre-filled with 123; an unwritten buffer fails by 10^5×.

Perf. 32/8/2048/64: I/O 84 MB; materializing P for every query head would be 537 MB, still fine on a T4. Causal backward is ~43 GFLOP, theoretical T4 fp32 floor ~5 ms, compute-bound as claimed. I have not approved or triggered a T4 run; the author reports a fused CUDA solution already passed --action submit on a T4.

CI is green. I have not approved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant