Repository navigation
fix(cpu): re-gate the FMA and BFDOT paged-attention arms -- the default kernel is bit-exact again - #3518
Merged
Conversation
Two CPU paged-attention contracts broke on main with the kolibri1 kernel batch of 2026-10-19 13:50Z, several commits of which landed as drafts with the goldens already red: glm4-moe-lite's strict committed oracle (69/128 exact positions) and the dtype-dispatch bit-identity test. The FMA contraction changes summation order in the shared kernel and is the leading suspect; the exact culprit is owed to the row via local bisect. The repair restores the contracts or re-gates the offending arm behind its own runtime knob; the assertions are not widened. The unrelated set_mempolicy runner failures are a separate row and stay untouched. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…nd VT_CPU_PAGED_ATTN_SCALAR_FMA=1 The local bisect (docs/bench-evidence/fix-paged-regression-20261028/README-bisect.md) names 2cd6b8a "draft(kolibri1): FMA-contract the paged-attention scalar body" as the first bad commit for tests/vt/test_ops_paged_attn_dtype.cpp: its 4-lane pairwise dot and fused V accumulate changed the summation order of the scalar body, and that body is the bit-identity reference the hoisted dtype dispatch contract pins and the order the glm4-moe-lite strict goldens license. The pre-batch sequential order is the default again; an explicit VT_CPU_PAGED_ATTN_SCALAR_FMA=1 opts into the draft arm so the perf hunt keeps its lane. Two gates measure that arm, not the default, so they pin the knob for their duration with a comment naming this row: the PERF-CPU-ATTN-NEON x4 sweep (test_ops_paged_attn_neon.cpp, which requires the NEON arm bit-equal to the FMA-contracted body) and the kolibri1 W3 golden-run gate (test_kolibri1_w3.cpp, whose bands were measured under the contracted body). On this commit alone the q=bf16 kv=bf16 arms of test_ops_paged_attn_dtype stay red until the second culprit is repaired — the paged-kernel BFDOT default flip, reverted by the next commit. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…_BFDOT=1 — the default returns to the pre-batch path The second bisect mechanism (docs/bench-evidence/fix-paged-regression-20261028/README-bisect.md): the kolibri1-bf16-dot-cpu default flip (3c4925d arm + default dispatch, ruled by b42c8f7) made PagedAttnBfDotUsable true on every FEAT_BF16 host, which silently moved the shared paged kernel's bf16 K-dot onto the pair-dot reduction DAG — not bit-exact with the pre-batch order, it kept the q=bf16 kv=bf16 arms of test_ops_paged_attn_dtype red until an escape hatch was set. The default is the pre-batch path again; VT_CPU_PAGED_ATTN_BFDOT=1 is the explicit opt-in. kolibri1's own GEMV/forward bf16-dot arms (VT_CPU_KOLIBRI_BF16_DOT) keep their ruling and default untouched — this re-gate is the paged kernel's own knob only. Re-pins under the restored contract, each with a comment citing the row: the PERF-CPU-ATTN-BFDOT default-dispatch assertion (test_ops_paged_attn_neon.cpp) now tightens — the default must be OFF and the opt-in must reach the arm — and the BFDOT-vs-NEON case reaches the arm through the opt-in, not a default flip. The default-tier stream anchors move 11622 -> 101807 in test_kolibri1_decode_bench, test_kolibri1_spec_probe, and test_kolibri1_ngram_spec (re-measured 2026-10-28 on this host; the restored default is the historical pre-batch 101807/109726 alternate-near-tie shape). The spec probe's bench-prompt exactness gate is tier-scoped like the golden case beside it — under the restored default the plain t==1 forward and the batched verify rows are different numerics of the same pre-batch kernel, and the bench prompt flips once at position 5; the byte-identity floor is re-measured 5/8. The n-gram gate's bench-prompt draft counts move to 59 proposed / 53 accepted and the golden band to 164/290. Verified on this aarch64 host, DEFAULT environment: test_ops_paged_attn_dtype 172/172; test_ops_paged_attn_neon, test_kolibri1_w3, test_kolibri1_bfdot_gemv, test_kolibri1_moe_glue, test_kolibri1_ngram_spec, test_kolibri1_decode_bench (2/2 identical chains), test_kolibri1_spec_probe green; test_glm4_moe_lite_paged_engine stays red — the deliberate localai-org#2906 bar, not a regression from this row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
… two culprits, one deliberate red Copies the local bisect record (both brackets, the runner recipe, the two first-bad-commit verdicts: 2cd6b8a for the FMA-contracted scalar body, the 3c4925d/b42c8f787 BFDOT default flip for the q=bf16 kv=bf16 arms) together with the per-test gate logs captured after the two re-gate commits, and the decision record for the three developer adjudications of 2026-10-28: the BFDOT default re-gate to VT_CPU_PAGED_ATTN_BFDOT=1, the VT_CPU_PAGED_ATTN_SCALAR_FMA=1 pins in the two gates that measure the FMA arm, and the glm4 69/128 classification — a pre-existing deliberate red landed by fd8eae9 on 2026-09-27 ("THIS PULL REQUEST MAKES A GATE FAIL, AND THAT IS THE RESULT"), owned by issue localai-org#2906 and out of this row's scope. The same change documents the two runtime knobs in docs/ENVIRONMENT.md, which the env-doc checker owes for any env var src/ reads: VT_CPU_PAGED_ATTN_SCALAR_FMA gets its own row (draft arm, off, not bit-identical to the pre-batch default), and the VT_CPU_KOLIBRI_BF16_DOT row stops claiming the paged-attention K dot for the bf16-dot default — it names the restored VT_CPU_PAGED_ATTN_BFDOT=1 opt-in and cites this row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
Three corrections from the fresh review of FIX-PAGED-REGRESSION (2026-10-29), all evidence-side, no product code: - DECISIONS.md Decision 3: the glm4 69/128 deliberate-red cite now names issue localai-org#2839 (row MODEL-TEXT-GLM4-MOE-LITE-GATE-2839) with commit de4c0ba as the canonical tree record, alongside localai-org#2906 — the merged pull request that landed the gate (fd8eae9). localai-org#2906 alone was an incomplete cite: it carries the landing, not the deliberate-red ruling. The final gate table's cite moves to localai-org#2839 with it. - Capture the raw measurement logs the decision record asserted without carrying: final-spec_probe-prompt-raw.log (DEFAULT env, taskset -c 32-39, row-HEAD binaries from /tmp/build-review-pagedreg — the position-5 knife-edge flip and the 5/8 byte-identity floor) and final-decode_bench-anchor-raw.log (the 101807/109726 tier-scoped anchor chain). README.md in the same directory lists both and states they back DECISIONS.md. - docs/ENVIRONMENT.md: drop src/vt/cpu/cpu_paged_attn_bfdot_arm.cpp from the VT_CPU_KOLIBRI_BF16_DOT GEMV list — the file is now the paged opt-in's arm only, reached solely through VT_CPU_PAGED_ATTN_BFDOT=1, not a kolibri1 GEMV/forward default arm. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
main's CPU paged attention has been numerically red since the kolibri1
kernel batch of 2026-10-19 -- several commits of which landed as drafts
with the goldens already failing. Two contracts broke on the SHARED
kernel:
BIT-IDENTICAL across arms). Local bisect names two culprits:
2cd6b8a (FMA-contract the scalar body -- the K-dot changed from
sequential accumulation to a 4-lane pairwise FMA combine, and the V
pass to fma, changing summation order) and the BFDOT default flip
(3c4925d, ruled by the kolibri1-bf16-dot-cpu spec) -- the q/kv
bf16 arms diverge ~1 bf16 ULP on FEAT_BF16 hosts because the paged
kernel defaulted to vbfdotq.
regression: a local pathspec bisect lands on fd8eae9
(2026-09-27), the deliberate strict-gate landing whose own message
says the red is the result. The exit is owned by issue Glm4MoeLite's only vs-vLLM assertion cannot fail: the gap artifact is identically zero and the goldens match 69/128 #2839 (row
MODEL-TEXT-GLM4-MOE-LITE-GATE-2839) per the canonical record
de4c0ba; recorded here so the reader does not mistake it for
batch damage. Out of scope; the test is untouched.
The repair re-gates both offending mechanisms behind their own runtime
knobs -- the DEFAULT path is the pre-batch bit-exact kernel:
sweep and test_kolibri1_w3 pin it; they measure the opt-in arm).
K-dot. kolibri1's own GEMV/forward bf16-dot ruling is untouched --
only the shared paged kernel reverts.
(decode_bench 11622->101807, spec_probe tier-scoped with a knife-edge
flip at position 5 and floor 5/8, ngram 59/53 band 164/290); each was
re-measured 2026-10-28 under the restored contract and re-pinned with
row-citing comments. M4 proved the pins discriminate: under the old
default they measure exactly the old values. The raw measurement logs
are in the evidence dir -- including a discarded stale-binary capture
whose decode_bench reproduced the pre-fix 11622 until re-run against
row-HEAD binaries.
Mutations (fresh review, MERGEABLE): FMA default-on RED 88/172;
BFDOT default-on RED on the bf16 arms + the default-off assertion;
knobs deleted RED; old-default anchor probe RED exactly; ngram band
+1 GREEN (band gate -- the exactness/band taxonomy for these suites is
recorded in the verdict). ENVIRONMENT.md gains the
VT_CPU_PAGED_ATTN_SCALAR_FMA row and loses the paged file from the
GEMV list (owed by the env-doc checker).
Known host failure check-windows-portability.py (dotnet) left named.
The set_mempolicy runner failures are a separate row and untouched.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]