fix(cpu): re-gate the FMA and BFDOT paged-attention arms -- the default kernel is bit-exact again

.agents/issues/FIX-PAGED-REGRESSION/ISSUE-LOCAL-01M61X3Q5N51S2AGHB5N8SS3HH.md

@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M61X3Q5N51S2AGHB5N8SS3HH
+Title: CPU paged-attention regression: the kolibri1 kernel batch broke glm4-moe-lite goldens and the dtype-dispatch bit-identity contract
+Row: FIX-PAGED-REGRESSION
+State: OPEN
+Kind: bug
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-10-28
+Updated: 2026-10-28
+Closed: -
+
+## Problem
+
+test_glm4_moe_lite_paged_engine and test_ops_paged_attn_dtype fail on main since the kolibri1 CPU paged-attention batch that landed 2026-10-19 13:50Z (draft commits 2cd6b8a14 FMA-contract, 3c4925def BFDOT K-dot, 6be7f0214, fa373385e, c975549b0, d63fb81fc; the 61d333e52 bf16-dot default landed 10-19). CI bracket: run 39200285700 (04:36Z) had glm4 failing only; run 39278997419 (15:34Z) failed both. Today both still fail: glm4 69/128 exact positions vs a STRICT oracle licensing 128/128 at gap 0; ops_paged_attn_dtype loses bit-identity across dispatch arms. The FMA contraction changes summation order in the SHARED kernel — prime suspect; exact culprit owed via bisect. The draft commits landed with failing goldens; the repair restores the contracts, never widens the assertions. Spec: .agents/specs/fix-paged-regression.md
+
+## Resolution
+
+-

.agents/specs/fix-paged-regression.md

@@ -0,0 +1,93 @@
+# SPEC — `FIX-PAGED-REGRESSION` — repair the CPU paged-attention regression from the kolibri1 kernel batch
+
+- Owning row: `FIX-PAGED-REGRESSION`
+- Kind: bug (correctness regression on main)
+- Issue: created in the same flow (row-scoped local issue)
+
+## Problem
+
+`test_glm4_moe_lite_paged_engine` and `test_ops_paged_attn_dtype` fail on
+`main`. CI bracket:
+
+- run 39200285700 (2026-10-19T04:36Z): glm4 failing (1 failure)
+- run 39278997419 (2026-10-19T15:34Z): both failing (2 failures)
+
+Between those: the 13:50Z batch `2cd6b8a14` (FMA-contract the paged-attention
+scalar body), `3c4925def` (BFDOT K-dot arm), `6be7f0214`, `fa373385e`,
+`c975549b0`, `d63fb81fc` — several committed with `draft(...)` subjects.
+`61d333e52` (bf16-dot as the aarch64 default) carries an author date of
+17:39Z the same day. Several of these landed with the goldens already red.
+
+Failure evidence (run 40712283192, job 122900454192):
+
+- `test_ops_paged_attn_dtype.cpp:304`: the hoisted dtype dispatch is NOT
+  bit-identical — "decode c1 [q=f32 kv=f32 out=f32] first differing storage
+  byte at index 0: got 3 want 4", likewise f16/bf16 KV arms.
+- `test_glm4_moe_lite_paged_engine.cpp:226`: 69/128 exact positions against
+  a STRICT oracle that licenses 128/128 with teacher-forced gap 0 mnats;
+  1/8 prompts token-exact.
+
+Also on the same batch's authority: the FMA contraction changes summation
+order in the SHARED paged kernel — the leading suspect for the bit-identity
+break; the bf16-dot/BFDOT default flips are the other candidates. The exact
+culprit commit is owed to this row via local bisect.
+
+## Scope
+
+Restore the two contracts on the shared CPU paged-attention kernel:
+
+1. the hoisted dtype dispatch is bit-identical across arms
+   (`test_ops_paged_attn_dtype`);
+2. glm4-moe-lite's strict committed goldens hold
+   (`test_glm4_moe_lite_paged_engine`).
+
+The assertions are NOT widened. If akolibri1's perf arm genuinely cannot
+coexist with bit-identity on the shared kernel, the repair re-gates the
+offending arm behind its runtime knob (the `draft` commits' own gates) so
+the DEFAULT path is the pre-batch bit-exact one, and the perf arm stays
+opt-in until it can satisfy the contract — that is the smallest correct
+change, not a performance verdict on the arm itself.
+
+## Design
+
+- Local bisect over `git log origin/main -- src/vt/cpu/cpu_paged_attn.cpp`
+  between the bracket commits, running both failing tests per step (each
+  < 1 s once built; builds on the build zone, `taskset -c 64-127`, `-j 64`).
+- The culprit commit identifies the mechanism (FMA order vs dtype dispatch
+  vs bf16-dot default); the repair is scoped to that mechanism.
+- glm4-moe-lite's oracle is committed and strict: after the repair, 128/128
+  exact positions must reproduce on this box.
+
+## Red-first
+
+Both tests are red on `origin/main` today (CI run 40712283192 and locally
+reproducible). Green-after is both tests passing with the default path, and
+the bisect record naming the culprit commit and the mechanism.
+
+## Tests
+
+- `test_ops_paged_attn_dtype`, `test_glm4_moe_lite_paged_engine` green.
+- The kolibri1 CPU suites that own the perf arm stay green, and the perf
+  arm's own bit-exactness claims (if any) are re-measured under its gate.
+- Full `build-test-cpu` ctest: no NEW failures vs the pre-repair main run
+  (the unrelated `set_mempolicy` NUMA-class failures are a separate row and
+  may still fail; this row does not touch them).
+
+## Gates
+
+- Focused: the two suites. Full: `build-test-cpu` ctest.
+- `check-agent-record.py`, commit style, trailers, preflight (known host
+  dotnet failure named).
+
+## Stop conditions
+
+- If bisect shows the culprit predates the bracket (i.e. glm4 broke even
+  earlier than 10-19), widen the bracket and record the real one; do not
+  guess a fix for an unconfirmed mechanism.
+- If restoring bit-identity requires reverting another model's shipped
+  behavior (not a draft perf arm), return `NEEDS_DECISION` — that is a
+  material disagreement between contracts, adjudicated by the developer.
+
+## Git integration
+
+One pull request, `row/FIX-PAGED-REGRESSION`, spec committed first.

docs/ENVIRONMENT.md

@@ -315,8 +315,9 @@ portable/reference path. In normal operation leave them unset.
 | `VT_CPU_REF` | off | Set on to force the portable reference path (dequantize-everything oracle), the standard "is this a kernel bug?" bisect switch |
 | `VT_CPU_MATMUL_TIER` | `auto` | Force the elementwise CPU GEMM tier in the same adaptive binary: `portable`/`neon` on arm64, or `portable`/`sse2`/`sse2+f16c`/`avx2`/`avx512` on x86_64. An unsupported or cross-architecture tier fails closed after the exact OS capability probe |
 | `VT_CPU_Q8_DOT` | `auto` | Select the Arm Q8_0 x Q8_0 inner dot: `portable`, compiler-generated `sdot`, or the Linux-only `a76-asm`. `auto` selects the A76 schedule on Cortex-A76, otherwise SDOT on any Linux/Darwin DotProd-capable host, otherwise portable. A forced unsupported tier fails closed |
-| `VT_CPU_KOLIBRI_BF16_DOT` | **on** (aarch64 with FEAT_BF16) | The kolibri1 bf16-dot CPU compute class is the DEFAULT wherever the HWCAP2 FEAT_BF16 probe (`DetectArmBf16`) proves the hardware: the routed-expert/attention decode GEMV over the warm-cache bf16 weights (`vbfdotq_f32`, bf16 in / f32 accumulate), the paged-attention decode K dot, and the lm_head / router-gate decode GEMV (`src/vllm/model_executor/models/kolibri1_bfdot_gemv.cpp`, `src/vt/cpu/cpu_paged_attn_bfdot_arm.cpp`). This mirrors vLLM, which dispatches CPU numerics kernels on feature probes, not env knobs. The f32 tier it replaces stays the only tier on x86 and on non-bf16 aarch64, so those hosts never move. `=0` is the single escape hatch: it forces the f32 tier for ALL of these arms in the same binary (the A/B leg, bit-identical to the pre-adoption production stream). The paged-attention engagement rides the NEON lane body (K dot on the BFDOT arm, V accumulation on the lane's widen+vfma path), so the lane's load-contention caveat (`VT_CPU_PAGED_ATTN_NEON` row) applies to the decode dot too. The routed-expert GEMV additionally needs the dequant cache (`VT_KOLIBRI1_DEQUANT_CACHE_MB`) for its warm-bf16 residency precondition. The legacy per-arm opt-ins `VT_CPU_KOLIBRI_BFDOT_GEMV=1` and `VT_CPU_PAGED_ATTN_BFDOT=1` keep working for one release and then go away (SPEC `.agents/specs/kolibri1-bf16-dot-cpu.md`; #3438) |
+| `VT_CPU_KOLIBRI_BF16_DOT` | **on** (aarch64 with FEAT_BF16) | The kolibri1 bf16-dot CPU compute class is the DEFAULT wherever the HWCAP2 FEAT_BF16 probe (`DetectArmBf16`) proves the hardware: the routed-expert/attention decode GEMV over the warm-cache bf16 weights (`vbfdotq_f32`, bf16 in / f32 accumulate) and the lm_head / router-gate decode GEMV (`src/vllm/model_executor/models/kolibri1_bfdot_gemv.cpp`). The PAGED-ATTENTION decode K dot is NOT part of this default any more: row FIX-PAGED-REGRESSION restored it to the explicit `VT_CPU_PAGED_ATTN_BFDOT=1` opt-in (2026-10-28), because the pair-dot reduction DAG is not bit-exact with the pre-batch order the shared paged kernel's dtype contract and the glm4 goldens license (docs/bench-evidence/fix-paged-regression-20261028/). This mirrors vLLM, which dispatches CPU numerics kernels on feature probes, not env knobs. The f32 tier it replaces stays the only tier on x86 and on non-bf16 aarch64, so those hosts never move. `=0` is the single escape hatch: it forces the f32 tier for the kolibri1 GEMV/forward arms in the same binary (the A/B leg, bit-identical to the pre-adoption production stream). The paged-attention engagement rides the NEON lane body (K dot on the BFDOT arm when opted in, V accumulation on the lane's widen+vfma path), so the lane's load-contention caveat (`VT_CPU_PAGED_ATTN_NEON` row) applies to the decode dot too. The routed-expert GEMV additionally needs the dequant cache (`VT_KOLIBRI1_DEQUANT_CACHE_MB`) for its warm-bf16 residency precondition. The legacy per-arm opt-in `VT_CPU_KOLIBRI_BFDOT_GEMV=1` keeps working for one release and then goes away; `VT_CPU_PAGED_ATTN_BFDOT=1` is the paged kernel's required opt-in, not a legacy spell (SPEC `.agents/specs/kolibri1-bf16-dot-cpu.md`; #3438) |
 | `VT_CPU_PAGED_ATTN_NEON` | off | Set `1` to run the aarch64 NEON lane for the CPU paged-attention K dot-product and V accumulation (`src/vt/cpu/cpu_paged_attn.cpp`). Not bit-identical to the scalar path (the K reduction runs four fused-multiply lanes): outputs agree to f32 noise plus one bf16 output ulp, and greedy chains can flip near-ties — the kolibri gates adjudicate them; the decode-bench anchor is byte-exact and has no near-tie instrument yet, which is why the default is off (docs/bench-evidence/kolibri1-neon-paged-attn-20261008.md). Measured: attn_core halves (4.9→2.3 s at t32 over the 64-step decode bench). Widths that are not multiples of 4 in head or v-head dim, and every non-aarch64 build, use the scalar path regardless |
+| `VT_CPU_PAGED_ATTN_SCALAR_FMA` | off | Set `1` to run the draft FMA-contracted scalar body of the CPU paged attention (4-lane pairwise K dot, fused V accumulate) instead of the default pre-batch sequential order (`src/vt/cpu/cpu_paged_attn.cpp`). NOT bit-identical to the default — the summation order changes — so it is a measurement arm only: the local bisect for row FIX-PAGED-REGRESSION (docs/bench-evidence/fix-paged-regression-20261028/) named the 2cd6b8a14 contraction as the first bad commit behind the dtype bit-identity and glm4 goldens regressions, and the default was restored to the pre-batch order. The PERF-CPU-ATTN-NEON sweep and the kolibri1 W3 gate pin `=1` for the cases that measure this arm |
 | `VT_CPU_QUANT_MMLA` | `auto` | Select the Arm i8mm quant-dot tier. `portable` (also `0`/`off`/`false`) disables it; `i8mm` (also `1`/`on`/`true`) forces it and fails closed unless Linux `HWCAP2_I8MM` or Darwin `hw.optional.arm.FEAT_I8MM` is present |
 | `VT_CPU_QUANT_REPACK` | `auto` | Select the Arm i8mm+DotProd Q8_0 repack tier. `portable` (also `0`/`off`/`false`) disables it; `i8mm` (also `1`/`on`/`true`) forces it and fails closed unless the exact Linux HWCAP/HWCAP2 or Darwin sysctl capabilities are present |
 | `VT_DFLASH_CTX_MAX_TOKENS` | unset (the resolved cap is `max_model_len + 1 + k`, itself capped at an 8 GiB budget shared across `--max-num-seqs`, so `8 GiB / max_num_seqs` per request) | Caps the DFlash/DFlash2/DSpark draft's CONTEXT STORE in tokens, PER REQUEST. The store is a per-request, per-draft-layer bf16 pool holding K and V, so its size is `tokens x num_draft_layers x num_kv_heads x head_dim x 2 x 2` bytes per concurrent request, and the runner builds ONE PER BATCH ROW — so what the device holds is that number times `--max-num-seqs`, and `gpu_memory_utilization` accounts NONE of it. The default budget is therefore the AGGREGATE rather than a per-request allowance: a 262144-token `--max-model-len` would ask for about a gigabyte per request, and unbounded device residency has OOM-rebooted a box here ([#1647](https://github.com/mudler/vllm.cpp/issues/1647)). Set this BELOW the advertised context to bound that memory, or ABOVE the default share to speculate further into a long context — the aggregate arithmetic (`tokens x bytes_per_token x max_num_seqs`) is then yours. The effective limit AND the aggregate it costs are announced at startup either way. A request whose context passes the limit keeps running WITHOUT speculation, on the target alone — never refused, and never fatal: before [#1919](https://github.com/mudler/vllm.cpp/issues/1919) the store was a fixed 4096 slots unrelated to `--max-model-len`, and the first prompt above it threw from inside an EngineCore step and left the server answering `[request submitted to a stopped AsyncLLM]` for every later request. Mirrors upstream, whose draft context lives in the engine's own paged KV cache sized `cdiv(max_model_len, block_size)` and whose speculators emit an EMPTY draft where they cannot serve a request |

docs/bench-evidence/fix-paged-regression-20261028/DECISIONS.md

@@ -0,0 +1,75 @@
+# FIX-PAGED-REGRESSION decision record — developer adjudicated 2026-10-28
+
+The bisect (README-bisect.md in this directory) found TWO independent
+culprits behind the post-2026-10-19 regressions, plus one deliberate red.
+Three decisions went to the developer; all three came back APPROVED on
+2026-10-28.
+
+## Decision 1 — APPROVED: re-gate the paged-kernel BFDOT default to opt-in
+
+`VT_CPU_PAGED_ATTN_BFDOT=1` is again the only way to reach the shared paged
+kernel's bf16-dot K reduction; the default path is the pre-batch bit-exact
+one. kolibri1's own GEMV/forward bf16-dot arms keep the bf16-dot ruling and
+default untouched (their `VT_CPU_KOLIBRI_BF16_DOT` semantics are unchanged).
+
+The default-dispatch assertion in `tests/vt/test_ops_paged_attn_neon.cpp`
+(PERF-CPU-ATTN-BFDOT) is re-pinned to the RESTORED contract — a tightening:
+the default must be OFF and only the explicit opt-in may reach the arm. The
+assertion text and comment state the restored contract and cite this row.
+
+Landed in the "re-gate the paged-kernel BFDOT arm" commit.
+
+## Decision 2 — APPROVED: pin VT_CPU_PAGED_ATTN_SCALAR_FMA=1 in the two gates that measure the FMA arm
+
+Inside `test_ops_paged_attn_neon.cpp` (the PERF-CPU-ATTN-NEON x4 sweep) and
+`test_kolibri1_w3.cpp` (the golden-run gate), the cases that measure the
+FMA-contracted body export `VT_CPU_PAGED_ATTN_SCALAR_FMA=1` for their
+duration, with a one-line comment naming the row and why: they measure the
+opt-in arm, not the default. The default stays the bit-exact pre-batch order.
+
+Landed in the "re-gate the FMA-contracted scalar body" commit.
+
+## Decision 3 — APPROVED: glm4 69/128 is a PRE-EXISTING DELIBERATE RED, owned by issue #2839
+
+`test_glm4_moe_lite_paged_engine` measures 69/128 strict positions because
+commit `fd8eae928f` (2026-09-27, the merged pull request that landed the
+gate, [#2906](https://github.com/mudler/vllm.cpp/issues/2906))
+DELIBERATELY landed the strict bar red —
+its own commit message says "THIS PULL REQUEST MAKES A GATE FAIL, AND THAT
+IS THE RESULT". The 69/128 predates the 2026-10-19 kolibri1 batch by three
+weeks and is NOT a regression from this row. Out of this row's scope; the
+glm4 test is untouched. Dated note recorded here 2026-10-28.
+
+The canonical tree record for the deliberate red is commit `de4c0ba03`,
+owned by issue
+[#2839](https://github.com/mudler/vllm.cpp/issues/2839) (row
+MODEL-TEXT-GLM4-MOE-LITE-GATE-2839). #2906 alone is incomplete as a cite:
+it is the merged PR that landed the gate, while #2839 / `de4c0ba03` carry
+the deliberate-red ruling itself.
+
+## Consequences re-measured under the restored default (2026-10-28, this host)
+
+The restored default stream is the historical pre-batch 101807/109726
+alternate-near-tie shape. The default-tier stream anchors in
+`test_kolibri1_decode_bench`, `test_kolibri1_spec_probe`, and
+`test_kolibri1_ngram_spec` moved 11622 -> 101807; the spec probe's
+bench-prompt exactness gate is tier-scoped (one knife-edge flip at position
+5, the same plain-t1-vs-batched mechanism the golden case documents) with
+the byte-identity floor re-measured 5/8; the n-gram gate's bench-prompt
+draft counts moved to 59 proposed / 53 accepted and the golden band to
+164/290. Each re-pin cites the row in its comment.
+
+## Final gate table (DEFAULT environment unless noted)
+
+| Suite | Result |
+|---|---|
+| test_ops_paged_attn_dtype | 172/172 assertions green |
+| test_ops_paged_attn_neon | green (15,200,901 assertions) |
+| test_kolibri1_w3 | green |
+| test_kolibri1_bfdot_gemv | green |
+| test_kolibri1_moe_glue | green |
+| test_kolibri1_ngram_spec | green |
+| test_kolibri1_spec_probe | green |
+| test_kolibri1_decode_bench | green (2/2 identical chains) |
+| test_kolibri1_threeway_capture | green |
+| test_glm4_moe_lite_paged_engine | RED 69/128 — deliberate #2839 bar, out of scope |