Repository navigation
feat(MODEL-TEXT-FLINT): the paged-engine arm — continuous batching on CPU, five defects fixed, the first throughput numbers - #3513
Merged
Conversation
lu-zero
added a commit
that referenced
this pull request
Oct 28, 2026
The shared runner's free disk drops under the strix worker's 20 GiB
min_disk_bytes floor mid-job, so the tools-suite step reds intermittently
on pressure that has nothing to do with the tree: 49
tests.tools.test_strix_vllm_oracle failures on ValueError('disk headroom
exhausted') at worker.py:257, plus CacheDropError where the cache-drop
proof cannot evict 6291456 resident bytes. Main carries the same
intermittent failure (runs 40521291883, 40452744901) and the #3513 rerun
reproduced it (job 122404172875).
Reclaim the image's largest unused tenants -- Android SDK, .NET, GHC,
JVM, stale docker layers -- in a best-effort step right after checkout,
and log df -h / so the run carries the evidence. No workflow in this
repository uses any reclaimed tree. The worker's 20 GiB floor is
unchanged: the fix removes the pressure, not the check.
Red-before is main itself: two of its last five agent-record runs failed
the same way without this branch. Green-after is this pull request's
agent-record job passing with the df line in its log.
Fixes ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
The Flint CPU arm serves only through the single-sequence reference forward today; this spec scopes the paged-engine arm, its gates, and the first CPU throughput measurement. The shared paged infra already expresses per-head sinks and the hybrid window split, verified at base. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM [maki]
… CPU, gated against the reference forward (W-a/W-b) The row's spec (.agents/specs/flint-paged.md) committed at 9cda5a6 carried the plan; this lands waves W-a and W-b. W-a: MakeFlintKVCache already emitted the hybrid full/sliding-window KV groups from layer_types (the GGUF loader builds them from flint.attention.sliding_window_pattern), so W-a is the focused gate: the group membership, the configured window, and the dims, asserted in test_flint_paged.cpp. W-b: FlintForwardPaged — the same architecture as the reference forward expressed through the shared paged seams (dense_attn StepInputs, vt::PagedAttention, per-layer PagedKvCache), reached through ForwardFlintForCausalLM when attn_kv is non-empty; multi-request batching, per-head sinks on every layer, the sliding window on the sliding layers only, and the sigmoid top-k renormalised MoE with the ungated shared expert. The reference forward stays byte-identical and remains the oracle. The gate caught three real defects, each fixed here: - THE SHARED KERNEL (new closed issue ISSUE-LOCAL-01M5YMB0XZ8723N0688YHMKP3N): cpu_paged_attn.cpp seeded the softmax denominator at exp(0) with the running max seeded at the sink, and the two-pass form never rescales the seed once a score exceeds the sink — the sink term silently became exp(m-m) instead of exp(sink-m). MiMoV2's large learned sinks always stay the max, so every earlier gate passed; Flint's small sinks exposed it (L0 attention output -0.128 vs hand-computed -1.139 over identical bf16 streams, red-first in this gate). Both lanes now seed exp(sink - m_final), bit-identical when the sink IS the max. - The arm passed a LIVE residual buffer to the fused vt::RmsNorm(residual=...) on both norms, so the second call added the residual again (residual += x) and every MLP input was rmsnorm(2x). The reference keeps one residual stream, so the arm now runs plain norms and explicit adds. - The per-head QK RMSNorm ran in place over the head view and corrupted the normed stream (the dead agent's own finding, kept). The arm also mirrors the reference's rounding points exactly where the stage dump can see them: projections round to bf16 before the head norm (MatmulBf16's own final store), one bf16 rounding after the f32 head norm, and the same bf16 g/u into the silu-mul. With the three defects fixed, L0 matches the reference control exactly and every later layer sits at bf16-rounding distance; the gate gates the house 2.5-nat near-tie band (request 0 replays the reference chain token-for-token; request 1's one divergence is a near-tie flip inside the band). Gates: test_flint_paged green (4 cases / 208 assertions); test_flint_forward, test_flint_gguf, test_ops_paged_attn{_neon}, test_paged_attn_route, test_mimov2_* green; check-agent-record OK. test_ops_paged_attn_dtype fails IDENTICALLY with and without the kernel change on this host (pre-existing, 2 of 3 cases; the kernel diff touches only the two sink-seed lines) — left as found. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…4_K_M serves continuous batches through the full engine (W-c) test_flint_paged_engine drives the published flint-Q4_K_M.gguf through LoadedEngine::FromModelDir -> InputProcessor -> Scheduler -> the W-b paged forward arm -> Sampler -> OutputProcessor with THREE requests arriving CONTINUOUSLY (request 0 decodes alone, request 1 joins mid-flight, request 2 later), and adjudicates each engine chain against the reference forward's greedy chains by the kolibri1 contract: a chain either replays the reference token-for-token or its first divergence must sit inside the house 2.5-nat near-tie band, measured as the reference's own top-2 margin at that step. Result: request 0 replays " Paris is the " and every chain's first divergence is a true near-tie — measured margins 0.114 / 0.036 / 0.042 nats, two orders of magnitude inside the band. Checkpoint-gated: skips loudly without the container. Getting there surfaced TWO engine-side defects, each filed and fixed in the same flow: - ISSUE-LOCAL-01M5YRQ1SWC93MDG52QZGDFBY4: FlintHfConfigFromGguf never mapped flint.context_length onto max_position_embeddings (every other GGUF config builder does), so the engine resolved max_model_len 0, InputBatch sized its per-request token rows at zero and the first add_request SEGFAULTED. Fixed with the same OptInt(gguf, "<arch>.context_length", 0) line the siblings carry; the FromModelDir serve now auto-fits (4064 on the default pool). - ISSUE-LOCAL-01M5Z079R7RFSCPB1F4P9C94S5: the multi-group topology carries ONE CommonAttentionMetadata per step, whose slot_mapping is the TARGET group's — and the groups' pools allocate DIFFERENT block ids for the same positions (full group's first block 1, sliding group's 2, measured). The arm now derives each layer's slot mapping from that layer's OWN published group block table (slot = table[pos/bs]*bs + pos%bs); before the fix every sliding layer's attention output was exactly 0.0. MiMoV2 reads the CAM mapping for every layer the same way and is exposed identically; the engine-side single-CAM seam is recorded on the issue as owed to the engine row. The arm also now declares consumes_multi_kv and resolves per-layer caches and per-group block tables by name through the MultiKvCacheIndex — the same seam MiMoV2 and DeepSeek-V4 consume — so the engine's multi-cache guard (KV-DSV4-MULTICACHE W3) admits it. Gates: test_flint_paged_engine PASS (chains adjudicated inside the band); test_flint_paged PASS (4/4, 208 assertions); vllm-bench serves the container end-to-end; check-agent-record OK. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
serving docs (W-d/W-e) W-d: vllm-bench on the published flint-Q4_K_M container through the full paged-engine stack, on the shared aarch64 box at 32 threads (taskset -c 0-31, nice -n 5, idle), commit 92a2173: prefill 7.44 tok/s (1x512 in), decode 1.26 tok/s (TPOT 797 ms, 1x8 in, c=1), 0.97 tok/s aggregate at c=4 (8x128 in / 32 out). These are the FIRST CPU throughput numbers for Flint in this repository and a host-specific baseline, NOT a parity claim: vLLM registers no flint architecture, so no vLLM denominator exists, and the llama-cpp-flint oracle serves single-sequence only. The open gap and the next traceable lever (generic per-expert gather/SiLU/scatter vs a grouped-GEMM expert path; batched decode scaling worse than batch 1 points at memory-bandwidth contention there) are recorded rather than called a ceiling. - docs/bench-evidence/flint-paged-20261027/: the report plus the three raw harness reports (c1-decode, prefill512, c4-serving), one invocation per leg. - docs/benchmarks/flint-cpu-paged.md: the benchmark record; index row in docs/BENCHMARKS.md (check-benchmark-index OK). W-e: docs/USAGE.md gains the Flint serving section — the command, the container pins with the sha256, the three gates and what each adjudicates, and the refused arms named; docs/FEATURES.md's Flint row moves from "W1, not gated" to the gated record with links. The row spec records the wave outcomes and its ## Outcome (what was measured, what was rejected — token-exact paged-vs-reference agreement as the gate form — and why each default has its value); the row issue is closed with the full evidence trail. Gates: check-benchmark-index OK; check-agent-record OK. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/FLINT-PAGED
branch
from
October 28, 2026 11:10
c6f0692 to
54f47f0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(MODEL-TEXT-FLINT): the paged-engine arm — continuous batching on CPU, five defects fixed, the first throughput numbers
The CPU paged-engine row (spec .agents/specs/flint-paged.md committed before the implementation; the reference forward's token gate and the GGUF loader are on main). Flint now serves through the FULL paged LLMEngine on aarch64: KV allocation for the hybrid window pattern (per-layer sliding 513 / full from the file's own pattern KVs), the paged forward arm with the numerics contract mirroring the reference's rounding points, continuous arrival gated on the real Q4_K_M container, and the first Flint CPU throughput numbers.
FIVE DEFECTS found and fixed on the way, each gated:
GATES: the real-weights paged gate — 3 continuously-arriving requests through the full LLMEngine on Q4_K_M — PASS, first divergences at 0.114/0.036/0.042 nats (band 2.5), request 0 replays the reference text exactly; test_flint_paged 4/4; test_flint_forward 8/8; test_flint_gguf 3/3 (46/53, worst 0.735 nats); paged-attn and route suites green; test_ops_paged_attn_dtype fails identically on base (pre-existing, no attn_sink in the test, the kernel diff leaves the null-sink path unchanged).
THROUGHPUT (first Flint CPU numbers; host baseline, no vLLM denominator exists — recorded as such in docs/benchmarks/flint-cpu-paged.md): prefill 7.44 tok/s (512 in), decode 1.26 tok/s (TPOT 797 ms, c=1), 0.97 tok/s aggregate (c=4); 32 threads, idle box, sha256-verified container.
A fresh reviewer verified all six claims: the shared-kernel fix by numeric delta (12x logit-gap move under the reverted seed) with the MiMoV2 masking CONFIRMED; the double-RmsNorm by a 158x move; the slot-mapping by an engine-level RED mutation (gaps 10.8/11.6 nats); the real-weights gate re-run with matching margins; no regressions across the flint + paged-attn suites (793 s gguf leg included). Two test-strength minors (the focused gate's 2.5-nat band cannot see the 0.07/0.9-logit corruptions; defect 4's fix has no synthetic multi-group coverage — only the engine gate detects it) are recorded as follow-up debt, non-blocking.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]