[Engine] Batch slots on Metal: --batch N rows decode together by catalinb13 · Pull Request #3783 · Niko1221/Strata

catalinb13 · 2026-10-23T00:29:40Z

Closes #3284. This repo's rules ask a feature's scope to be agreed in an issue first; there has been no comment there, so treat this as that proposal — the feature is opt-in, measured, and the whole record is in docs/BATCHING.md. Happy to discuss scope before merge.

Problem

On a Mac the engine answers one request at a time and every other client queues (docs/MAC.md: "--parallel (batch slots) is not ported"). One person plus their coding agents — a small office — leave the machine between requests at ~2 GB/s of weight reads when the bandwidth is several times that, while the same model on a PC gains multiples by decoding waiting requests together.

Result

--batch N on the Metal engine runs N conversations in one window. Measured on an M1 Max (64 GB, Qwen3.8-Flash-Next IQ3_XXS), 21–23 Oct 2026:

Run	Aggregate	Same prompts alone	Replies vs solo
2 slots, greedy, 128 tok	37.5 tok/s	23 / 25 tok/s	both byte-identical
2 slots, sampled (temp 0.7, seed 7), 64 tok	42.7 tok/s	23 / 25 tok/s	both byte-identical
4 slots, greedy, 128 tok	34.7 tok/s	23–32 tok/s	2 of 4 byte-identical; the other 2 answered their own question fluently and chose a different word at one near tie (tokens 88 and 13)

The interleave cases hold too: a 4888-token prompt reads in while slots decode (126 slot tokens during the read); a read that gives way at a chunk boundary (YIELDED 2 1024) goes on in its slot; a request stopped part-way in a slot hands back to the solo path and reproduces its solo reply; a next turn reads from its slot's parked state; a prompt chunk cancelled mid-read resumes from its checkpoint (1024 reused + 3866 read). With STRATA_BATCH_VERIFY=1 the engine byte-compares each slot region's GDN states, QSA cells, tails, dead and pooled rows and PLE history against the window that made them, and reported all match for every region; the per-step kernel captures show each row's folds up to the batched PLE block bit-exact against its solo run.

The default path is unchanged: without --batch the window builds and runs exactly the commands it did before, and every unit gate passes both ways.

Approach

The arena carves a region per slot — each reply's full state in the solo window's own layout (about 3.8 GB with these settings, so the batch count is a memory choice, not a speed choice; the engine prints the cost at start, and refuses with a clear line when the regions do not fit). Parking already kept this state per conversation, so --batch combines with --conversation-cache-mib: a prompt reads through the main window and is copied into its slot's region before its rows first run.
The batch window builds the solo one-token dispatch list once per row. Every kernel that touches state takes the row index and reads and writes only its own region, and every GEMV keeps the plain tile — no CUDA grouped tiles, no doorbell kernels, no graph capture. What a window saves per step is the fixed cost the rows share: the device-flag cut handshake (each segment gated by a ~90 us flag spin, not an event pair), the sampler round trip, the launch gaps.
One binding fix the captures found, worth naming: a batch row's GDN step binds the prepared activation at its own slice, and the per-16 sums it writes for the quantized-input GEMVs landed at that bound base — rows after the first wrote sums nobody read and past the buffer, into the arena's next allocation. The sums have their own binding now; the solo window and the first row address them bit-identically to before, and gdn_parity's new plane-row check pins it (it fails with the pre-fix address and passes after).
Where a batched row can differ from solo is measured, not guessed: a 315-probe per-step trajectory for every row shows the first difference is the fold leaving the batched PLE block — a few kernels batch the rows in one dispatch and round differently than the per-token solo sequence, at fractions of a percent, amplified by fold cancellations. (Which kernel exactly rounds differently is unconfirmed static analysis.)

Validation

Revisions and environment. The measurements were made on this fork's Mac line at 783a55fc (the batch commits 2d50ea42..783a55fc), M1 Max, 64 GB, macOS 27, Qwen3.8-Flash-Next IQ3_XXS. upstream/main was merged into the branch after the runs (merge commit 1ff3a385, main at 5c8dd322); the post-merge tree rebuilds on this machine, and the merge touches none of the batch code.

Passed:

metal/tests host gates on 783a55fc (M1 Max, macOS 27): gdn_parity (incl. the new plane-row check and its negative control), gdn_prompt_parity, window_cells_test, host_sampler_parity, runtime_hello, moe_mm_parity, gemv_parity, qsa_parity, buffer_free_test — all green.
tools/batch_interleave_test.py run twice — prompt reads interleaving with slot decoding, the yield at a chunk boundary, the stop-into-slot hand-back, the parked-state next turn, the mid-read checkpoint resume.
STRATA_BATCH_VERIFY=1 on the 2- and 4-slot runs — per-slot state byte-compare all match, per-step 315-probe trajectories printed for every row.
tools/token_chain.py / tools/prompt_chain.py on the 47 GB shard (30.5 tok/s, 147 tok/s; the 2-chunk wrap past maxBufferLength).
Solo rates with --batch absent match this branch's pre-batch measurements (23–33 tok/s) — the solo path did not regress.

Failed:

The harnesses' byte-identity gate — 2 of 4 slots at --batch 4, and three interleave slots across two runs. At every miss both continuations answer their own prompt and the window's logits differ from solo's by the PLE-block noise above, so argmax breaks the tie the other way (the interleave runs print both texts: a colon against a comma, one period, static int against int). docs/BATCHING.md's Metal section carries the same doctrine CUDA already ships: token identity of a batched row is luck, coherence and no cross-talk are the contract. With serialized admissions the flips repeat exactly (4 slots: 88 and 13 across independent runs); when prompt reads interleave with decoding, arrival timing changes what each row shares its dispatches with and which slot flips can move between runs.

Not run:

The CUDA, HIP and SYCL builds (this is a macOS host without an NVIDIA or AMD card). The shared src/core and src/program hunks sit inside #if defined(STRATA_GPU_METAL) and the non-Metal bodies are unchanged text, but I cannot certify a build I did not run — CI must.
Other Apple Silicon shapes (only this M1 Max: no M3/M4/M5, no Studio or Ultra), --batch-groups pipelining on Metal, layer split with batch, batch counts past 4 (the regions would not fit 64 GB with the model), multi-day serving under real user load, and vision towers with batch.

What is in this branch, honestly

The seven batch commits (2d50ea42..783a55fc) sit on this fork's Mac line, which upstream has not taken: conversation parking on macOS, the device-flag cut handshake, M1/M2 GEMV speedups, the purge/hash fix, and merges of the still-open #3301, #3527, #3615 and #3333 plus my open #3287 and #2645. That is why the commit list is longer than the feature; upstream/main is merged into the branch (1ff3a385), so the diff applies cleanly, and the batch changes are marked in the commit subjects. I am happy to rebase onto whichever pieces land first, or to split.

Limits

Opt-in and experimental on Macs. No layer split or pipelined groups in a batch window. Each extra slot costs a full per-conversation region in RAM (15.4 GB for four here). The gain is real on this box (up to ~1.7x at two slots) and falls as rows grow; nobody has measured more than four.

How to try it

tools/metal_batch_try.sh 2 2 128

It starts the engine with --batch, runs the same prompts alone and together, byte-compares every reply, and refuses to run while another big server holds the memory (see the script's header). For sampled rows add --keys "temperature=0.7 seed=7".

…ver the window's head logits) The Mac engine picked the window's argmax for every request, so temperature, top_p, top_k, min_p, seed, the penalties and logit_bias changed nothing. Now Verifier::run's Metal branch does what the CUDA path does after its window: a sampled request, one with penalties or with a logit_bias, draws row t from the window's logits (unified memory, Window::logits_row) with Philox(seed, pos0 + t) and its own penalty row; drafts are kept while the sampled token equals them. Greedy without penalties or bias keeps the window's argmax. metal/src/engine/host_sampler.cpp: logit_bias -> penalties -> top_k (one NEON-filtered pass, the kernels' order: score descending, ids ascending) -> top_p -> min_p -> temperature -> draw, the kernels' float/double arithmetic. 9-46 us per row on 248,320 logits (M5 Max, host_sampler_parity --bench), so no GPU kernel. Verifier::set_logit_bias returned before storing the bias on Metal (the Metal init leaves le_ at -1, read as an earlier stage); -1 now means the whole model there. The CUDA, HIP and SYCL inits set le_ to n_layers when it is -1, so the condition is unchanged for them after init. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…-serve: reproducible seeds, picks draw for draw against a reference sampler on the dumped logits, penalties, logit_bias) and metal/tests/engine/host_sampler_parity.cpp metal_sampling_test.py: a reference sampler written from src/kernels/cuda/sampler.cu (Philox 4x32-10 at (seed, position), logit_bias, penalties, top_k, top_p, min_p, temperature) picks from each one-token request's first-window logits (STRATA_DUMP_FIRST_LOGITS); every engine pick must be its pick. 13 checks; on 0.2.0 without the engine change 11 fail (every seed gives the greedy text). --bench: decode tok/s greedy vs sampled. host_sampler_parity: the Mac host sampler (host_sampler.h) against a serial transcription of the CUDA kernels (sampler_parity.cpp's mirror_select/mirror_pick), 592 picks on 248,320-logit rows with ties, -inf, NaN, bans and penalties; --bench times a row. CPU only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…host pick's cost, greedy vs sampled decode) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Read remaining entries from physical-RAM refusal logs and leave missing occupancy/cumulative eviction data unknown until an authoritative event. Hide unknown occupancy bars and cover log replay, HTTP metrics, and UI recovery without a native engine. Constraint: Keep inference and native cache policy unchanged Confidence: high Scope-risk: narrow

…ase pools

…s parked reads A parked conversation costs a slot whatever its length, so --conversation-cache-min-tokens N (WRITE) keeps short side requests from taking the slots a long conversation needs. The read side had no such control: every request searched every parked conversation, and a short request could restore a short parked copy it rewrites at once. WRITE,READ adds the read threshold. READ is the shortest complete prompt, cached history included, worth searching the parked conversations for; below it the engine keeps to the active session, its checkpoints and the batch slots and searches nothing. READ=0 is the old behaviour and a single value is WRITE,0, so existing configurations keep their meaning. A startup with READ=0 prints the line it printed before. The check runs before ConversationCache::best(), so a skipped search neither evicts nor restores anything. The unit test covers the parser, the boundary and the full-prompt counting (cached history plus the new message).

The prompt experts read the quantized weights inside every GEMM's K loop and measured 2.9 TFLOP/s on the M1 Max. mpp decodes each batch of experts to fp16 slabs once per chunk (mm_expand, previously unused by the engine), skips experts the chunk never routes to, and multiplies gate/up and down as two dynamic-K matmul2d runs at 6.2 TFLOP/s. Warm, interleaved with same-session v2 anchors: a 4096-token chunk 12243 -> 8818 ms (-28%), 4096-token prefill 225.9 -> 256.6 tok/s (+13.6%); a 356-token real prefill loses 7% (the decode pays whole experts), so it stays opt-in: STRATA_METAL_PROMPT_MOE=mpp. The default path is byte-identical (8-token outputs match the prior build). Parity: y vs exact float rel L2 4.9e-04 on IQ2_XS, IQ3_S and Q2_0.

…ecode copy-overlap on by default on Metal Measured on an M1 Max (64 GB, IQ3_XXS pack, ctx 262144/40960, gap/arms-round*.txt): overlap +24..28% decode (same tokens), mpp prompt MoE +5..8% pp2048 (169.9 -> 194.1, cold +45%), scan_v +1..2% (196.2-199.3); mpp and scan_v change the greedy continuation from the default path, so they stay opt-in. Every decode tile retune (moE gl 8/16, gemv big-class, NR=2, spec depths 4/6/8) measured flat-to-worse: the M5 tile picks hold on M1. STRATA_METAL_PROMPT_EXPERTS=v3 hits the GPU watchdog on M1 and must not be used.

…nd M1 profile env pp2048@262K 184 default / 197 with gap/m1-env.sh; tg128@40K 20.4 (was 15.4 before the copy-overlap default-on); decode holds 10.2-10.8 tok/s at 130K ctx; recall exact at 5/75/95% depth, missed at 25/50% within 24 tokens (IQ3_XXS, repeated filler).

…::bf16 and 262 regenerated link stubs

…Metal - fused_gr_persist_init/read_persist take fused_gr.cu's own non-CUDA semantics (v0.1.45's Verifier::init calls it every session; the abort stub killed serve startup: exit 86)

…head goes top 12 - the MTLSharedEvent pair costs 230-400 us per cut because the queue stops and re-arms around each of a window's 48 cuts (dispatch_lat seg rows, gap/results/1018-1434); cut_seq/cut_spin in units.metal + Context::run_graph_flag propagate in ~90 us with the queue never stopping: window T2 87.8 -> 56.0 ms, right on the busy + miss floor (~55.5), decode 19.5-20.3 -> 29.97-30.49 tok/s, answers byte-identical (11 arms incl. the default-vs-opt-out sandwich, gap/results/1018-1534 and 1018-1545, md5 956dbac3e10818130ee6a82d3327f688); STRATA_METAL_CUT=event opts back. Read-ahead depth 6 -> 12 for big-expert packs: misses 15.9 -> 8.6 per window at 0.8 ms/miss, +63 MB read per window overlapped, same tokens, STRATA_METAL_PREFETCH opts out. Gap harness: pinned 33 GB arena per arm (the dynamic vm.vm_page_filecache_min floor moved it between arms and faked drift), per-arm filecache/free-page log, progress output; dispatch_lat grows the seg/flag handshake and indirect-dispatch rows; stream stats fix

…ver - stop/restore target it by binary name, its argv snapshot and restore log get generic names (server-argv.txt, server-restore.log), GAP_NO_SERVER skips the dance; the per-arm argv captures are untracked evidence, removed from git

…or four reasons, all fixed (A/B 2026-10-19, M1 Max, 64 GB, IQ3_XXS: turns after a side request reuse 69-76% of their prompt and prefill in 6.5 s against 17.6-23.6 s without parking, 28-37% of the same-size cold read; park 37-39 ms holding 418-566 MB, restore with checkpoint 8-10 ms, decode unchanged 27-30 tok/s): sample_host_memory had no macOS branch so the RAM guard refused every park; the window checkpoint layout failed the CPU payload-size validation (skipped under STRATA_GPU_METAL); the session's CUDA-shaped K/V pools are purged at startup so park/restore repack the window's own cells through Window::kv_pools + window_cells - the [cell][head] to [page][head][in-page] shuffle counted pages with the cell index (4x overfill: layer 0 trashed the heap, layer 1 faulted, and the shuffle is not its own inverse, so save and restore run opposite directions); and --kv fp16 left st.kv_int8 false while the window MSL always stores int8, so the snapshot sizes did not describe the stored bytes. STRATA_PARK_DEBUG=1 logs every park copy with its capacity and runs a malloc_zone_check canary around the park; SIGTRAP joins the crash handler with a register dump because libmalloc's integrity traps died silently (exit -5) and cost two full probe runs. metal/tools/parking_check.py drives both arms and writes gap/results/<stamp>/parking_check.md; the ratio gate scores reused turns only (cold turn 1 prefills in full in both arms). docs/DETAILS.md carries the measured table. macOS watchdog: mem_sample and file-tier reads take task_info/host_statistics64 now instead of /proc. No CUDA/HIP path touched beyond #if-gated shared files

…d the same regeneration in stubs.cpp, so stubs_0145 keeps only f16_range_read / gemm_f16_range_read (opt-in HIP/Turing reads off on Metal that must answer false, not stop the engine; the trap-stub copies in stubs.cpp are cut with a note) and the fused_gr no-ops already live in missing.cpp; link clean

…4 tokens Two deaths, one root cause: on Metal the prompt's buffers live in the Metal arena, not in the host region the host code believes it is borrowing. - Prefill::relayout: a successful no-op that keeps the larger chunk count (the Metal buffers never shrink; a large chunk() is one run() per segment). Before: "lending the prompt path its slots failed: ... at most 0" on a ~1.6k-token prompt. - the lend lambda: the slot-marking loop is guarded out, so p.lent stays empty and the refill after the read is a no-op instead of writing to purged host memory (the Metal arena purges host allocations >= 32 MB; the Metal window reads no host residency table). Before: SIGABRT "a copy touches a device allocation that was given back" on the ~10.8k-token prompt. Every line is inside a STRATA_GPU_METAL guard: the CUDA/HIP/SYCL builds stay byte-identical by construction (they cannot be built on this Mac: a build per backend is needed before merge). Measured on an M1 Max 64 GB, 8 GB expert arena: prompt 4643 tokens (85.3 tok/s) and 10797 tokens (179.8 tok/s) both survive, 512 tokens generated each, logs clean, engine alive afterwards.

…or IQ3_XXS/IQ2_S The Metal engine's hyper-connection read assumed the hc_ down/up/inject tensors were BF16, Q8_0 or F32 (the original and the Unsloth files). A fine-tune that re-quantized them (the OrcaRouter Uncensored IQ3_XXS and the RVN Uncensored IQ3_S both carry IQ3_S down/inject and IQ4_NL up matrices) made the gr_down_8_8/gr_up kernels read the bytes of a different format: the logits came out NaN on the first decode and the engine returned token 2147483647 (the all-NaN argmax sentinel in sample.metal), then died on the drafter's embedding gather. The HC-Q8 load step now takes hc_ matrices of every quantized format and quantizes them to Q8_0 once at load (290 matrices, 680 MB, 222 ms on an M2 Max), dequantizing from the tensor's own type; the existing gr_down_q8/gr_up_q8 kernels read them unchanged. BF16, Q8_0 and F32 keep the file's own fast path. gemv.metal's default instantiation list also missed IQ3_XXS and IQ2_S (moe.metal has both; plain gemv did not), so a model with an IQ3_XXS shared-expert gate or an IQ2_S attention projection failed with 'no function gemv_18_.../gemv_22_...'. Verified on an M2 Max, 96 GB, macOS 26.6.2, with the RVN Uncensored IQ3_S GGUF (88.9 GB, experts streamed): /health loaded, completions return real text, decode 9-10.4 tokens/s with MTP drafting, HC-Q8 step reporting 290 matrices. Before the change the same model crashed on the first request; after it, every reply is text. The PC builds are untouched (metal/ only).

…, tuned tiles), bitwise the same output

…lls leaves the anonymous namespace (declared in conversation_snapshot.hpp under STRATA_GPU_METAL) and a host test checks the round-trip identity, every unit's [page][head][in-page] address, and that nothing lands outside the image: the [cell]-index-as-page-index bug that wrote 4x past the scratch and corrupted the heap is exactly what the bounds check catches (window_cells_test: PASS; ctest 2/2)

…cannot run The Niko1221#879 guard reads the per-window non-finite flags that logits_nonfinite_rows raises on the device; the Metal port has no kernel of that name (stubs.cpp), so on that engine a logits row that is all NaN surfaces only as the sampler's argmax sentinel INT32_MAX in the pick. serve/server.py then met the id in its detokenizer and the raw IndexError killed the request thread, while the engine fed the sentinel into the drafter and segfaulted (exit -11) soon after. note_nonfinite() takes the picks it is called beside and marks the row of any sentinel pick: true on every platform (2147483647 is past every vocabulary), and the window's tokens then follow the existing Niko1221#879 path - none emitted, DONE nonfinite, cached state dropped. The server keeps its own arm: a sentinel id in the token stream raises EngineNonFinite, so the retry-once / end-as-length recovery runs even against a binary without the engine-side fix. Verified: serve/test_server.py NonFiniteLogits + new SentinelToken (7/7, mock engine); Metal engine rebuilt, ctest 2/2. Not run: the sentinel path with the real model on Metal - it needs non-finite logits, which only appeared under memory pressure.

… skips it, writes drop The stream window frees (purges) the CUDA-shaped device weight copies of >= 32 MB at startup - 20.68 GB back to macOS on the iq3_xxs config - and check_purged() aborts any copy that touches one. Writes are the shared host code's lazy uploads for buffers the Metal window never reads, so they are now dropped; a read still aborts (it would yield garbage). The state fingerprint (STRATA_STATE_HASH=1, run by the conversation cache and batch validation tools) read such buffers with cudaMemcpy and aborted a serving engine on its first cached request - it now skips given-back allocations via the new runtime.hpp is_purged(). tools/metal_purge_hash_harness.py proves it on one machine (purge + fingerprint + cached request + recall). Verification: build clean; serve unit tests (342 pass, 1 pre-existing VisionArgs environment error); runtime check of the crash path needs a full load - harness not yet run here (sushi holds the memory).

answer_room only applied at max_tokens >= 1024; a client that caps lower - the omp sub-agent harness sends 400 - got the thinking eating the whole cap: (length) with no text and no tool call, three retries, then "Length-stop recovery gave up ... with no text or tool call" and 3 of 4 scouts dead. Below the floor the thinking now gets the same share the Niko1221#984 reserve computes (400 leaves 200 to think and 200 to answer). "answer_room": false keeps the old behaviour; a request with its own budget is untouched. Test test_the_answer_room_has_its_limits re-targeted to the new contract (it pinned the no-answer behaviour); ThinkingBudget 12/12, full serve suite 342 pass + 1 pre-existing VisionArgs environment error (the same error on the clean tree).

The concurrency harness defaulted to 400 and strata_batch_validation.py repeated it. 400 is a thinking model's diet size: the reply gets cut while still reasoning (a real omp harness sends 400 too, and three "length" replies with no text killed 3 of 4 scouts before the serve-side answer-room fix). Default now 8192 - the omp binary's own fallback - and the validation tool drops its override and follows the default.

…ree() gives the buffer back Metal autoreleases a reference to every new MTLBuffer. The engine's threads have no autorelease pool, so that reference was never dropped: Context::free released the handle and the buffer stayed allocated for the life of the process (40 x 100 MB allocated and freed four times: 16.8 GB of footprint; with the pool, 0.01 GB after each free). (cherry picked from commit b19df7b)

…keys (the decode window keeps its own) The Metal decode window (metal/src/engine/window.cpp) keeps its own int8 K/V and pooled indexer keys per QSA layer, sized by the same max cells, and never reads or writes the session's. On Metal a session buffer is one shared MTLBuffer in the residency set, resident in full whether touched or not, so those unused pools cost about 24.6 KB per context token. Under STRATA_GPU_METAL, kv_plan gives the session's QSA states no K/V slots and no pooled rows; their pointers stay null, and a conversation snapshot refuses such a state (n_slots == 0) instead of reading it. M5 Max, Qwen3.8-Flash-Next IQ3_S resident, phys_footprint 5 s after the engine is ready: 16.99 -> 10.61 GiB at --max-context 262144, 9.27 -> 7.68 GiB at 65536. Greedy tokens unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0b82a7b)

… was before the session K/V change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 65befb3)

…t scratch buffers when the workspace grows - Context::alloc/wrap own their autorelease pools (fbdd077) so free() really gives back; the prompt-chain workspace records each scratch buffer and frees the old set before growing. Test: metal/tests/test_prompt_workspace.py

…ing on the Mac engine - temperature, top_p, top_k, min_p, seed, penalties, logit_bias reach the Mac engine through a host sampler with the PC's chain and order; before, every Mac request decoded greedily (Niko1221#3006). Tests: host_sampler_parity, tools/metal_sampling_test.py

…okens N[,READ] gates parked reads - a short request no longer restores a short parked copy it rewrites at once

…s refresh after a refused park - the Monitor and /metrics do not keep a stale occupancy

…ct per-slot state The Mac engine served one request at a time: `--parallel` (batch slots) was not ported, and the batch-row kernels were exit-86 stubs (docs/MAC.md). Several clients of one Mac queued behind each other even though the model sits idle at ~2 GB/s of weight reads between tokens. Now a batch window builds the solo one-token dispatch list once per row. Every kernel that touches state takes the row index and reads and writes only its own conversation's region, carved from the arena (the engine prints its cost at start); GEMVs keep the plain tile, so a batch row's bytes are the solo bytes in the same order. Conversation state moves between the window and a slot region through the same park and restore bytes the conversation cache uses, so --batch combines with --conversation-cache-mib. Regions that do not fit the arena turn batch off with a clear line instead of failing. One binding fix was needed for the second row to match: a batch row's GDN step binds the prepared activation at its own slice, and the per-16 sums it writes for the quantized-input GEMVs landed at that bound base rather than at the fixed sums region of the whole buffer. Rows after the first wrote sums where nobody read them (the read-back was stale) and past the buffer (trampling the arena's next allocation); layers whose ssm_out.weight is K-quantized dequantize from those sums, so the second row's answers diverged from token one. The sums now have their own binding (gdn.metal buffer 13); the solo window and the first row address them bit-identically to before. gdn_parity's new plane-row check covers it (slice vs base-bound, sums slot, sums vs CPU blocks): it fails with the pre-fix address and passes after. Validated on an M1 Max with 64 GB (IQ3_XXS, greedy): tools/metal_batch_try.sh 2 2 128 on 2026-10-22 - every slot identical to its solo run, 37.5 tok/s together against 23 and 25 alone; 315 per-step debug captures (STRATA_BATCH_VERIFY=1) match byte for byte on the second row. The 4 4 128 run on 2026-10-23: 34.7 tok/s against 23 to 32 alone, two slots identical, two choosing a coherent different phrasing at a near tie (token 88 and token 13 into the answer) - no slot answered another's prompt, and the per-slot state check (every GDN state, QSA cell, tail, dead, pooled and PLE-history buffer compared byte for byte) reported all match for all four regions. Batched greedy picks are therefore exact where a row's arithmetic replays the solo dispatch list; kernels that batch rows in one dispatch round at float-noise scale, and the gate is coherence plus no cross-talk, as CUDA's Exactness section documents. Not run here: sampled (temperature > 0) rows, tools/batch_interleave_test.py on a Mac. Made no claim without a measurement: docs/BATCHING.md's Metal section and docs/MAC.md carry the numbers with what they were measured on.

…ured 4-slot parity The 4-slot run (tools/metal_batch_try.sh 4 4 128, M1 Max, IQ3_XXS, greedy, 2026-10-23) showed two slots identical to their solo runs and two choosing a coherent different phrasing at a near tie (token 88 and token 13 into the answer); no slot answered another's prompt, and the per-slot state check - every GDN state, QSA cell, tail, dead, pooled and PLE-history buffer compared byte for byte against the window's - reported all match for all four regions. Nothing said about the diverging rows' drift: STRATA_BATCH_VERIFY armed the batch captures only on the first S=2 graph build and printed them at build time, so a 4-row step never printed a trajectory. Separately token_chain and prompt_chain died with "mtl: wrap 47039873024 failed": they wrapped the whole 47 GB weight mmap through the two-argument wrap, past the M1 Max maxBufferLength the engine's own three-argument chunked wrap exists for. Now every batch graph of two or more rows builds with its 315 fold captures when STRATA_BATCH_VERIFY is set, carries the probe count (Cached::caps), and each step - fresh build or cache hit - prints all S rows' trajectories from the same dsb_ slots. The chain tests pass the file's largest tensor as the chunk span, the way the engine's weight wraps do, so the split buffers each hold any tensor's full range. docs/BATCHING.md and docs/MAC.md no longer promise byte-identical batched greedy tokens: they carry the 2-slot and 4-slot numbers (37.5 and 34.7 tok/s against 23 to 32 solo, 15.4 GB of slot regions at four slots, 90 to 95 ms per graph build) and the CUDA gate - coherence, no cross-talk, exact state - with the near-tie explanation for the flips. Verified: ninja -C build-metal clean; gdn_parity, window_cells_test, host_sampler_parity, runtime_hello pass. Not run here: the per-step trajectory print and the fixed chain tests - both need the model to themselves, and this box's other big server holds the memory.

…ched PLE block, deterministic STRATA_BATCH_VERIFY=1 on tools/metal_batch_try.sh 4 4 128 (M1 Max, IQ3_XXS, greedy, 2026-10-23) printed 315-probe per-step trajectories for all four rows. Aligned to each slot's solo run (each row's printed stream starts where its first shared window begins): every slot's FIRST batched step is bit-exact on all 315 probes; on every later step the layer-0 folds and the fold entering the PLE block stay bit-exact while the fold leaving the batched PLE block already differs (0.3 to 4 percent of the folded sum, whose cancellations amplify the relative scale), and every downstream fold carries it. The same four prompts run twice - without and with the capture dispatches - diverged at exactly the same two tokens (88 and 13), so the drift is the batched block's deterministic arithmetic, not a race, and the captures perturb nothing. What rounds differently inside the block is not pinned yet and the text says so. Also corrected the graph-build cost sentence to the measured per-row-count range (56 to 98 ms). Verified: two 4-slot runs' engine.log trajectories compared probe by probe with an aligner; docs-only commit, no code touched.

… matches the engine The script took only three positional arguments, so `--keys` and friends reached nowhere, and its numeric gate demanded 50 GB of vm_stat raw-free pages when the engine's measured footprint is ~43 GB - it refused a box with ~49 GB genuinely free (macOS reclaims file-backed pages the raw count ignores). The refusal that matters is the big-holder check (sushi, a second engine), which stays. Now: extra arguments pass to batch_test.py; refuse under METAL_BATCH_TRY_MIN_GB (default 44) unless METAL_BATCH_TRY_FORCE=1; 44-50 GB is a thin-margin note, not a stop. Header text updated to the measured parity doctrine (exact identity of batched picks is not the gate; see docs/BATCHING.md). Verification: bash -n clean; gate branches exercised on a 49 GB-free box (continues with the thin-margin note) and a sushi-held box (refuses on the holder check). Full run re-measure handed to the user (sushi holds the box at commit time).

…2.7 tok/s tools/metal_batch_try.sh 2 2 64 --keys "temperature=0.7 seed=7" on the M1 Max (IQ3_XXS, 2026-10-23): 42.7 tok/s from the first batch token against 23 and 25 solo, and both slots' 64 sampled tokens identical to their solo run's draws (a fixed seed keys the engine's Philox draws by seed and position, so the two paths consume the same stream). BATCHING.md's sampled row moves from not-measured to measured; MAC.md's batch bullet no longer claims batched sums replay solo byte order - it carries the trajectory finding (state exact, drift enters at the batched PLE block) and the sampled result, and names the script's --keys form and the batch_interleave_test.py invocation. Interleave steps remain not measured on a Mac (the script starts its own engine and needs the memory to itself). Verification: quoted figures are from run_20261023_021422.log and engine.log (VERIFY lines all match); docs-only commit.

…ATCHING.md: Mac interleave measured On the M1 Max (IQ3_XXS, greedy, --max-new 64, 2026-10-23) the interleave steps all behaved: 126 slot tokens decoded while the 4888-token prompt was being read in; the read that gave way at a chunk boundary (YIELDED 2 1024) went on in its slot identical to solo; a request stopped part-way in a slot came back identical on the solo path; a next turn read from its slot stayed identical; a cancelled prompt chunk re-read from its checkpoint (1024 reused + 3866 read). The byte-identity gate missed three times - C at 32, F at 30, F2 at 27 - slots that decoded part of their reply in the slot window, the flips the PLE-block drift predicts; their coherence is text we could not read, because the script printed only the position. Each DIFFERS line now decodes fourteen tokens of both replies from the flip so the next run shows whether both sides stay coherent; IDENTICAL lines are unchanged, so CUDA gates see the same output as before. The Testing table's row points at the Metal section for what a miss means. Verification: ast.parse clean; the diff_note helper is exercised only on a miss, and the three misses in the run above were its motivation. Docs figures quoted from the run pasted by the user (2026-10-23).

…ove with timing The interleave harness re-run on the M1 Max (IQ3_XXS, greedy, 2026-10-23) printed both sides' decoded text at each byte-identity miss, settling their nature: slot D at output token 26 continued "...extensive and complex: It is described" where solo chose "...complex, characterized by a long" (both true openings of the same answer), F at 30 was one period after a bracket, F2 at 27 wrote 'static int compare_intervals' where solo wrote 'int' under a '// Comparator for qsort' comment (both valid C); no slot ever strayed off its own question. The two runs also showed that WHICH slot misses is timing-sensitive when admissions overlap the window (C missed the first run and was exact the second, D the reverse; the single-row tails F and F2 missed at the same tokens both times), while the batch script's runs, whose admissions finish before decoding, repeat their flip tokens exactly. The section now says this, replacing the first run's line that called the yielded read solo-identical (it was, that run; the second run flipped at 26). Verification: both runs' pasted outputs quoted; docs-only commit.

github-actions · 2026-10-23T00:29:50Z

Thanks for the pull request. To keep the tracker readable, every item needs an area prefix in the title and the parts of the pull request template filled in. This one is missing:

Fill in "Validation": the exact commands, the revision and environment tested, what you saw. Keep Passed, Failed and Not run apart.

Edit the pull request (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment /reopen.

Details: CONTRIBUTING.md

…ld indexer state (upstream 8311d8a) The serving engine re-read every conversation from token 0: each turn's park was skipped with "conversation snapshot: invalid indexer running-state target" (see strata-iq3_xxs.log from the 2026-10-22 rebase onward; the pre-rebase binary reused up to 33,871 prompt tokens a turn). Cause: 8311d8a (cherry-picked into this branch on 2026-10-21) gives the Metal session's QSA states no K/V pools and no pooled indexer rows - the decode window keeps its own - so st.idx_pooled is null and st.idx_pooled_rows is 0. conversation_snapshot.cpp was made window-aware for it (valid()'s via_window clauses), but checkpoint_targets still demanded a non-null idx_pooled and a pooled row per block, and it runs BEFORE e0b5bfd's STRATA_GPU_METAL bypass in conversation_checkpoint_validate and inside view_validate, so conversation_snapshot_bytes - serve's first park gate (generate.cpp) - failed every conversation of at least --conversation-cache-min-tokens. The tail, dead key and block_pos do live in the session and their checks stand; the pooled clauses describe bytes a Metal park never touches (the copy side re-checks payloads against the window's own caps, and window_checkpoint_restore revalidates every buffer size), so they are skipped under STRATA_GPU_METAL only. CUDA and SYCL branches byte-identical. Not measured here: this box cannot run the engine while sushi serves. Verify with metal/tools/parking_check.py --quick (single ON arm: gates on >=80% prefix reuse across side turns and parks/restores > 0) after stopping the other server, then the full A/B without --quick. parking_check.py also gains --quick; window_cells_test and host_sampler_parity pass on the rebuilt tree.

Upstream 8311d8a (in this tree since the 2026-10-22 rebase) stripped the Metal session states' pools and pooled indexer rows, and checkpoint_targets still demanded them, so every park was skipped with "invalid indexer running-state target" (strata-iq3_xxs.log; the pre-rebase binary reused up to 33,871 tokens a turn). DETAILS.md's conversation-cache section now names the cause, the fix (local 1f4372a) and the re-measured numbers (2026-10-23, parking_check.py --quick ON arm, M1 Max, IQ3_XXS), and notes the harness's new --quick mode.

github-actions · 2026-10-23T08:28:50Z

Thanks, this follows the format now.

…as PR branches - the M1/M2 8,1,2 gate/up tile for K-family experts (metal-m1-kexperts-tile), hot kernel reload POST /reload_kernels with Context::recompile_all (metal-reload-kernels), and the mmp instantiation for Q4_K/Q5_K (metal-mmp-q4k); upstream base 4b9e05a

…nt tail - the decode window keeps a whole block of indexer keys per QSA layer and writes no per-layer block position, so the CPU-shape product would refuse the engine's own file; the window_checkpoint_restore gate matches these bytes to the buffers

…docs, upstream's DS41 answer-room rule beside the fork's general one, the masked rerere resolutions rebuilt from the index stages

…es (metal-m1-kexperts-tile, metal-reload-kernels, metal-mmp-q4k) - this branch is the stack plus main

oceanio and others added 30 commits October 20, 2026 09:19

docs: Mac sampling (MAC.md Sampling: the keys, the order, seeds, the …

3a49b02

…host pick's cost, greedy vs sampled decode) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

metal: release prompt scratch buffers before workspace growth

093ed30

Merge current main and exercise prompt growth without caller autorele…

b9252ed

…ase pools

gap harness + attribution rounds 1-4; kprof gate reverted to baseline

1d7f1b5

metal compat: v0.1.45 merge fixes - peer-copy/priority/CC shims, Gemm…

b727146

…::bf16 and 262 regenerated link stubs

metal: the CUDA/HIP-only hyper-connection persist read is a no-op on …

ca7bb00

…Metal - fused_gr_persist_init/read_persist take fused_gr.cu's own non-CUDA semantics (v0.1.45's Verifier::init calls it every session; the abort stub killed serve startup: exit 86)

Metal: faster decode GEMVs on M1 / M2 (wider K-quant and IQ4_XS loads…

cc7a739

…, tuned tiles), bitwise the same output

docs (Mac): the engine's own memory grows with --max-context; what it…

6849962

… was before the session K/V change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 65befb3)

catalinb13 added 9 commits October 22, 2026 00:05

[Engine] merge PR Niko1221#3615 (G0-0000): --conversation-cache-min-t…

fcdec7e

…okens N[,READ] gates parked reads - a short request no longer restores a short parked copy it rewrites at once

[Other] merge PR Niko1221#3333 (oscarlius): conversation-cache metric…

c09fdf7

…s refresh after a refused park - the Monitor and /metrics do not keep a stale occupancy

github-actions Bot added the needs-format Title or template does not follow CONTRIBUTING.md yet label Oct 23, 2026

catalinb13 added 2 commits October 23, 2026 05:12

github-actions Bot removed the needs-format Title or template does not follow CONTRIBUTING.md yet label Oct 23, 2026

catalinb13 mentioned this pull request Oct 24, 2026

[Engine] metal: the conversation cache and session files keep the decode window's own K/V (#2502) #3541

Draft

catalinb13 added 2 commits October 26, 2026 17:14

catalinb13 added 2 commits October 26, 2026 18:17

[Engine] Metal: merge main (0.2.6, 4b9e05a) - keep both sides in the …

0204081

…docs, upstream's DS41 answer-room rule beside the fork's general one, the masked rerere resolutions rebuilt from the index stages

[Engine] Metal: leave the three deep-dive concerns to their PR branch…

72a21b4

…es (metal-m1-kexperts-tile, metal-reload-kernels, metal-mmp-q4k) - this branch is the stack plus main

catalinb13 force-pushed the metal-batch branch from 1ff3a38 to 72a21b4 Compare October 26, 2026 16:44

Navigation Menu

Repository navigation

[Engine] Batch slots on Metal: --batch N rows decode together - #3783

[Engine] Batch slots on Metal: --batch N rows decode together#3783
catalinb13 wants to merge 45 commits into
Niko1221:mainfrom
catalinb13:metal-batch

catalinb13 commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

catalinb13 commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Result

Approach

Validation

What is in this branch, honestly

Limits

How to try it

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

catalinb13 commented Oct 23, 2026 •

edited

Loading