[Engine] Batch slots on Metal: --batch N rows decode together

.gitignore

@@ -65,6 +65,7 @@ build-vision-cuda12.bat
 
 # local Windows HIP packaging output
 dist-hip/
+gap/.server
 
 # Terminal / Desktop split: local venvs, build output and archives never enter git
 .terminal-venv/

docs/BATCHING.md

@@ -512,15 +512,90 @@ several chats that alternate, come back without reading their history again. Mea
 two long conversations alternating through the HTTP server: the first turns took 3.9 s and 5.7 s to the first token
 (their prompts read), the follow-ups 0.53 s and 0.46 s.
 
+
+## On Apple Silicon (Metal) (experimental)
+
+The Mac engine does not run batch rows through CUDA's grouped tiles. With `--batch N` it builds a batch window of
+its own: the same dispatch list the solo one-token window makes, once per row. Every GEMV keeps the plain tile (no
+Gyro tile that changes the chunk order may enter a batch window), and every kernel that touches state takes the row
+index, and the row reads and writes only its own conversation's region. The region is the solo window's own layout -
+the GDN state and scan history per layer, the QSA K/V cells, the indexer's tail, dead and pooled rows, and the PLE
+conv history - carved from the window's arena, and the engine prints the cost at start (`strata metal: N batch slot
+region(s): X GB`). Batched greedy tokens are built to come out *identical* to the solo ones where a row's
+arithmetic is the solo dispatch list replayed on its own bytes. Kernels that batch the rows in one dispatch -
+and the conv add read all rows at once - round differently than the per-token solo sequence, so a row can drift
+from its solo run at float noise scale and pick a different word where two openings are a near tie. The gates
+that hold are the CUDA ones: every answer coherent, every slot reading only its own conversation, and every
+state buffer exactly right.
+
+The slot sessions are set up as on CUDA; their CUDA-shaped buffers are given back when the window's stream starts
+(the window's cells are the live K/V, as they are for the main session). A conversation's state moves between the
+window and a slot's region through the same park and restore bytes the conversation cache uses, so `--parallel`
+combines with `--conversation-cache-mib`: an admitted prompt reads through the main window and is copied into its
+slot's region before its rows first run. If the regions do not fit the arena the engine says
+`--batch is off: the window's slot regions do not fit` and serves one request at a time, as before.
+
+Measured on an M1 Max with 64 GB (IQ3_XXS, greedy, 128 tokens per prompt, `tools/metal_batch_try.sh`, 2026-10-22
+and 23). Two slots (`2 2 128`): **37.5 tok/s from the first batch token** against 23 and 25 tok/s for the same
+two prompts alone, and every slot's tokens identical to its solo run. Four slots (`4 4 128`): **34.7 tok/s**
+against 23 to 32 alone; two slots identical, and the other two diverged from their solo run by choosing a
+coherent different phrasing at a near tie (one at token 88 of a TCP-versus-QUIC answer, both continuations
+true; one at token 13 of a French history of pi, both fluent). No slot answered another slot's prompt, and the
+engine's per-slot state check - every GDN state, conv history, QSA cell, tail, dead and pooled row and PLE
+history of every region compared byte for byte against the window's - reported all match for all four regions.
+The captures (`STRATA_BATCH_VERIFY=1`, a 315-probe trajectory per step for every row) locate where the difference
+is born: every slot's first batched step is bit-exact against its solo run on all 315 probes, and on every later
+step the probes up to the fold entering the batched PLE block stay bit-exact while the fold leaving it already
+differs by fractions of a percent. The PLE block's rounding is therefore where a batch window's rows part company
+with solo, and the noise rides every downstream fold; a slot's tokens flip only where that noise crosses an
+argmax near tie (what exactly rounds differently inside the block is unconfirmed static analysis). A second run
+of the same four prompts with captures on diverged at exactly the same two tokens, so the drift is the batched
+block's deterministic arithmetic, not a race or corruption, and the captures themselves change nothing.
+The four regions cost 15.4 GB of the arena (the engine prints it at start); each row-count graph builds once
+(two-row windows in 56 to 60 ms, three-row in 73 to 81, four-row in 92 to 98).
+
+Sampled on the same machine and the same two prompts (`--keys "temperature=0.7 seed=7"`, 64 tokens each, the
+seed keeping the two runs' random draws comparable): **42.7 tok/s from the first batch token** against 23 and
+25 solo, and both slots' 64 sampled tokens identical to their solo run's - with a shared seed a sampled chain
+that never crosses a near tie stays byte-identical, but identity under sampling remains luck, not a gate.
+
+Getting the second row's sums right needed a binding fix also found with the captures: a batch row's GDN step
+binds the prepared activation at its own slice, and the per-16 sums it writes for the GEMVs must land in the
+fixed sums region of the whole buffer, not a slice-row's width past it. Rows after the first wrote sums no one
+ever read back, and the layers whose
+`ssm_out.weight` is Q2_0/Q3_K/Q4_K/Q5_K/Q6_K quantized (the IQ kinds read the plain float plane instead)
+dequantize from those sums, so their outputs were garbage on the second row onward while the first row stayed
+exact. The sums now have their own binding; the solo window and the first row address them bit-identically to
+before, and `metal/tests/gdn_parity.cpp` checks the row-slice layout against the base-bound one.
+
+The prompt-interleave steps of `tools/batch_interleave_test.py` are measured on a Mac too (same machine,
+2026-10-23, `--exe build-metal/strata --config <run config>`). Every structural step worked in both runs:
+about 126 slot tokens decoded while the 4888-token prompt C was read in; a read that gave way at a chunk
+boundary (`YIELDED 2 1024`) went on in its slot to the end; a request stopped part-way in a slot came back
+identical on the solo path; a next turn read from its slot stayed identical; a prompt chunk cancelled
+mid-read re-read from its checkpoint (`1024 of 4890 read`, cancelled, then `1024 reused + 3866 read`).
+The byte-identity gate missed three slots each run, and the second run printed both sides' decoded text at
+each miss, which settles what they are: near ties, not wrongness - D at token 26 continued "…extensive and
+complex:" with "It is described" where the slot wrote "…extensive and complex," with "characterized by a
+long" (both true sentences of the same history), F at 30 added one period after a bracket, F2 at 27 wrote
+`static int compare_intervals` where solo wrote `int` under a `// Comparator` comment (both valid C);
+no slot ever strayed off its own question. Which slot misses moves between runs (C missed the first and was
+exact the second, D the reverse, while the single-row tails F and F2 missed at the same tokens twice),
+because a slot's steps interleave with live prompt reads, whose arrival timing sets what each window row
+shares its dispatch with; the batch script's runs, where admissions finish before decoding, repeated their
+flip tokens exactly. To try it on your own Mac: `tools/metal_batch_try.sh 2 2 128` (add
+`--keys "temperature=0.7 seed=7"` for sampled rows); it refuses to run while another big server holds the
+memory (see the script's header).
+
 ## Testing
 
 Four scripts drive a built engine or a running server; each exits non-zero on a failure. `serve/test_parallel.py`
 tests the server's side with a scripted engine (no GPU).
 
 | Script | What it checks |
 | --- | --- |
-| `tools/batch_test.py` | the same prompts alone (`GEN`) and together in the batch slots (`BGEN`): every slot's greedy tokens equal its solo tokens; prints the aggregate rate. `--batch-groups` in `--extra` tests the pipeline, `--keys "temperature=0.7"` the sampled rows. |
-| `tools/batch_interleave_test.py` | a long prompt read while two slots decode, a prompt that gives way (`BYIELD`) and goes on, and a next turn continued from its slot: each equal to its solo tokens. On a two-stage layer split with pipelined groups add `--batch 4 --batch-groups 2` to `--extra` (its 3 slots do not divide into two groups) and set `STRATA_PIPED_READ_PIECE`; run it without a conversation cache, or step 4 restores the prompt instead of reading it. |
+| `tools/batch_test.py` | the same prompts alone (`GEN`) and together in the batch slots (`BGEN`): every slot's greedy tokens equal its solo tokens; prints the aggregate rate. `--batch-groups` in `--extra` tests the pipeline, `--keys "temperature=0.7"` the sampled rows. On a Mac run it through `tools/metal_batch_try.sh` (which guards the box; it refuses while another big server holds the memory): its exit counts exact identity, which near-tie flips can miss on a quantized MoE without any wrongness in the answer (see the Metal section). |
+| `tools/batch_interleave_test.py` | a long prompt read while two slots decode, a prompt that gives way (`BYIELD`) and goes on, and a next turn continued from its slot: each equal to its solo tokens, and it prints both sides' decoded text at the first difference (on a Mac the identity gate can miss at near ties without any wrongness - see the Metal section). On a two-stage layer split with pipelined groups add `--batch 4 --batch-groups 2` to `--extra` (its 3 slots do not divide into two groups) and set `STRATA_PIPED_READ_PIECE`; run it without a conversation cache, or step 4 restores the prompt instead of reading it. |
 | `tools/parking_test.py` | a follow-up to a conversation decodes the same tokens whether its state stayed live or came back from the parking cache (with a layer split: every stage's image). |
 | `tools/early_close_test.py` | a client that stops reading a streamed answer early (alone, and with a second request running) does not leave its tokens to the next request (server). |