[oceanio]

Problem and result

On the Mac (Metal) engine every request decodes greedily. temperature, top_p, top_k, min_p, seed and the
penalties reach the engine on its GEN line but change nothing, and logit_bias is dropped. Since 0.1.45 the server
also sends the model card's sampling when a request gives none, and on a Mac that has no effect either. Nothing in the
log or the response says so (#3006).

With this PR the Mac engine samples as the PC engine does: the same chain and order (logit_bias, penalties, top_k,
top_p, min_p, temperature), one Philox draw per token keyed by the seed and the position, so a fixed seed repeats its
text and another seed gives another. Temperature 0 (or none) stays greedy and gives the same tokens as before.

Related discussion

#3006 (the bug, reproduced again on 0.2.0-beta; see Validation). A focused fix, so no scope issue first. #4126 takes
on the same bug; the differences are in my comment there (#4126 (comment)).

Approach

Verifier::run's Metal branch returned the window's argmax. The CUDA path samples on the host after its window; this
PR does the same on the Mac, on the CPU, over the window's head logits, which are already in unified memory (no copy).
A request that is greedy and has no penalties and no logit_bias keeps the window's argmax, so its tokens are exactly
the old ones. The MTP drafts stay greedy and a draft is kept when the sampled token equals it, as on the PC. Each kept
token is then the target's own draw at its position (Philox keyed by the seed and the position), so the drafts change
the speed, not the distribution of the text.

• metal/src/engine/host_sampler.cpp / .h: the CUDA sampler's chain with the kernels' float/double arithmetic and
  tie order. 9-46 µs per row on 248,320 logits (M5 Max), so no GPU kernel.
• metal/src/engine/window.cpp: an accessor for row t of the last window's head logits.
• src/core/verify.cpp: the sampling after the Metal window, and one line in the Metal init,
  if (le_ < 0) le_ = g.n_layers;, as the CUDA init does further down. The Metal init returned before that line, so
  le_ stayed -1, set_logit_bias read it as an earlier stage of a layer split, and the bias was never stored. Both
  changes are inside #if defined(STRATA_GPU_METAL): unifdef -USTRATA_GPU_METAL on this verify.cpp gives main's
  file byte for byte, so CUDA, HIP and SYCL compile the same code as before.

Validation

Re-run on v0.2.7 (2026-10-28). Rebased onto main 5e8a4cca (v0.2.7). No conflicts; the code is the same lines (one context line in Verifier::run moved: v0.2.7 added a guard above it). At the new tip f8fa7906, same Mac and model:

• Metal build 0.1.52-metal: 0 errors, the same 11 warnings as main.
• unifdef -USTRATA_GPU_METAL src/core/verify.cpp is identical to main's. The other files outside metal/ are
  docs/MAC.md and the test script.
• host_sampler_parity: 592 picks, all equal the reference.
• tools/metal_sampling_test.py: all 13 checks pass, among them logit_bias and the same seed giving the same tokens
  in a second engine process.
• Greedy against main v0.2.7 (temperature 0 through each tree's server, 64K context): 9 requests from 58 to
  15,099 prompt tokens, among them a shared 1.6K-token prefix, a follow-up and a rewind. 8 of 8 give identical text
  (the 9th carries a per-run tag). Decode on the 4 short prompts is 92.4-96.1 tok/s on main and 92.2-95.5 here.

The rest of this section is the first run, on 3fa1b524.

Environment: M5 Max 128 GB, macOS 26.7.1, Command Line Tools 27.0 (Apple clang 21). Qwen3.8-Flash-Next GSQ-RCO IQ3_S
(SC117's abliterated build, the official two-file layout), resident, --spec 2 with MTP, --mmap-experts,
--max-context 262144. Base = main 3fa1b52 (0.2.0-beta + 7 commits), built from the same tree.
Rebased since onto main e88ee39 (2026-10-20; its 5 new commits change no engine, Metal or server file): the
3 commits build again with all Metal targets, host_sampler_parity passes again; the runs below are on 3fa1b52.

Passed:

• ninja -C build-metal (all Metal targets), at each of the 3 commits: 0 errors, no new warnings.
• build-metal/host_sampler_parity (CPU): 592 picks, all equal a serial transcription of the CUDA kernels
  (sampler_parity.cpp's mirror); Philox matches 4,096 draws.
• python3 tools/metal_sampling_test.py --exe build-metal/strata --config <cfg>: 13 of 13 pass. On the base build the
  same script fails 11 of 13 (every seed gives the greedy text; penalties and logit_bias change nothing). 208
  one-token requests drew exactly the token a reference sampler (written in the test from sampler.cu) draws from the
  same dumped logits.
• [Engine] The Mac (Metal) engine ignores a request's sampling (temperature, top_p, top_k, min_p, seed, penalties) and drops logit_bias #3006's reproduction through serve/server.py (9 requests, the curl script in the issue): base, 9 of 9 print the
  greedy sentence; this PR, the greedy request prints the same sentence as the base, and the other 8 (seeds 1-3 at
  temperature 1.0, top_k/top_p/min_p, the card's defaults, penalties, logit_bias 760:-100) each print another
  sentence; the banned first token is not picked.
• Greedy unchanged: temperature 0 through the server on a 28-, a 2,257- and a 16,180-token prompt, identical text to
  the base build.

Failed: none observed.

Not run:

• CUDA, HIP and SYCL builds (no such toolchains on a Mac). The code they compile does not change (see Approach).
• Other Macs and models; Swift / Unsloth packs; batch slots (Metal has none on main, --batch on the Metal backend is accepted at boot and exits the engine with code 86 on the first request that runs two sequences together #2715).
• Not ported: the opt-in draft schemes STRATA_SPEC_COUPLED, STRATA_SPEC_PROB, STRATA_SPEC_GUMBEL (the Mac drafts
  greedily).

Limits

• Sampled decode is a little slower than greedy: 74.8 / 72.9 / 68.8 against 76.3 / 83.3 / 78.3 tok/s over 3 runs
  (4 short prompts, 256 tokens, MTP; --bench), because fewer drafts are kept (0.70 against 0.72) and the texts
  differ. The pick itself costs under 0.05 ms a token.
• The card's presence penalty now applies on a Mac too: a client that wants the old greedy text sends
  temperature: 0 and presence_penalty: 0 (documented in docs/MAC.md).

🤖 Generated with Claude Code

[joseedson18jc]

Tested on an M1 Max (64 GB, macOS 27), ported onto main at 0.1.51 (4b9e05a2, engine 0.1.50-metal): Qwen3.8-Flash-Next Q2_0, --spec 2 + MTP + --lookup-chain 8, --mmap-experts --expert-cache 64 --vision, through serve/server.py.

The sampler works on the Mac:

• host_sampler_parity: 592 picks, all equal the reference.
• Seeded requests are reproducible, and different seeds differ: 4/4 checks pass. The pre-[Engine] metal: the request's sampling on the Mac engine (temperature, top_p, top_k, min_p, seed, penalties, logit_bias) #3301 engine passed 1/4, because it decoded greedily whatever the request asked.
• No quality change on our probe set.
• The rest of the Metal parity suite: 16/17. fuse_parity fails on router_route, identically without [Engine] metal: the request's sampling on the Mac engine (temperature, top_p, top_k, min_p, seed, penalties, logit_bias) #3301 (the known M1 Max routing issue).

Cost: sampled decode is ~5% slower than greedy (34.0 -> 32.4 tok/s), because fewer MTP drafts match the drawn token (62.6% -> 54.1% kept).

Offsetting that cost on M1/M2: #4272. For a Q2_0 pack, the routed experts' tiles 8,1,2 (gate/up) + 4,2,4 (down) cut the expert block 17-24% at 1-3 tokens in expert_parity --bench. Live, bracketed B C B C on this same #3301 build:

	knob off	knob on
sampled	31.1 / 32.4 tok/s	33.8 / 33.3 tok/s
greedy	32.7 / 34.0 tok/s	35.8 / 35.7 tok/s

With both, sampled decode (33.3-33.8) is back at the old greedy-only engine's speed. #4272 is an opt-in env knob and is independent of this PR (one file, window.cpp).

This build has served a production site since 26 Oct with no sampler issues.

[oceanio]

Thanks @joseedson18jc for the M1 Max run, the parity numbers and the time in production.

The ~5% matches what I'd expect. With sampling, a draft is kept only when it equals the token actually drawn, so fewer MTP drafts survive than under greedy (your 62.6% -> 54.1%). #4272 is independent of this PR, as you say, so I'll leave it to its own review.