[oceanio]

Problem and result

On the Mac (Metal) engine the conversation cache never keeps a conversation, and session files cannot be saved (#2502). A short side request, such as a client's title call, pushes the long conversation out, and its next turn reads the whole prompt again. There are two causes:

1. The memory guard has free-memory readings for Windows and Linux only. On a Mac it logs memory guard: the available RAM cannot be read; parking and session save/restore are refused at every start, and refuses every park and session save/restore.
2. A conversation snapshot copies the session's QSA K/V pools, but the Metal decode window keeps its own K/V and running state and never uses those pools. The checkpoint check also expects CUDA's running-state layout, so parking fails with invalid checkpoint running-state payload. After [Engine] metal: the session no longer holds unused QSA K/V pools (6.4 GiB resident at 262K context) #3536 the session no longer has the pools at all.

With this PR, parking and session files work on the Mac and carry the decode window's own state. Measured on an M5 Max with Qwen3.8-Flash-Next IQ3_S: a 28,785-token conversation is parked as 798 MB in 49 ms and restored in 41 ms, and its follow-up decodes the same tokens as the conversation that was never parked. As a session file it is 679 MB, saved in 0.16 s and restored in a new engine process in 0.10 s, with the same follow-up.

Related discussion

• Metal: a short side request drops the conversation cache (parking is refused, then "invalid checkpoint running-state payload") #2502 (bug report, both causes; closed in the 0.2.0 reset).
• Earlier fixes, also closed in the reset: macOS: read available memory for the conversation memory guard #2503 by @joesinvestments (cause 1) and Metal: the conversation cache keeps a conversation across a side request (cause 2 of #2502) #2631 by @PiMatyja (cause 2, parking only). This PR covers both causes, and differs in that:
  • the macOS reading is the Metal engine's own (free and purgeable pages and the clean file cache, as its streaming arena uses). Other programs' inactive anonymous memory is not counted, because macOS frees it only by compressing or swapping it. macOS: read available memory for the conversation memory guard #2503 counts inactive pages;
  • session files work too. Metal: the conversation cache keeps a conversation across a side request (cause 2 of #2502) #2631 refuses them on Metal. Their fingerprint carries the window's layout, so a file from a build with another layout is refused instead of misread;
  • it builds on [Engine] metal: the session no longer holds unused QSA K/V pools (6.4 GiB resident at 262K context) #3536, so a snapshot takes the window's K/V instead of pools the session no longer has.
    If either author would rather continue their own PR, I'm happy to rebase mine on theirs or close it.

Approach

Under STRATA_GPU_METAL only:

• src/core/conversation_memory.cpp: sample_host_memory() takes host_memory_now() (the Metal engine's free-memory reading) on macOS.
• metal/src/engine/window.cpp:
  • SlotState gathers the window's per-conversation state (a refactor, the same buffers);
  • SlotState::parts / layer_cells / cells_of walk that state as byte ranges in a fixed order;
  • Window::kv_layer and checkpoint_sizes hand the snapshot code the K/V cells [0, upto) of each QSA layer and of the MTP draft layer, plus the running state's sizes.
  • The draft layer's cache uses the same per-cell layout.
• src/core/conversation_snapshot.cpp: a snapshot's K/V layer is the window's own (format key 0x4D570001, page size 1), copied with memcpy (unified memory, the window idle). A K/V layer in any other format is refused.
• src/core/conversation_state.cpp: the checkpoint and session-read sizes come from the window's running state.
• src/program/generate.cpp: the session fingerprint names backend metal-window, the window's layout, and the MTP GGUF the window actually loads. The PLE residency policy (0.2.7) reads the same RAM figure as the memory guard; it stays off on Metal, as in a build that cannot read the RAM. It was measured on the PC, and with the macOS reading it would lock the 26.8 GiB PLE table of Qwen IQ3_S in RAM.
• Tests: tools/metal_snapshot_test.py and tools/session_restart_test.py are new. tools/parking_test.py reads the engine's own log path, and tools/batch_test.py's Engine starts the engine with the config's env, as serve does.
• Docs: MAC.md (Limits: the conversation cache and session files, measured) and DETAILS.md (the macOS memory reading, Metal in the fingerprint).

Cost: about 15.4 KB per token plus 118 MB per running state (the live one and each checkpoint kept with it). Nothing changes unless --conversation-cache-mib is set or a session file is saved or restored. With STRATA_GPU_METAL undefined, unifdef gives the same four shared C++ files as main's, apart from blank lines.

Validation

Re-run on v0.2.7 (2026-10-28). Rebased onto main 5e8a4cca (v0.2.7), on #3536, with no conflicts. The code commits are the same lines plus one change. v0.2.7 added a PLE residency policy that reads the same RAM figure as the memory guard. With this PR's macOS reading it switched on for the Mac and locked the 26.8 GiB PLE table in RAM: the footprint at 262K was 37.45 GiB instead of 10.61. It now stays off on Metal (see Approach). At the new tip 450e9623:

• Metal build 0.1.52-metal: 0 errors, the same 11 warnings as main.
• unifdef -USTRATA_GPU_METAL on the four shared C++ files, and on [Engine] metal: the session no longer holds unused QSA K/V pools (6.4 GiB resident at 262K context) #3536's layer.cpp: identical to v0.2.7's (blank lines aside).
• metal_snapshot_test.py: all pass, with and without MTP.
• parking_test.py at 3,682 and 28,692 tokens: the follow-up is identical (99/99 and 95/95), and with MTP the drafter's K/V read-back (STRATA_SNAPSHOT_VERIFY) passes. Without MTP at 28,692 tokens: identical (95/95).
• session_restart_test.py at 3,785 and 28,785 tokens: files of 294 and 679 MB, saved in 62 and 216 ms, restored in 37 and 203 ms. The follow-up is identical and reuses the whole conversation. Without MTP at 3,784 and 28,784 tokens (290 and 645 MB): the same.
• v0.2.7's engine refuses this build's file and keeps serving.
• Default path against v0.2.7 at 262K: 8 of 8 greedy requests are identical, and the footprint is 10.61 GiB (v0.2.7: 16.99).

The 28K-token runs, the runs without MTP and the refusal ran on the tip before the PLE change (cd2075d6). That change only decides how the PLE table is read.

The rest of this section is the first run, on 9de04cd4.

Environment: Apple M5 Max (128 GB), macOS 26.7.1, Apple clang 21.0.0. Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S (SC117's abliterated build, two-file layout), resident, --expert-cache 64 --prefill auto --spec 2 --mtp <rt> --mmap-experts. First run's revision: this branch's tip, on #3536 (main 9de04cd4 + 2 commits).

Passed:

• Metal build (cmake --build build-metal) at every one of the 7 commits; no new warnings.
• unifdef -USTRATA_GPU_METAL on the four shared C++ files (conversation_memory, conversation_snapshot, conversation_state, generate): identical to main's, apart from blank lines.
• tools/metal_snapshot_test.py --max-context 8192: all checks pass.
  • The cache stays on (conversation_cache_mib=1024), and a second conversation parks the first.
  • SAVE answers SAVED. RESTORE of a non-session file answers SERR invalid, and RESTORE of the saved file answers RESTORED 35 ....
  • The engine keeps serving.
• tools/parking_test.py --extra "--conversation-cache-mib 8192 --conversation-cache-slots 4":
  • 3,682-token conversation: follow-up identical (99 of 99 tokens);
  • 28,692-token conversation (--repeat 470): follow-up identical (95 of 95). Engine log: parked 28785 tokens in 48.8 ms ... bytes=797571752, restored 28785 tokens (live) in 41.1 ms. The second park of the conversation reused 439 MB of K/V it still held.
• tools/session_restart_test.py --extra "--max-context 65536":
  • 3,785 tokens: SAVED ... 294433688 73.5, RESTORED ... 39.4 in a new engine;
  • 28,785 tokens: SAVED ... 679433728 160.5, RESTORED ... 103.2.
  • At both sizes the follow-up is identical (99/99 and 95/95), and the restored engine reused the whole conversation.
• tools/session_restart_test.py --restore-exe <main's engine> --expect-refused: main refuses this build's file cleanly (SERR io 0 session fingerprint: ...), and the engine keeps serving.
• Default path against main at --max-context 262144, through each tree's own serve/server.py, without --conversation-cache-mib:
  • greedy output identical on 8 requests (4 short prompts, a shared system prompt with two questions, a follow-up turn and a rewind, each reusing 1,577 tokens);
  • footprint 10.61 GiB (main 16.98: [Engine] metal: the session no longer holds unused QSA K/V pools (6.4 GiB resident at 262K context) #3536's saving);
  • decode 92.0-95.3 tok/s (main 92.0-95.4).

Failed:
None observed.

Not run:

• CUDA, HIP and SYCL builds: no such toolchains here, and the shared files are compiled by every backend, so this is a draft. All changes are inside #if defined(STRATA_GPU_METAL). Could someone build it on one of them?
• The refusal of a file from a Mac build with another window layout was not re-run on this branch. Main's refusal above comes from its own fingerprint (it looks for the CUDA layout's MTP files), not from the layout key. It was checked before the split: a file from an earlier build of this change was refused, and the engine kept serving.
• Other Macs, the SSD-streamed experts, vision with parking, contexts above 262K, a file from the PC engine restored on the Mac.

Limits

• Builds on [Engine] metal: the session no longer holds unused QSA K/V pools (6.4 GiB resident at 262K context) #3536: until that merges, this PR also shows its 2 commits.
• Snapshots copy the window's state with memcpy while the window is idle (unified memory). They do not overlap with decoding.
• Only the solo conversation is snapshotted. Batch slots on Metal (--batch on the Metal backend is accepted at boot and exits the engine with code 86 on the first request that runs two sequences together #2715) are a separate change.

🤖 Generated with Claude Code

[catalinb13]

We carried the same goal to the end on an M1 Max (64 GB) box - a Metal engine 0.1.47 (a private fork with the Mac prompt path ported) serving Qwen3.8-Flash-Next UD-Q4_K_XL with the conversation cache and --slot-save-path: save, full engine stop and start, restore, continuation - and can report both the numbers and three gates a Metal session-file PR has to clear. All three were seen live in failed or passing runs, not from reading code.

Result (measured here, 2026-10-23):

• save: session saved 3348 tokens, 1 of 1 checkpoints, 283740824 bytes in 164.5 ms (state 40.3 ms)
• restore across a full restart: session restored 3348 tokens, 1 checkpoints, 283740824 bytes in 107.5 ms (read+check 100.7 ms)
• continuation: prompt 3375 tokens = 3348 reused + 27 read; the next turn 3423 reused + 18 read in 533 ms
• session file cost for this model: ~157 MB fixed (the window's GDN state 48 × (3145728 + 122880) B, PLE history 368640 B, indexer tails and dead keys) plus ~38 KB per token

Gate 1 - the file parser's read limits enforce the CPU indexer layout. A save that passes the park-side validation (payload written by the window's own checkpoint save) was still refused at restore, inside the parser: session file: element count 24576 exceeds this engine's limit 18432. Those are 12 QSA layers × the window tail - a whole key block, idx_block × idx_key_dim × 4 = 2048 B per layer - against the read limit's (idx_block - 1) × idx_key_dim × 4 = 1536 B (conversation_session_read_limits, max_state_bytes.tails). The window also writes no per-layer block_pos (its indexer derives the block from the token count). The GDN rows, the PLE history and the dead keys match the CPU counts byte for byte, so the tail is the only array that differs; whichever layout the save side writes, SessionReadLimits must describe it - bypassing the count checks on the validate/restore side is not enough.

Gate 2 - a model with no MTP tensors: the draft layer must be null on both ends. Our first save attempt failed with conversation snapshot: invalid or overflowing K/V byte estimate because the save side passed the drafter's ring as the draft layer. That model has no MTP tensors, the Metal window parks no drafter ring, and conversation_kv_bytes swallows the failed valid() and returns 0, which the byte estimate then refuses. The fix is the parking path's own choice (draft0 = ... !use_mtp ? nullptr : ...), applied symmetrically to save and restore: conversation_snapshot_validate demands an exact K/V layer count (layers + (draft ? 1 : 0)), so ends that differ break restore. Related to #1492.

Gate 3 - once parking works, the save side is nearly free: reuse the parking capture. We did not port the CUDA linear SessionKvSource streaming at all. The host image built by conversation_snapshot_save - exactly what a park builds, window K/V already repacked through the window's own pools - is what the session_file_write overload taking a SavedConversation& writes, the same format parking already round-trips on this platform. The engine-side delta is a Metal branch in the SAVE/RESTORE handlers plus the read limit from gate 1.

Verification. Passed here: save, restore across a full engine restart, continuation reuse (harness gate reused >= saved - 200). Not run: CUDA/HIP/SYCL builds - this Mac has no CUDA/HIP toolkit; the CUDA branches are textually unchanged under #else. Not measured: batch slot-region restores, K/V formats other than int8, restore across engine versions.

Suggestion: if it helps, we can rebase the delta (SAVE/RESTORE handler branches, the read-limit layout, the draft handling) onto current main and fold it in rather than open a third PR on the same ground.

[fzql]

We validated a complementary working implementation of Metal conversation parking on an Apple Silicon machine (M1 Ultra, 64 GB unified RAM) serving Qwen3.8-Flash-Next-IQ3_S at 160K context (--max-context 163840, STRATA_METAL_ARENA_GB=35).

Rather than opening a separate PR and fragmenting the tracker, we want to report our live hardware measurements, our patch approach, and a specific crash trap we observed with purged expert arenas that PRs targeting this area will want to guard against.

1. The Purged Arena Crash Trap (exit code -6 / SIGABRT)

In setups using expert streaming (STRATA_METAL_STREAM=1 and STRATA_METAL_ARENA_GB=35), Window::stream_arena purges the 4.45 GB startup buffer (sbuf). When --kv int8 is omitted from CLI flags, SessionState::kv_int8 defaults to false.

When conversation parking triggered:

1. conversation_kv_save fell back to FP16 and attempted to copy st.k_pool.
2. st.k_pool pointed to the purged sbuf device allocation.
3. runtime.cpp:check_purged() detected the access to the released allocation and aborted with exit code -6:

   strata metal runtime: a copy touches a device allocation of 4449.7 MB that was given back
   

Fix: In Window::bind_session(), we explicitly wire the live unified memory buffers (kq_, vq_, ksc_, vsc_, pooled_, tail_, dead_) and set:

st.kv_int8 = true;
st.kv_mode = 0;
st.n_slots = max_cells_ / 4;
st.n_pages = max_cells_ / 4;
st.max_cells = max_cells_;

This directs conversation_kv_save to copy the live Metal INT8 buffers instead of touching the purged FP16 arena allocation.

2. Live Hardware Measurements (Apple M1 Ultra, 64 GB)

Tested on engine 0.1.47-macos-arm64-metal, macOS 15, Qwen3.8-Flash-Next-IQ3_S (24 GB weights), STRATA_METAL_ARENA_GB=35, --conversation-cache-slots 4, --conversation-cache-mib 6144, --conversation-cache-min-free-mib 1024:

• Turn 1 (Session 1): 60,598 prompt tokens read, 6,685 generated tokens. Completed cleanly with zero crash.
• Turn 2 (Session 1 continuation): 67,283 tokens reused in 1.29 s (15.4 tok/s reuse, replacing ~2.5 minutes of re-prefill). 3,335 tokens generated. Parked 70,637 tokens in 963.7 ms (bytes=1831386928, 1.83 GB in RAM).
• Turn 3 (Session 2 alternating conversation): 119,037 tokens prompt read, 975 tokens generated. Parked 120,011 tokens in 1,246.6 ms (bytes=4365521844, holding both conversations in RAM simultaneously: 4.37 GB total).
• Turn 4 (Session prefix reuse): Borrowed 12,634 tokens in 137.3 ms across sessions (parked=2, borrowed=1).

3. Memory Admission Floor

On a 64 GB machine with a 35 GB arena and ~24 GB model footprint, host RAM headroom hovers around 1.5–2.5 GB. The default --conversation-cache-min-free-mib of 4096 MiB causes parking to be skipped. Lowering the admission floor to --conversation-cache-min-free-mib 1024 allowed snapshots to be admitted safely without thrashing unified memory.

4. Patch Delta

Our patch spans 74 lines across 3 files:

1. src/core/conversation_memory.cpp: sample_host_memory() on __APPLE__ via mach_host_self() / sysctl(CTL_HW, HW_MEMSIZE).
2. src/core/conversation_state.cpp: Tolerance for empty block_pos and 4-token aligned tails (2048 vs 1536 B).
3. metal/src/engine/window.cpp: Window::bind_session() wiring live INT8 buffers and restoring dead keys into pooled_ at row ids.size() / 4.

We have this tested and committed on branch fix/metal-conversation-parking (assisted by Gemini 3.8 Flash High). If helpful, feel free to fold these parts (especially the INT8 KV mode binding and purged buffer fix) into #3541 or #3564.

[catalinb13]

Two chips clearing the same path independently is good news - our M1 Max numbers for parking (A/B 2026-10-19: turns after a side request reuse 69-76% of their prompt and prefill in 6.5 s against 17.6 s without parking; park 37-39 ms, restore 8-10 ms, decode unchanged) now have an M1 Ultra witness at 160 K context. A few notes for the fold-in, all from runs on our box, not from reading your patch.

The SIGABRT trap: same root, fixed at the mode flag here. We do not let the FP16 fallback exist: the window's K/V kernels (qsa.metal) store int8 codes with an fp16 scale per 64 whatever --kv says, so the Mac engine forces qsa_set_kv_int8(true) when --kv fp16 is the default (generate.cpp, under STRATA_GPU_METAL). st.kv_int8 then always describes the bytes, and the fallback copy that reads the pool pointer never runs. And the park never reads the session's pool pointer even in int8 mode: save/restore resolve the state to the window's live kq_/vq_/ksc_/vsc_/pooled_ at copy time (window_kv_pools in window.cpp), so a purged sbuf is never touched even if a future flag combination gets the mode wrong. Your bind_session() wiring and our copy-time shim are equivalent on a single-window base.

One shape note for the merge, from --batch (#3783): a batch slot's SessionState must resolve to its own region's buffers, not the window's. Our shim resolves state-to-region and falls back to the main window only when the state is not one of the regions'. A window-side-only binding would park slot 1 onward out of slot 0's cells - not a crash (the buffers are live, so check_purged stays quiet) but silent cross-talk in the parked copy. Batch-slot region saves and restores are measured on neither box yet; #3783 carries the per-slot state work on top of the parking commits.

Tails and block_pos: exact agreement. Our parser-side number was the same 2048 B against the 1536 B read limit - the refusal we saw was session file: element count 24576 exceeds this engine's limit 18432 (12 QSA layers x a whole key block, idx_block x idx_key_dim x 4), and the window writes no per-layer block_pos either. Two independent trees converging on idx_block x idx_key_dim x 4 as the tail says the read limit - not the writer - is what should move.

Host-RAM sampler: worth picking one formula before the fold. Our macOS branch credits free + purgeable + min(inactive + speculative, external_page_count) - the rule the stream path's arena budget already uses - rather than free + inactive. On a wired-arena Mac we believe most of the inactive list is only reclaimable through the compressor, and that margin is what kept admission honest for us at the default floor; your min-free 1024 data point may be partly the formula, partly the 35 GB arena on 64 GB. Our daily envelope for comparison: arena 33 GB, cache 8192 MiB / 6 slots / min-free 2048 MiB.

Happy to run whichever shape lands - #3541 or #3564 - through our harness (park A/B, plus save and restore across a full engine restart with the continuation-reuse gate above) before it merges.

[lqdchrm]

Nice work, would be great to see this merged and released.

I did a hardware test on Apple M5 Max, 128 GB, macOS 26.6.2 (25G83) — the fix works here.

Revision / build

• this PR at 8542332b, engine reports 0.1.46-metal
• metal-cpp_26 (SHA-256 verified), llama.cpp 3cf03257, Apple clang 21.0.0, Homebrew cmake + ninja

cmake -S metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=third_party/metal-cpp -DSTRATA_GGML_DIR=third_party/llama.cpp
ninja -C build-metal strata

Model / settings: Qwen3.8-Flash-Next IQ3_S (pack, GGUF and MTP from Strata Terminal 0.1.49.2), --max-context 262144 --expert-cache 64 --prefill auto --spec 2 --mtp … --mmap-experts, STRATA_METAL_STREAM=0, expert profile from this branch's data/. <cfg> below is a config with exactly these args.

Commands (all with STRATA_METAL_STREAM=0)

python3 tools/metal_snapshot_test.py --exe build-metal/strata --config <cfg> --extra ""
python3 tools/parking_test.py        --exe build-metal/strata --config <cfg> [--repeat N] --extra "--conversation-cache-mib 8192"
python3 tools/session_restart_test.py --exe build-metal/strata --config <cfg> --repeat N
python3 tools/session_restart_test.py --exe build-metal/strata --config <cfg> --repeat 60 --restore-exe <released 0.1.48 engine> --expect-refused
python3 tools/parking_test.py        --exe build-metal/strata --config <cfg> --repeat 480 --extra "--conversation-cache-mib 512"
python3 tools/parking_test.py        --exe build-metal/strata --config <cfg without --mtp/--spec> --repeat 480 --extra "--conversation-cache-mib 8192"

Passed

Memory guard now reads macOS: limit 128.0 GiB (source: meminfo) … available 90.1 GiB (MemAvailable 90.1 GiB). No invalid checkpoint running-state payload in any run.

Parking (parking_test.py, A → B → A follow-up vs. live reference, greedy):

conversation A	park A	restore A	follow-up parked vs live	follow-up tokens
3,682 tok (default)	3,787 tok in 22.7 ms	3,787 tok in 14.9 ms	1.58 s vs 1.37 s	IDENTICAL 110/110
29,302 tok (--repeat 480)	29,404 tok in 70.2 ms	29,404 tok in 61.7 ms	1.42 s vs 1.45 s	IDENTICAL 100/100
100,672 tok (--repeat 1650)	—	100,781 tok in 157.6 ms	1.84 s vs 1.56 s	IDENTICAL 101/101
29,302 tok, no MTP / no --spec	—	29,404 tok in 19.6 ms	2.14 s vs 2.16 s	IDENTICAL 100/100

Second park of A reused retained K/V (448,821,600 bytes at 29k; 1,538,318,592 bytes at 100k, snapshot 2.47 GB). SNAPSHOT_VERIFY lines present with MTP (none without, as expected).

Session files:

• metal_snapshot_test.py: ALL PASS (cache on, park, SAVE, a non-session file refused with SERR invalid 0, RESTORE, engine still in step).
• session_restart_test.py --repeat 60: file 294 MB, restored 3,787 tok in a new process in 50.6 ms, reused 3,787, IDENTICAL 110/110.
• session_restart_test.py --repeat 480: file 689 MB, restored 29,404 tok in 122 ms, reused 29,404, IDENTICAL 100/100.
• --restore-exe <released 0.1.48> --expect-refused: refused with SERR, engine kept serving, follow-up re-read. Note: 0.1.48 refused via session fingerprint: mtp …/mtp/rt/dense.bin: No such file or directory (that MTP dir only has draft_vocab.bin), so the layout-mismatch path itself was not exercised.

Failed (expected fallback)

• parking_test.py --repeat 480 --conversation-cache-mib 512 exits 1: skip parking (snapshot 884 MiB exceeds available budget), no crash, the follow-up is re-read (29,430 tok in 29.2 s) and differs from the live one at token 46 — a re-read, not a restore, so the tool's comparison doesn't apply.

Not run

The HTTP server path (serve/, Anthropic /v1/messages) with this build; STRATA_METAL_STREAM=1; --vision; other Macs or memory sizes; any non-Metal backend. This branch is based on 0.1.46, so not tested on top of current main.

Before (same machine, released engine 0.1.48 5c8dd3225)

memory guard: the available RAM cannot be read; with --conversation-cache-mib 8192 every park fails with skip parking (conversation snapshot: invalid checkpoint running-state payload). A Claude Code session then re-reads its full 34–40k-token prompt after every side request (52–67 s each, see #3847). With this branch that is a ~60 ms restore.

[fzql]

Tested this PR rebased onto latest main (4b9e05a2, v0.2.6) on Apple M1 Ultra (64 GB unified RAM), macOS:

• Multi-client verification: Our earlier branch only held up for a single client, from which our use cases evolved. Under multi-client interleaved turns (~70k–90k tokens each on Qwen3.8-Flash-Next IQ3_S), naive host-RAM sampling and the redundant host KV buffer caused premature cache evictions and full ~150 s prefill resets. We are discarding our branch in favor of this PR.
• Results with this PR: Confirmed @catalinb13's points: setting p.slots = 0; p.pooled_rows = 0; on Metal in layer.cpp recovers ~6.0 GiB of unified RAM from the unused host session buffer, and arena_avail() in host_memory.cpp keeps admission honest so multiple large client sessions stay parked simultaneously without thrashing. Will further tune our setup.
• Build & Rebase: Rebased cleanly onto v0.2.6 (ninja -C build-metal strata passed with zero errors). +1 for merging.

(Assisted-by: Gemini 3.8 Flash High)

[oceanio]

Thanks all three of you. An M1 Max, an M1 Ultra and an M5 Max (the same machine as mine) give the same result, which is more than I could test here.

Rebased onto v0.2.6 (4b9e05a2), and #3536 under it too. The only conflicts were in the rewritten docs/MAC.md and docs/DETAILS.md. The code commits are the same lines as before. I re-ran the checks at the new tip (debc0b88) on the M5 Max with Qwen3.8-Flash-Next IQ3_S:

• metal_snapshot_test.py: all pass.
• Parking a 3,682-token and a 28,692-token conversation: the follow-up is identical (99/99 and 95/95).
• Session files across a full engine restart, at 3,785 and 28,785 tokens: 294 MB and 679 MB, saved in 64 and 150 ms, restored in 40 and 91 ms. The follow-up is identical and reuses the whole conversation.
• v0.2.6's own engine refuses the file (SERR) and keeps serving.
• Default path against v0.2.6 at 262K: 8 of 8 greedy requests are identical (the 9th carries a per-run tag). Footprint is 10.61 GiB against 16.99.

@catalinb13, on your three gates. This PR clears them this way:

1. Read limits. On Metal, conversation_session_read_limits takes the running-state sizes from the window itself (window_checkpoint_sizes): the tail is a whole key block per QSA layer, as the window writes it, and there is no block_pos. So the parser expects exactly what the save wrote.
2. No MTP. Now run too, with the same config minus --spec and --mtp. Parking a 28,692-token conversation gives an identical follow-up (95/95). Session files across a full restart at 3,784 and 28,784 tokens (290 MB and 645 MB) save and restore, and the follow-up is identical and reuses the whole conversation.
3. The save side. It works the same way here. The file is the image parking builds (conversation_snapshot_save), and the K/V comes from the window's own pools at copy time (window_kv_layer). No streaming SessionKvSource is involved.

So I don't think there is code to fold in, but a run of your harness on this branch would still be welcome as an independent check.

Host-RAM formula. This PR uses the same rule as yours. host_memory_now() passes free, purgeable, inactive, speculative and external pages to the streaming path's arena_avail(), so it computes free + purgeable + min(inactive + speculative, external).

Batch slots. Out of scope here. This PR snapshots the solo conversation only (Limits), and your note about a slot's SessionState mapping to its own region is the right condition for #3783.

@lqdchrm, thanks for the full matrix. You are right about the --expect-refused run: the released engine refuses the file through its MTP fingerprint, not through the layout key (v0.2.6 does the same), so that path is still only checked from before the split (see "Not run" in the description).

@fzql, thanks for running it on v0.2.6 with several clients.

[catalinb13]

Ran our harness on debc0b88 (M1 Max, 64 GB) - green. Thanks for the rebase and for clearing the gates the way you did.

Environment: the PR build 0.1.50-metal (cmake -S metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=... -DSTRATA_GGML_DIR=..., ninja strata, zero errors); Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, --mmap-experts with SSD-streamed experts (arena 33 GB), --max-context 262144, --spec 2, MTP as --mtp <pack>/mtp/rt with mtp-q2_0.gguf beside it. Streaming on (STRATA_METAL_STREAM=1), our daily tuning env.

Parking A/B (parking_test.py, reference vs parked, greedy):

conversation A	follow-up vs live	restore	second park of A
3,682 tokens	IDENTICAL 91/91	18.5 ms	53.7 MB K/V reused
28,692 tokens	IDENTICAL 95/95	52.6 ms	409.1 MB K/V reused
3,682 tokens, with MTP	IDENTICAL 91/91	26.6 ms	57.7 MB; SNAPSHOT_VERIFY present

Session files across a full engine restart (session_restart_test.py):

conversation	file	save	restore	follow-up reuse	follow-up
3,779 tokens	290 MB	318 ms	75 / 79 ms (two runs)	3779 = saved	IDENTICAL 91/91 both
29,406 tokens	654 MB	304 ms	184 ms	29406 = saved	IDENTICAL 95/95
29,407 tokens, with MTP	689 MB	1036 ms	188 ms	29407 = saved	IDENTICAL 95/95

Continuation-reuse gate (reused >= saved - 200): clears at every size, with and without MTP, with reused == saved exactly - so your parser accepted the window's own file on all six restores (gate 1 is closed by window_checkpoint_sizes), and both the MTP and no-MTP shapes round-trip (gate 2).

metal_snapshot_test.py: ALL PASS with and without MTP (parked 32 tokens in 125.2 / 52.9 ms, SAVED, SERR invalid 0, RESTORED, engine in step). One tool note: our daily config carries --conversation-cache-min-tokens 2048, so the 32-token conversation tripped the tool's own park check on the first pass; the rerun added --conversation-cache-min-tokens 16 to --extra. The tool may want to pass its own floor so a config's admission floor cannot fail the check.

Cost on our window geometry: a 32-token snapshot is 236.6 MB (fixed ~237 MB per state) and the marginal cost is ~14.2 KB/token, ~15.4 KB/token with MTP - in line with the numbers in your description.

One observation, not reproduced and probably outside this diff: of ten PR-build engine starts, one stalled on its first prefill at token 0 (the serve stall watchdog ended it after 60 s; all nine expert workers asleep, host idle 76.8 s). Its two reruns and the nine other starts were clean.

+1 from the M1 Max.

[oceanio]

Thanks @catalinb13 for running the whole harness on debc0b88. The IQ3_XXS with SSD-streamed experts is a setup I can't test here.

The tool's floor. You're right: metal_snapshot_test.py should not depend on the config's admission floor. Fixed in b0fe589a: the tool now passes --conversation-cache-min-tokens 16 after the config's arguments, the way it already passes --conversation-cache-mib 1024 (the engine takes the last value). A config's floor can no longer fail the park check.

The stall at token 0. I haven't seen it here. This machine runs with the experts resident (STRATA_METAL_STREAM=0), so it never takes the streaming path. I can't tell from here whether it is related to this PR. If it happens again, the engine log from the start up to the watchdog would help, mostly the last lines before the stall. Have you ever seen it on a build without this PR?

[catalinb13]

Numbers from a second M1 Max 64 GB box, plus one gap this route runs into when --mtp is on.

A build carrying this window-capture shape with the disk-save port, huihui Qwen3.8-Flash-Next UD-Q4_K_XL, 6 slots, --mtp ... --spec 2: a 100,882-token conversation saves to a disk slot in 1112.9 ms (1.67 GB, state part 288.4 ms); boot restore 492.7 ms; the first follow-up reuses 100,724 of 100,772 tokens in 7.4 s against 382.7 s cold; the recall needle stays correct across save, park and restart; the K/V parity check passes at 2.0e-05 relative L2; greedy decode 22.9-23.1 tok/s. Park alone: 323 ms and 141 ms at those sizes.

The gap: under --mtp the session-side MtpDrafter is a default state - the Metal window runs the draft rounds itself - so mtp.kv_state().max_cells is 0 and every readback through it is zero. The serve callsites that pass &mtp.kv_state() unconditionally when use_mtp is set (park, borrow, stage and whole restore validation, the disk slot validate/read-limits, and the STRATA_SNAPSHOT_VERIFY readback) then describe a draft part that does not exist: every park logs discard invalid snapshot and the conversation re-prefills from zero, a disk read refuses the engine's own file, and STRATA_SNAPSHOT_VERIFY=1 prints ERR verifying the drafter's K/V to the protocol stdout and return 1 stops the engine - which is how our parking_test.py run died before its restore-line check, with nothing new in the stderr log.

Gating those callsites on mtp.kv_state().max_cells > 0 (null draft part, and empty draft bounds in conversation_session_read_limits) restored byte-identical round-trips on the same build; the numbers above are with the gates in. #4384 carries the gate set; if #3564 lands first its bind makes the predicate true and the gates inert, so the two compose in either order.

Two harness notes that cost us a run each tonight, in case they survive into this PR's tooling: batch_test.py:59 writes the engine log under tempfile.gettempdir() while parking_test.py:55 reads a literal /tmp/batch_test_engine.log, so under a GUI shell's $TMPDIR the test dies without an engine log - pinning BATCH_TEST_LOG=/tmp/batch_test_engine.log fixes it; and Engine ignores cfg["env"], so the STRATA_METAL_* tuning keys from the serve config must be exported before the run or the harness tests an untuned engine.

[oceanio]

Thanks @catalinb13 for the 100K run on a second box.

The zero drafter on this PR. You're right that on Metal the session-side MtpDrafter keeps a default state (max_cells == 0): its load goes straight to the window's mtp_load. But this PR never reads that state. On Metal every snapshot step (layout, pools, the part sizes, the verify read-back) takes the draft layer's cells from the decode window: Window::kv_layer without an index returns the window's own draft caches (mkq_ and the rest). The &mtp.kv_state() that serve passes only names the draft layer. Your run on debc0b88 shows this: with MTP the park restored identically with SNAPSHOT_VERIFY present, and the session file was 689 MB against 654 MB without MTP. The ~35 MB is the draft layer's K/V (your 15.4 vs 14.2 KB/token).

So the failure you saw comes from a build whose draft state still goes through the CUDA layout. It does not happen with this PR alone. Could you change that line in #4384's description ("with #3541 alone the park/borrow/disk callsites still hand in the zero ring")? Otherwise a reviewer may read it as a gap here.

#4384 with this PR. The two don't compose the way #4384 says. With this PR, max_cells is still 0 on Metal, so every gate would take the nullptr side and drop the draft part this PR carries. A parked or restored conversation would then come back with another conversation's draft cache: the tokens don't change, but fewer drafts are kept. If both land, the gates should stay off on Metal (or ask the window whether it has a draft layer) instead of testing max_cells.

Harness notes. The log path is already fixed on this branch (81f4d24e): parking_test.py reads eng.log_path, the path batch_test.py writes. You're right about the config's env too: metal_snapshot_test.py passes it, but parking_test.py and session_restart_test.py use batch_test.Engine, which doesn't. Fixed in fa68278e: Engine now starts the engine with the config's env, and the test's own keys still win.