[xrip]

Problem and result

A conversation the RAM conversation cache evicts (its slots or its byte budget full) is dropped, and a client that comes back to it reads the whole prompt again: an agent that switches between its main conversation and sub-agents too large for the budget together reads ~150K tokens again on every switch. Every restart of the engine (an update, a settings change, a reboot, a poweroff) also reads every conversation from token 0 again.

• Before: an evicted conversation, and every conversation at a restart, costs a full re-read (73.3 s for 165,558 tokens on the box below).
• After, with --conversation-disk-dir DIR: the evicted conversation goes to DIR and comes back from there when a request continues it. With --conversation-disk-persist on|exit-only the conversations outlive the engine: the same 165,558-token conversation came back after a restart in 8.4 s, with the same answer.

Related discussion

The maintainer said in #1090 that one disk-persistence design will be chosen; this is one of them. It was closed in the 0.2.0-beta reset and is rebased here onto 0.2.1. #3404 (separate, in the conversation cache) drops stale parked copies of a conversation, which the disk tier would otherwise write out.

Approach

Off by default (opt-in options, help text and docs/DETAILS.md):

• conversation_disk.hpp/.cpp: the tier. Each image (one per stage with --layer-split) is one session file, written and read with the session files' own code (size, header and hash checks, the free-space preflight). A background writer (nice 10 on Linux) means a switch never waits for the disk, and queued images can be read back from RAM. Every checkpoint is kept, as in RAM. --conversation-disk-mib N (default 32768): oldest out first.
• conversation_cache.hpp: set_on_evict, so an evicted conversation goes to the tier instead of being dropped. A park that does not fit the cache's byte budget is also sent to the disk.
• generate.cpp: options, wiring, the read back when the disk holds a longer prefix than the session / a batch slot / the RAM cache (it needs RAM for the whole image above --conversation-cache-min-free-mib, else the prompt is read again), and the request line says when it was reused from the disk.
• --conversation-disk-persist on|exit-only: the files are bound to the session files' identity (model, engine version, settings), with an index per conversation, so the next start takes them in. Files of another identity are removed at start. On QUIT the engine saves the current conversation and the parked ones; while persist is on it ignores SIGINT/SIGTERM/SIGHUP and ends through its stdin, so a poweroff's SIGTERM does not cut the save short. The server waits up to 300 s for QUIT (serve/server.py, 4 lines). exit-only writes nothing while running and saves only at QUIT.
• A file is read back only when that is faster than reading the tokens it adds, by this run's measured disk and prompt speeds (the match reports its files' bytes; the prompt path's parts of 2,048 tokens or more are timed). A new conversation that shares only the system prompt with a saved one no longer reads that one's whole file. Until both speeds are measured the file is read; a skip is logged as not read. Second commit.
• conversation_disk.cpp is added to the CUDA/HIP, SYCL and Metal engine lists (all three compile generate.cpp).

Compatibility: with no --conversation-disk-dir nothing is written or read. One change stays in the parking path: before the RAM admission evicts a conversation, it calls glibc's malloc_trim(0), so freed images count in MemAvailable. Output is unaffected.

Validation

Passed:

• Rebased onto 0.2.1 (90767252) as two commits: 65a79b51 (the tier) and c6b99976 (the read check). CUDA build in Docker (nvidia/cuda:13.3.0-devel-ubuntu24.04, GCC 14, -DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=ON -DSTRATA_BUILD_CONVERSATION_TESTS=ON): strata and the tests build; conversation_disk_test 69 checks, conversation_cache_test 4,772, conversation_file_test 201, all pass.
• python -m pytest -q serve/test_server.py serve/test_parallel.py serve/test_parallel_yield.py serve/test_request_accounting.py: 392 passed, 3 skipped, 444 subtests passed (Windows 11, Python 3.14).
• Hardware end to end on my fork's main 342185bd: that is 0.2.0-beta plus this disk tier, plus one diagnostic log line and fork-only serve/tool patches (the disk-tier code is otherwise this commit). Box: 2x CMP 50HX 20 GB (sm_75, layer split), 2x Xeon E5-2670 v2, 64 GB, NVMe ext4, Ubuntu 24.04.4, driver 610.43.03, Flash-Next IQ3_XXS; exit-only, --adapt-swaps 0, greedy, 48 tokens.
  • Three conversations that share no prefix (41,839 / 41,846 / 165,558 tokens, cold 22.5 / 23.3 / 73.3 s), then SIGTERM to the server: saved for the next start in 17.0 s (5,786 MiB).
  • The next start took in all three. Each came back from the disk with the same answer as before the restart, in 2.6 / 2.9 / 8.4 s.
  • The disk tier has been in daily use on that box since 2026-10-11. Since 2026-10-20 on a 12 TB 7200 rpm HDD with on (my fork on 0.2.1 with both commits and [Engine] conversation cache: drop an older copy of the conversation rewritten turns back (#342 follow-up) #3404): a 304,364-token conversation came back from the disk in 47.6 s (5,465 MiB read in 44.1 s, then the restore) and a 286,199-token one in 41.7 s, where reading them takes ~140 s.
  • The read check, live on the same box: after a 124,097-token conversation came back from the HDD in 19.6 s (2,856 MiB at 169 MB/s), a new sub-agent that shared only the 16,384-token system prompt with a saved 2,018 MiB conversation skipped that file: not read: its 16384 more tokens read in ~9.4 s, its file (2018 MiB) in ~12.5 s. On the NVMe days the same case read whole files: four reads of 4-6 GB for 13,764-16,384 tokens (6,033 MiB in 11.4 s for 16,384 tokens).

Failed: the same end-to-end check with adaptive expert swaps on (the live config) gave one restored conversation a different first token. Swaps make decode non-repeatable on this box even without a restore; with --adapt-swaps 0 the check passed (above).

Not run:

• HIP, SYCL, Metal and Windows builds and runs: I have no environment for them.
• The restart check on these exact commits: it ran on my fork (above), whose disk-tier code is the same, and that code has served the box live since.

Limits

• Linux, one box.
• With RAM this short, a park waits for an evicted conversation's write before it has room: 28.6-31 s for 5.5-6 GB on the HDD, 7-8 s on the NVMe.
• The streamed park/restore of the earlier version of this PR (used when RAM is short) is left out to keep this one smaller; it can follow as its own PR.

[midhatn]

This looks useful for parent/worker conversations that need to survive a full engine unload. I reviewed eb2493bd and found one restart correctness case worth checking before adopting it.

read_index() supplies the prefix keys, but take() reads the session payload without checking that its actual keys match the selected index. The restore path then keeps the index's resume / from_live decision.

Could the prefix be recomputed from the parsed payload, or explicitly validated before replacing the outgoing state? A regression case would pair a valid A payload with an equal-length B key in the persistent index, then request B. It should reject that association or reread B, never reuse A's state as B. Please include image keys and the cvec flag in that check; an index checksum alone would not bind a valid index to the right payload.

This is a source-review finding, not a runtime reproduction. I have a bounded Windows A/B/unload/A multimodal test ready, but have not run this PR yet.

[xrip]

Thanks, good catch. Fixed in 1d52b02: after the read, take() now checks that the payload's live and checkpoint token ids, image keys and cvec flag equal the index's, in every stage; on a mismatch the read fails and the prompt is read cold. conversation_disk_test has your case: a valid A payload under an equal-length B index, then a request for B. It is refused and A stays usable; without the check the test fails.

[midhatn]

I’ve now run the multimodal unload test with 1d52b02c, composed with my experimental adaptive unload wrapper and local bounded-read guards. The same executable was used in both arms; the separate current-session snapshot mechanism was disabled. Only disk persistence changed.

On Windows with a Ryzen 9 7940HS, RTX 4070 Laptop 8 GB and 64 GB RAM:

Sequence	Disk off: full handoff	Disk on: full handoff
A → unload → C++ build → A	78.30 s	66.72 s
A → B → unload → C++ build → A	79.91 s	65.19 s

The full handoff includes draining/persistence, confirmed process exit, the real four-job build, release, reload and the answer. It also includes the test’s bank-file verification overhead. These are reductions of 14.8% and 18.4% in this single matched pair.

Both disk-enabled unload cases recorded an actual disk hit followed by restoration and 1,431 reused tokens; the disabled cases reused zero. Image embedding bytes and token IDs matched across the transition. All 16 requests and four compilation cycles passed the answer, memory-floor and owned-process cleanup checks. The A/B/A controls without unloading also passed, separately as RAM-cache tests.

Return-request prompt processing fell from 14.38 to 0.59 seconds for A, and from 14.24 to 0.71 seconds for parked A. Generated reasoning lengths varied despite matched seeds, so the whole-handoff figures are end-to-end observations, not isolated kernel-speed measurements.

This was a small ~1.4K-token image prompt with 64K capacity, not a filled-64K or complete Hermes workload. I kept the build frozen during testing, so this does not include the later f0ed90e8 timing-log change. The result supports the parent/worker handoff use case without claiming a general speedup.

[xrip]

Thanks for running it, and for the careful setup (same build in both runs, only disk persistence changed). Good to see it on Windows and with an image prompt; I had only run Linux and text.

For comparison, numbers from a real working session on our side: the live server log of one engine run at 8db7df60 (with the index check, before the log-line change).

Setup: 2x NVIDIA CMP 50HX 20 GB (Turing sm_75, unlocked to RTX 2080 Ti clocks, PCIe Gen2 x16, --layer-split auto), 2x Xeon E5-2670 v2 (AVX1 only), 64 GB DDR3-1866, NVMe ext4, Linux; Qwen3.8-Flash-Next IQ3_XXS, 262K context, int8 KV; --conversation-cache-mib 8192 --conversation-cache-slots 3 --conversation-cache-min-tokens 32768 --conversation-disk-dir ... --conversation-disk-persist (32 GiB budget). Client: the pi coding agent with the pi-blackhole extensions, a main agent plus many sub-agents.

The session: 271 requests, 33.9M prompt tokens (median 127K, max 230K), 92.1% of them reused; prompt time 1,897 s, decode 305K tokens in 3,873 s.

• Restart: the engine took in 9 conversations (20.1 GB) left by the previous run's stop (saved in 4.6 s). The first request continued an 84,385-token one from the disk: 2,747 ms read (1,456 MiB) + 572 ms GPU restore = 3.3 s, where a cold read runs at ~1,850 tok/s here (~45 s).
• RAM cache: 54 restores (0.3-2 s each) did most of the work; the disk tier is its overflow.
• Disk tier: 21 conversations (1.84M tokens, 31.5 GiB) were written instead of dropped, at 60-790 MB/s; the budget filled and removed one. None of these 21 was asked for again in this session: most are finished sub-agent tasks, which the agent does not come back to.
• A cost I had not measured before: 5 requests waited for queued writes to give their RAM back, 25.5 s in total (1.7-6.8 s each); the low-priority writer is slow (60-70 MB/s) while the prompt read keeps the CPUs busy. So in this session the disk tier saved ~42 s on its one hit and cost 25.5 s of waits; its main win here was the restart.

Fixed in 29a7110: a request that needs RAM now drops the queued images (they leave RAM at once, as they did before the disk tier) and waits only for the one already being written; the totals line counts them as "dropped for RAM". conversation_disk_test covers it (79 checks); not yet measured in a live session.

[midhatn]

Thanks for measuring the queued-write cost. That is especially relevant when the same machine needs its RAM back for a tool.

One additional Windows result on the same frozen 1d52b02c composition: a 31,904-token image-bearing parent, a short worker, full unload, a real four-job C++ build, then the identical parent request. The return restored 31,899 tokens and read five; prompt processing took 0.836 s. The complete unload/build/reload/answer handoff took 67.125 s, including 1.532 s of test-only cache-file hashing. All three answers, image/token identities and owned-process cleanup passed.

The unchanged 2 GiB RAM bank and 2 GiB disk budget accommodated this case: about 1.018 GiB of RAM-cache accounting and 0.908 GiB of persisted payloads/indexes. Lowest sampled available RAM was 7.389 GiB. This was one synthetic long-parent/short-worker case, not two long conversations or a matched long disk-OFF comparison.

I have not tested 29a71104 yet. I am keeping the next complete Hermes media trial on the already-tested build so its result remains attributable, and will include queued-write waits in the review.

[xrip]

A follow-up from the same live sessions, now in its own PR: #1959 (--prompt-cache-spread, opt-in).

To see what the caches still miss, I added a diagnostic line to our build: when a request reads 4K+ tokens again that the live session, a parked conversation or one on the disk already had, it logs how many. In one ~40-minute pi coding agent session it caught 421K such tokens (~227 s). Most of it (~200 s) was not about parking at all: the agent (the pi coding agent with the pi-blackhole extensions) changed its history a few thousand tokens back, and the conversation, still held in RAM, resumed from the system prompt:

• a 210,528-token prompt, the same as the session for 204,349 tokens: resumed at 9,525, 201,003 read again (100 s);
• a 205,253-token prompt, the same for 163,508: resumed at 0 (101 s).

The cause is the checkpoint retention: least recent use, and every turn adds a checkpoint, so after a few short turns all 6 sit in the last 1-2K tokens. #1959 keeps them spread back from the end instead (the same number of checkpoints, no more RAM): an edit 6K back reads ~8K again instead of ~200K. On the box an edited 82K conversation resumed at 73,610 tokens in 10.2 s instead of 40.0 s, with the same answer as a full read when the expert placement is fixed.

It is independent of this PR, so a separate one. Conversations parked in RAM keep their spread checkpoints; the disk tier here writes only the live state and the deepest checkpoint (the session files' rule), so a conversation back from the disk starts with those two.

[midhatn]

The complete Windows Hermes media trial has now finished on the same frozen 1d52b02c composition. It restored four disk prefixes (11,555 / 19,814 / 11,132 / 24,974 tokens) across sequential image and video handoffs. Both workers returned and the supervisor completed its report; the native logs recorded no disk read/write failures.

The two natural unloads logged 402 ms and 465 ms waiting for queued writes, with 0.7 s and 0.6 s total save-for-next-start times. These are nested native measurements, not extra time to add to the full handoff. The first supervisor requests after the worker returns reused 19,814 and 24,974 tokens; their full request times were 49.59 s and 72.82 s, including new input and generation.

Exact scope, source differences and the separate audit correction are recorded. This is one complete workload, not a matched persistence speedup test, and does not validate your later queued-write change. Thanks for pointing out #1959 too; its history-edit case is worth a separate matched test. Our largest remaining prefill pause in this run came from genuinely new verbose tool output, so spread checkpoints would not remove that particular cost.

[Interpause]

A stopgap solution in the meantime might be to mark conversation cache as MADV_COLD or MADV_PAGEOUT

[xrip]

Thanks - that would help with the RAM pressure on Linux, as a quick measure. A few reasons it doesn't cover what this PR is for: it needs swap sized for the parked images (1-4 GB each here, while swap is often a few GB); it doesn't survive an engine restart (the --conversation-disk-persist case); there is no equivalent on Windows that keeps the pages; and the cache's byte budget still counts the parked images, so the evictions that started this still happen. On the restore side, swap-in is page faults while the tier reads one file sequentially (500-950 MB/s here).

Interestingly, on our box the kernel already swaps part of the parked images (3 GB of swap in use, 4 GB available), which matches some slow 6-8 s restores of 220K-token conversations - so I'll look at keeping the RAM budget below that point and letting the tier take the rest.

[xrip]

Correction to my earlier comment about 29a7110: dropping the queued writes when a request needs their RAM turned out to be the wrong trade. In a longer session (657 requests, the same box and agent) it dropped a 180K-token conversation (3.3 GB) whose write would have taken ~4 s, and the next request of that conversation read it again from token 0 (~90 s). A few seconds of waiting for a write is far cheaper than losing a long conversation, so 3fe88e6 reverts it: a request waits for the queued writes again, as before 29a7110. Your tests on 1d52b02 predate both, so they are unaffected.

[Interpause]

A stopgap solution in the meantime might be to mark conversation cache as MADV_COLD or MADV_PAGEOUT

I ended up combining MADV_COLD plus this PR for best of both worlds, if anyone wants to reference main...Interpause:fork-Strata:new-main but im not going to PR probably since it probably only helps a few people.

[Interpause]

Suggestion from testing: would it be possible to evict slots straight to disk when insufficient RAM?

conversation cache: skip parking (physical RAM admission; need 3878 MiB plus 0 MiB floor, 2474 MiB available, source meminfo; evicted 0, 0 still parked)

To mitigate, I already maxed vm.swappiness and limited to 1 slot (so every slot immediately gets evicted to disk), but I still run into cases where it refuses to cache because of insufficient ram. So it would be great if in such cases, it bypassed ram and went straight to disk.

[xrip]

Thanks - I hit the same line here. In both of ours the telling part is evicted 0, 0 still parked: nothing is parked, yet MemAvailable is short. In my case that was the freed images of the evicted conversations staying in glibc's heap (9.6 GiB of it, the expert arena aside), which MemAvailable does not count, so the admission refused a 190K-token conversation that the process had room for.

8552d06 (now in this PR) calls malloc_trim(0) before the admission evicts or gives up, and after each eviction. On 2x CMP 50HX it took the refusals from 53 in 657 requests to 0 in 1,401 requests over four sessions. Could you try it before anything bigger? If you still get refusals with it, the RAM is really short, and streaming the image straight to disk (and back) is the next step; session SAVE already streams the K/V from the pools on one GPU, so that would be its extension.

[WarlockSyno]

This would be an amazing feature. I only have enough SRAM and VRAM to support 1 large conversation, so there's LOTS of prefill time on my box when I run more than one conversation.

[xrip]

Thanks - that is the case it is for: with --conversation-disk-dir the conversation that does not fit stays on the disk instead of being dropped, and comes back in seconds instead of a full re-read (here a 165K-token one in 6.4 s against 77.8 s). If you try the branch, the conversation disk: lines in the engine log show each write and hit; a log of a session would help.

[xrip]

I cleaned up the branch for review: it is rebased onto the current main (no conflicts) and the 9 working commits are now 3, one per part:

1. 561bc75d The disk tier (--conversation-disk-dir). Evicted conversations are written to the disk in the background and read back on a later request. It has a byte budget and a time limit, All-logits-NaN degeneration (one repeated token forever) still fires on 0.1.39b / current main — first poison traced to the MoE output rows: garbage bits, varying layer #879 clears it too, and the request line says when a prompt came from the disk.
2. f772a530 --conversation-disk-persist. On top of 1: the files keep the session files' identity, and an index beside each lets the next start take them in. QUIT saves first; then Ctrl+C ends the engine through the server. A read refuses files that do not hold the conversation their index names.
3. f731322c malloc_trim in the RAM admission. This one does not depend on the disk tier: freed images stay in glibc's heap, which MemAvailable does not count, so parking was skipped. On our box skipped parkings went from 53 in 657 requests to 0 in 1,401.

At each commit generate.cpp compiles and conversation_disk_test passes (59 checks after 1, 77 after 2 and 3). The final code is the same as before the split, except for small cleanups: wait_idle() returns nothing now, put() takes its lock once, and one duplicate include is gone. The drop_queued change and its revert are no longer in the history.

If you want only part of it, 1 works alone, and 3 applies cleanly on 1 without 2.

[WarlockSyno]

Following up with the log I mentioned. Self-hosted, single user, 2x RTX 3080 20 GB with --layer-split auto, 62 GB DDR4, Samsung MZVLB256HAHQ 256 GB NVMe (1.9 GB/s sequential, measured with a 1 GiB O_DIRECT read), Strata 0.1.43.1, Qwen3.8-Flash-Next IQ3_S, 524288 ctx, --conversation-cache-mib 12288 --conversation-cache-slots 3 --conversation-cache-min-tokens 8192 --conversation-cache-min-free-mib 1536 --session-save-reclaim.

The shipped session-file path is not usable on this box at all (session files do not support --layer-split), so this branch is the only option here - and the failure mode it does not cover is the one that dominates.

What the gate does on this box

One 7.7 h window from the engine log, 551 requests, segmented at the start markers:

parks	109
refusals (physical RAM admission)	109 (50%)
refusals reading evicted 0, 0 still parked	96 of 109
refusal arithmetic	need p50 5658 MiB + 1536 floor vs p50 4460 MiB available (short ~1.2 GiB)
parked images	p50 2.38 GiB, max 5.97 GiB
snapshot copies written to host RAM	249.5 GiB in the window

A representative line:

conversation cache: skip parking (physical RAM admission; need 5385 MiB plus 1536 MiB floor,
2126 MiB available, source meminfo; evicted 0, 0 still parked)

Because the refusal happens before the snapshot is captured, there is nothing for the tier to take: all 109 refusals dropped the conversation, and the next request read the whole prompt again. 138 of 551 requests reused nothing, and those cold reads are ~85% of all prompt-read time (a 20-50K cold read is ~16 s, 50-80K ~23 s, 200K+ p50 81.5 s / p90 104.2 s). Alternating between 2-3 long sessions, one is parked (2.2 s to switch back) and the rest pay full prefill on every turn. The 109 refused conversations are p50 4.03 GiB, p90 4.96 GiB images, i.e. roughly 2.3-2.8 s of reads on this NVMe.

The ask

set_on_evict covers eviction. It does not fire on an admission refusal, and on this config that is the common case. Would you consider capturing on refusal too - when the physical-RAM gate refuses, build the image straight to the tier instead of dropping the conversation? With the numbers above, a 2-3 s read replaces a 20-120 s re-read for exactly the conversations that currently have nowhere to go. If that is out of scope for this PR I am happy to prototype it on top of f731322c and post measurements either way; I can build and run it on this box (layer split, 524288, IQ3_S) and report parks, refusals, restores and per-request read times before/after.

Two smaller questions while I am here:

1. malloc_trim in f731322c matches this box's signature exactly: 109 of 109 refusals show evicted 0, 0 still parked while MemAvailable reads 5.0 GiB beside an engine at 53 GiB RSS and 3.2 GiB swapped. Did it change anything in your case where the need genuinely exceeded available memory, rather than only the heap residue?
2. Does a restore have to materialise the whole image in RAM on this config, or does the 16 MiB two-pass path apply here? A p50 4.0 GiB restore plus the 1.5 GiB floor fits in the ~10 GiB that should be free when nothing is parked (5.0 GiB measured available plus the 4.85 GiB parked image, so an estimate) - but if two long conversations are alternately resident, the restore preflight may refuse for the same reason the park does.

One cost data point for the design: 109 parks in 7.7 h is 249.5 GiB of images, so a per-turn full-image write is a real amount of churn on a 256 GB consumer drive - the same pathology as #2264, on the RAM side. An incremental/delta write would matter for us eventually.

For reference

The design question - when an entry no longer fits, is it dropped or does it fall to a lower tier? - has a well-documented answer in the serving stacks, which is why I am asking rather than assuming:

• LMCache treats the RAM/CPU tier as a budget with the disk below it rather than as a gate that discards state ("a tiered storage hierarchy spanning CPU memory, local storage, and remote backends"), and runs its cache in a process independent of the engine ("no fate-sharing"), which is the property --conversation-cache-persist is reaching for. Paper: arXiv 2510.09665.
• Alex Ziskind's video from today, "My Coding Agent Took 19 Seconds to Say Anything" (https://www.youtube.com/watch?v=bYpdCFz7VtA), is an independent public measurement of the same effects: against a GPU cache that holds ~1M tokens, 48 documents (1.5M tokens) give vLLM's GPU prefix cache 0% hits and it "ended up being about as slow as no cache at all"; with an NVMe tier below it the same workload answers in ~265 ms, and still ~265 ms after a vLLM restart, where the in-process caches lost everything. He also runs Qwen3.8-Flash-Next - the model Strata serves - where vLLM's own CPU offload would not start at all, and his non-prefix (CacheBlend) attempt could not run on vLLM either. Worth 15 minutes for anyone following this thread.

[q8atnight]

Hardware: 2x RTX 3090 24 GB (NVLink bridge), Ryzen 9 3950X, 121 GB RAM, PCIe 4.0 x8 per card. Qwen3.8-Flash-Next unsloth UD-Q4_K_XL, --layer-split 24, experts in locked RAM, 262K context, int8 KV, spec 4, one stream, v0.1.43.1 + this PR's 3 commits, --conversation-disk-dir on.

Two rows for the thread:

It works on a 2-GPU layer split. Most reports here are single-GPU or unload cases; the disk tier also behaves on a split engine. Same build with conversations kept in RAM vs parked on the disk: word-for-word identical on our exactness gate (10 greedy prompts, a 20K long gate, an image prompt, a 10-question verifiable probe). Resume latency: a 111K-token chat parked on disk came back in ~1.9 s against ~36 s re-reading it.

An observation we cannot explain, from a different box — is this config expected to resume? 2x RTX 5060 Ti 16 GB layer split + RTX 2080 Super for vision, Ryzen 7 3700X, ~32 GB RAM, IQ3_XXS, --resident-experts, 196K context, --conversation-disk-dir plus --session-state-codec bf16 (#2247), conversation cache 4096 → 1024 MiB. Conversation .bin files were written (83–107 MB each), but two alternating ~60K-token chats still re-prefilled on every switch through /v1 (~37 s each) — no resume gain measurable over the plain chat API. Not claiming a bug (the box above resumes fine); if the bf16 codec or this shape is expected to read cold, it would be useful to know which part decides that.

[xrip]

/reopen

[github-actions]

Reopened, thank you. A maintainer will pick it up from here. If it conflicts with main, rebase it first; that speeds up review.

[xrip]

Updated on 2026-10-20. What changed since you last looked:

• Rebased onto 0.2.1 (90767252). It is now two commits: 65a79b51 is the disk tier as before, c6b99976 is new.
• New in c6b99976: a file is read back only when that is faster than reading the tokens it adds, by this run's measured disk and prompt speeds. Before, a new conversation that shared only the system prompt with a saved one read the saved one's whole file: on my box 6,033 MiB in 11.4 s from the NVMe for 16,384 tokens, which take ~8 s to read. A skip is logged as not read. Changes: generate.cpp (+25/-4), conversation_disk.hpp (the match reports its file's bytes and whether it is still queued), one test check, one DETAILS.md bullet.
• New measurements in the description: from a 7200 rpm HDD a 304,364-token conversation came back in 47.6 s (reading it takes ~140 s). When RAM is short, a park waits 28.6-31 s for an evicted conversation's write to the HDD (Limits). The read check fired live: a new sub-agent sharing only the 16,384-token system prompt with a saved 2,018 MiB conversation skipped the file (not read: its 16384 more tokens read in ~9.4 s, its file (2018 MiB) in ~12.5 s).
• Split out: stale copies of a conversation, which the disk tier would otherwise write out, are now dropped by [Engine] conversation cache: drop an older copy of the conversation rewritten turns back (#342 follow-up) #3404 in the conversation cache. That PR does not depend on this one.
• Checks: CUDA build (sm_75); conversation_disk_test 69, conversation_cache_test 4,772, conversation_file_test 201; server tests 392 passed, 3 skipped.

Now ready for review: the same disk-tier code has run live on my box all day (the measurements above), and HIP / SYCL / Metal / Windows stay under "Not run" because I have no environment for them.

[edmondhma]

Thanks for bringing this back to V0.2. Is it possible to write/read from disk directly, without reserving slot in RAM. This is useful when the system RAM is all used for model or other shared task.

[phg1024]

A data point from the SYCL side (Arc Pro B70), plus a port, since the tier is not reachable on the Intel engine as it stands.

The SYCL engine does not get the wiring

conversation_disk.cpp is added to sycl/CMakeLists.txt, but the SYCL build compiles its own migrated sycl/src/program/generate.cpp, not src/program/generate.cpp. strata_resolve() picks the copy under sycl/ when one exists. The 166-line wiring therefore never reaches the Intel engine: --conversation-disk-dir is an unknown option there (grep -c conversation-disk sycl/src/program/generate.cpp → 0 on this branch).

Port

I ported the wiring to sycl/src/program/generate.cpp, on top of this PR's head c6b99976, as one commit:

• branch: https://github.com/phg1024/Strata/tree/sycl-port-1937
• the whole change is one commit on top of c6b99976: phg1024@337e2ba (sycl/src/program/generate.cpp, +159/−16)

It is the same 12 pieces: options, usage, parse, the persist check, the tier and set_on_evict, over-budget parks to the disk, the RAM admission, the restore with the read-vs-recompute check, the prompt-speed sample, the request line, and the save at QUIT. The SYCL copy differs in two places:

• RAM admission. The SYCL park path has no eviction loop: it only checks conversation_memory_admit(available_host_bytes(), ...) and skips. The port keeps that exact check when the tier is off. With the tier on it uses the make_ram loop (trim → wait for queued writes → evict oldest to the disk), built on available_host_bytes(), because the SYCL file has no ram_note/HostMemoryReading call sites.
• No borrow path and no nan_flush clears exist in the SYCL copy, so those hunks have no counterpart.

Measured (SYCL, Arc Pro B70 32 GB, i7-12700K, 61 GiB RAM, NVMe ext4, Ubuntu 26.04, kernel 7.0 xe)

Flash-Next IQ3_S native pack, --stream-experts, --kv int8 --kv-resident 32768, --prompt-cache-every 8192, conversation cache 16 GiB / 2 slots, --conversation-disk-persist on, greedy. Engine main @ 011c9e6 plus this PR plus the port (adaptive swaps are off on this setup):

step	result
160,000-token conversation, turn 1 cold	TTFT 139.8 s
turn 2, live	TTFT 2.4 s
server SIGTERM → QUIT save	saved for the next start in 4.6 s (4,817 MiB at 1,840 MB/s)
next start	1 conversations from earlier runs (4817 MiB)
the same turn 2 again	TTFT 4.3 s: disk hit: 160142 tokens, 4817 MiB in 1 file, 3552 ms (1422 MB/s), restore 433 ms, 7 tokens read, answer byte-identical to the live turn 2
three other 32K conversations, then turn 3	it was evicted to the disk (written in the background) and came back in 5.0 s TTFT (reused from the disk in 4.1 s)
through systemctl --user restart (unit TimeoutStopSec=90)	saved at stop, taken in at start, reused with the same answer

The restore after the restart came from a checkpoint (160,142), not from the saved live state, because the repeated turn 2 diverges from the saved conversation after its user message. So the checkpoint path is exercised, not just the exact-match one.

I re-checked on this same branch (0.2.1 base, c6b99976 plus the port, engine 0.1.46, AOT bmg-g31 build with 0 errors), with a shorter run. A 32,000-token conversation took 24.2 s cold. After a server restart, the same turn came back from the disk in 1.2 s TTFT with a byte-identical answer (saved for the next start in 1.1 s, 1,145 MiB). After three other conversations evicted it, it came back from the disk in 2.1 s.

Merge note for main

Against current main the only conflict is one hunk in serve/server.py: the engine-priority block (engine_nice) that landed next to your quit_s lines. Keeping both sides works.

Happy to re-run on the B70 if the design changes, or to test whichever disk design gets picked (#1090).

[Pallav0099]

Thanks for the disk tier. We run a backport of an earlier revision of this PR (047192e, with the streamed park and restore) on RX 7900 XTX machines. One problem we hit is still present at the current head, and we have a small fix to offer.

Two waits have no beat and no bound (line numbers at c6b9997):

• make_ram()'s conv_disk.wait_idle() (generate.cpp:9299), when a park or a read back's RAM admission needs the RAM of queued writes;
• take()'s wait for an entry that is being written (conversation_disk.cpp:462).

What we saw (on the 0.1.44.1 backport, all on one DRAM-less NVMe whose plain dd rate is ~4,000 MB/s but which drops to tens of MB/s under sustained writes):

• A 4 h agent soak: two 60 s stalls, each ending in a watchdog restart, both parked behind a 2,339 MiB write that took 56.8 s (43 MB/s).
• A 24 h agent soak of the unbounded build: 8 watchdog restarts. 5 of them came while parking, behind queued writes at 60–130 MB/s; one park finished at 62.2 s, 2 s after the watchdog had fired. The other restarts in that soak were unrelated to the tier.
• With a park bound (2000 ms in our build, together with a write gate and a daily write cap that also reduce the write load): a 1 h run on the same drive logged 2 "gave up waiting after 2000 ms … the park is skipped" and 0 stalls. Two 4 h tier-on soaks showed 0 stalls and 0 restarts. In one of them the daily cap engaged after ~1.3 h, so the slow-drive exposure was limited.
• We have not seen the take() wait fire in the field. It is reachable by inspection (a follow-up to a conversation whose park write is still in flight). Its bound is unit-tested, and it is newer and still in testing on our side.

The fix (one commit on c6b9997; the patch is below):

• Both waits beat about every 250 ms (progress_beat), so a slow write no longer looks like a hang to the watchdog.
• Both are bounded, so a write that truly hangs still ends the wait. The defaults are far above what a working drive needs: STRATA_CONV_DISK_WAIT_MS=10000 (park) and STRATA_CONV_DISK_RESTORE_WAIT_MS=60000 (read back). -1 means no limit, beating only. With -1 the waits beat but never end, so a write that truly hangs is no longer caught by the watchdog; that's why the defaults keep a (generous) bound. These could just as well be --conversation-disk-* options if you prefer.
• Past the park bound, make_ram returns false and the outgoing conversation is not parked. That is a later cache miss, not a stall, and nothing is kept in RAM over budget. Past the take bound, take() fails with "still being written", the entry stays with its write, and the request reads the prompt.
• One consequence to know: take() removes the entry before it reads, so if a read back's RAM admission times out, that parked conversation is dropped and the prompt is read.
• On a park timeout the existing "skip parking (physical RAM admission …)" line follows the tier's "gave up waiting … the park is skipped". You may want a distinct message there.
• wait_timeouts / take_timeouts counters. The QUIT save and clear() keep their plain waits.
• conversation_disk_test: +19 checks (88 pass, also under ThreadSanitizer). Compile-checked (HIP, gfx1100, ROCm 7.2.4) with conversation_disk_test passing (88 checks). Not yet run in a full engine on v0.2.1.

Happy to adjust it, split it, or turn it into review suggestions, whichever is easiest for you.

The patch (one commit on c6b9997, git am-able)

From d3cbe74c0644110300a4b9d2ebbdc34e372e8d13 Mon Sep 17 00:00:00 2001
From: Pallav <pallavsworkspace@gmail.com>
Date: Sat, 24 Oct 2026 03:44:45 +0530
Subject: [PATCH] conversation disk: beating, bounded waits for queued writes
 and take()

Two waits on the disk tier have no progress beat and no bound:
- make_ram's wait_idle() (generate.cpp, make_ram), when a park or a read
  back's RAM admission needs the RAM of queued writes;
- take()'s wait for an entry that is being written
  (conversation_disk.cpp, take()).
On a slow drive the request sits in one of them for as long as the queued
writes take. With no beat, the 60 s watchdog restarts the engine.

- Both waits now beat about every 250 ms (progress_beat), so a slow
  write no longer looks like a hang to the watchdog.
- And they are bounded, so a write that truly hangs still ends the wait.
  The defaults are far above what a working drive needs; -1 is no limit
  (beating only):
  - STRATA_CONV_DISK_WAIT_MS (default 10000): a park that needs the RAM
    of queued writes gives up after it, and the outgoing conversation is not parked
    (make_ram returns false: "skip parking"; nothing is kept over budget).
    A read back's RAM admission uses the restore bound; if it times out,
    that entry is gone (take() removes an entry before it reads it) and
    the prompt is read.
  - STRATA_CONV_DISK_RESTORE_WAIT_MS (default 60000): take() gives up
    on an entry still being written ("still being written"); the entry stays with its write
    and the request reads the prompt.
- ConversationDiskTier::wait_idle_for(max_ms, beat),
  ConversationDiskOptions::take_wait_ms + wait_beat; wait_timeouts and
  take_timeouts counters; a log line when a bound is hit.
- The QUIT save and clear() keep their plain waits.

conversation_disk_test: 88 checks (+19): the bounded and the unbounded
waits beat; past a bound the wait gives up and is counted; the entry a
timed-out take() left is intact, is read once written, and is not handed
to a prompt whose prefix changed.
---
 include/strata/core/conversation_disk.hpp |   9 ++
 src/core/conversation_disk.cpp            |  58 +++++++++-
 src/core/conversation_disk_test.cpp       | 134 ++++++++++++++++++++++
 src/program/generate.cpp                  |  30 ++++-
 4 files changed, 226 insertions(+), 5 deletions(-)

diff --git a/include/strata/core/conversation_disk.hpp b/include/strata/core/conversation_disk.hpp
index 568d999..c54b45d 100644
--- a/include/strata/core/conversation_disk.hpp
+++ b/include/strata/core/conversation_disk.hpp
@@ -30,6 +30,11 @@ struct ConversationDiskOptions {
     SessionFileIdentity identity;    ///< persist: the session files' identity (model and settings fingerprints)
     /// Called with every diagnostic line (no trailing newline), from the caller's or the writer's thread.
     std::function<void(const std::string&)> log;
+    /// take() waits at most this long for its entry's write to end (negative: no limit, the default), calling
+    /// `wait_beat` (if set) about every 250 ms meanwhile so a watchdog sees a wait that moves.  Past a bound the take
+    /// fails ("still being written"), the entry is left to its write, and the caller reads the prompt instead.
+    int64_t take_wait_ms = -1;
+    std::function<void()> wait_beat;
     /// Tests: called on the writer thread before each write starts.
     std::function<void()> before_write;
 };
@@ -38,6 +43,7 @@ struct ConversationDiskOptions {
 struct ConversationDiskStats {
     size_t queued = 0, written = 0, write_failed = 0, ram_hits = 0, disk_hits = 0, read_failed = 0;
     size_t evicted = 0, waits = 0;
+    size_t wait_timeouts = 0, take_timeouts = 0;   ///< bounded waits (wait_idle_for, take) that gave up
     uint64_t bytes_written = 0, bytes_read = 0;
     uint64_t tokens_back = 0;                 ///< prompt tokens given back (not read again) by ram or disk hits
     double write_ms = 0, read_ms = 0, wait_ms = 0;
@@ -101,6 +107,9 @@ public:
     bool pending() const;
     /// Blocks until the queue is empty and no write runs; returns the ms it waited.
     double wait_idle();
+    /// Like wait_idle(), but calls `beat` (if set) about every 250 ms while it waits so a watchdog sees a wait that
+    /// moves, and gives up after `max_ms` (negative: no limit).  True when the tier went idle.
+    bool wait_idle_for(int64_t max_ms, const std::function<void()>& beat);
     /// #879: every conversation goes - queued, being written (waited for) and on the disk, files too.  Returns how many.
     size_t clear();
 
diff --git a/src/core/conversation_disk.cpp b/src/core/conversation_disk.cpp
index 513f464..4a45345 100644
--- a/src/core/conversation_disk.cpp
+++ b/src/core/conversation_disk.cpp
@@ -459,7 +459,35 @@ bool ConversationDiskTier::take(uint64_t id, int64_t tokens, SavedConversation&
     double waited = 0;
     if (writing_ && writing_entry_.id == id) {
         const auto w0 = std::chrono::steady_clock::now();
-        cv_.wait(lock, [&] { return !(writing_ && writing_entry_.id == id); });
+        const auto written = [&] { return !(writing_ && writing_entry_.id == id); };
+        const int64_t bound = options_.take_wait_ms;
+        if (bound < 0 && !options_.wait_beat) {
+            cv_.wait(lock, written);
+        } else {
+            // A slow disk held the request here for as long as this entry's write took, with no beat (a watchdog
+            // restart past 60 s).  It beats while it waits; with a bound, past it the caller reads the prompt instead.
+            const auto deadline = w0 + std::chrono::milliseconds(std::max<int64_t>(bound, 0));
+            bool done = written();
+            while (!done) {
+                const auto now = std::chrono::steady_clock::now();
+                if (bound >= 0 && now >= deadline) break;
+                const auto slice = bound < 0 ? std::chrono::steady_clock::duration(std::chrono::milliseconds(250))
+                    : std::min<std::chrono::steady_clock::duration>(deadline - now, std::chrono::milliseconds(250));
+                if (cv_.wait_for(lock, slice, written)) { done = true; break; }
+                if (options_.wait_beat) { lock.unlock(); options_.wait_beat(); lock.lock(); }
+                done = written();
+            }
+            if (!done) {
+                ++stats_.take_timeouts;
+                const double ms = ms_since(w0);
+                lock.unlock();
+                if (options_.wait_beat) options_.wait_beat();
+                error = fmt("still being written: gave up after %.0f ms (bound %lld ms)", ms, (long long) bound);
+                log(fmt("%lld tokens: gave up waiting after %.0f ms for its write (bound %lld ms): the prompt is read",
+                        (long long) tokens, ms, (long long) bound));
+                return false;   // the entry stays with its write and is still there for a later request
+            }
+        }
         waited = ms_since(w0);
     }
     size_t index = entries_.size();
@@ -533,6 +561,34 @@ double ConversationDiskTier::wait_idle() {
     return ms;
 }
 
+bool ConversationDiskTier::wait_idle_for(int64_t max_ms, const std::function<void()>& beat) {
+    if (max_ms < 0 && !beat) { wait_idle(); return true; }
+    std::unique_lock<std::mutex> lock(m_);
+    if (queue_.empty() && !writing_) return true;
+    const auto t0 = std::chrono::steady_clock::now();
+    const auto deadline = t0 + std::chrono::milliseconds(std::max<int64_t>(max_ms, 0));
+    const size_t depth = queue_.size() + (writing_ ? 1 : 0);
+    bool idle = false;
+    for (;;) {
+        const auto now = std::chrono::steady_clock::now();
+        if (max_ms >= 0 && now >= deadline) break;
+        const auto slice = max_ms < 0 ? std::chrono::steady_clock::duration(std::chrono::milliseconds(250))
+            : std::min<std::chrono::steady_clock::duration>(deadline - now, std::chrono::milliseconds(250));
+        if (cv_.wait_for(lock, slice, [&] { return queue_.empty() && !writing_; })) { idle = true; break; }
+        if (beat) { lock.unlock(); beat(); lock.lock(); }
+    }
+    const double ms = ms_since(t0);
+    ++stats_.waits;
+    stats_.wait_ms += ms;
+    if (!idle) ++stats_.wait_timeouts;
+    lock.unlock();
+    if (beat) beat();
+    log(idle ? fmt("waited %.0f ms for %zu queued write%s (their RAM was needed)", ms, depth, depth == 1 ? "" : "s")
+             : fmt("gave up waiting after %.0f ms for %zu queued write%s (bound %lld ms): the park is skipped", ms, depth,
+                   depth == 1 ? "" : "s", (long long) max_ms));
+    return idle;
+}
+
 size_t ConversationDiskTier::size() const {
     std::lock_guard<std::mutex> lock(m_);
     return entries_.size();
diff --git a/src/core/conversation_disk_test.cpp b/src/core/conversation_disk_test.cpp
index 0d028fa..85ac022 100644
--- a/src/core/conversation_disk_test.cpp
+++ b/src/core/conversation_disk_test.cpp
@@ -3,13 +3,16 @@
 #include "strata/core/conversation_disk.hpp"
 
 #include <algorithm>
+#include <chrono>
 #include <cstdio>
 #include <cstdlib>
 #include <filesystem>
 #include <fstream>
 #include <future>
+#include <memory>
 #include <mutex>
 #include <string>
+#include <thread>
 #include <utility>
 #include <vector>
 
@@ -329,6 +332,137 @@ int main() {
         }
     }
 
+    {   // wait_idle_for gives up after its bound and beats meanwhile; the write still ends later
+        std::promise<void> release;
+        std::shared_future<void> gate = release.get_future().share();
+        auto o = opts(dir, GiB);
+        o.before_write = [gate] { gate.wait(); };
+        ConversationDiskTier tier(std::move(o));
+        check(tier.open(e) && tier.put(image(400, 50)), "bounded wait: a write held");
+        int beats = 0;
+        const bool idle = tier.wait_idle_for(600, [&beats] { ++beats; });
+        check(!idle && beats >= 2 && tier.pending(), "bounded wait: gives up after the bound, beating meanwhile");
+        check(tier.stats().wait_timeouts == 1 && logged("gave up waiting"), "bounded wait: counted and logged");
+        release.set_value();
+        check(tier.wait_idle_for(-1, nullptr) && !tier.pending() && tier.size() == 1,
+              "bounded wait: no limit (-1) waits like wait_idle; the held write ended");
+        check(tier.wait_idle_for(10, nullptr), "bounded wait: an idle tier returns at once");
+        tier.clear();
+    }
+
+    {   // take() of an entry still being written gives up after its bound, beating meanwhile
+        std::promise<void> release, entered;
+        std::shared_future<void> gate = release.get_future().share();
+        auto in_write = std::make_shared<std::promise<void>>(std::move(entered));
+        std::future<void> started = in_write->get_future();
+        int beats = 0;
+        auto o = opts(dir, GiB);
+        o.take_wait_ms = 600;
+        o.wait_beat = [&beats] { ++beats; };
+        o.before_write = [gate, in_write] {
+            try { in_write->set_value(); } catch (...) {}   // only the first write signals
+            gate.wait();
+        };
+        ConversationDiskTier tier(std::move(o));
+        check(tier.open(e) && tier.put(image(400, 70, 7000)), "take bound: a write held");
+        started.wait();   // the writer holds the entry: take() finds it being written, not queued
+        const auto m = tier.best(prompt_of(450, 7000), {}, false);
+        check(m.tokens == 400 && m.id != 0, "take bound: the entry being written still matches");
+        SavedConversation back;
+        const auto t0 = std::chrono::steady_clock::now();
+        const bool ok = tier.take(m.id, m.tokens, back, e);
+        const double ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count();
+        check(!ok && e.find("still being written") != std::string::npos, "take bound: fails as still being written");
+        check(ms >= 550 && ms < 5000 && beats >= 2, "take bound: gives up after the bound, beating meanwhile");
+        check(tier.stats().take_timeouts == 1 && logged("gave up waiting after") && logged("for its write"),
+              "take bound: counted and logged");
+        release.set_value();
+        check(tier.wait_idle_for(-1, nullptr) && tier.size() == 1, "take bound: the held write still ends, stored");
+        check(tier.best(prompt_of(450, 7001), {}, false).tokens == 0,
+              "take bound: the late-written entry is not handed to a prompt whose prefix changed");
+        check(tier.take(m.id, m.tokens, back, e) && same(back, image(400, 70, 7000)),
+              "take bound: the entry is intact and taken once written");
+        tier.clear();
+    }
+
+    {   // within the bound, take() waits for the write and reads the entry as before
+        std::promise<void> release, entered;
+        std::shared_future<void> gate = release.get_future().share();
+        auto in_write = std::make_shared<std::promise<void>>(std::move(entered));
+        std::future<void> started = in_write->get_future();
+        auto o = opts(dir, GiB);
+        o.take_wait_ms = 30000;
+        o.wait_beat = [] {};
+        o.before_write = [gate, in_write] {
+            try { in_write->set_value(); } catch (...) {}
+            gate.wait();
+        };
+        ConversationDiskTier tier(std::move(o));
+        check(tier.open(e) && tier.put(image(400, 71, 8000)), "take wait: a write held");
+        started.wait();
+        const auto m = tier.best(prompt_of(450, 8000), {}, false);
+        std::thread later([&release] {
+            std::this_thread::sleep_for(std::chrono::milliseconds(300));
+            release.set_value();
+        });
+        SavedConversation back;
+        check(tier.take(m.id, m.tokens, back, e) && same(back, image(400, 71, 8000)) &&
+                  tier.stats().take_timeouts == 0,
+              "take wait: the write ends within the bound and the entry is read");
+        later.join();
+        tier.clear();
+    }
+
+    {   // no bound (-1): wait_idle_for and take() wait as long as the write takes, but beat meanwhile
+        std::promise<void> release, entered;
+        std::shared_future<void> gate = release.get_future().share();
+        auto in_write = std::make_shared<std::promise<void>>(std::move(entered));
+        std::future<void> started = in_write->get_future();
+        int take_beats = 0;
+        auto o = opts(dir, GiB);
+        o.wait_beat = [&take_beats] { ++take_beats; };   // take_wait_ms stays -1
+        o.before_write = [gate, in_write] {
+            try { in_write->set_value(); } catch (...) {}
+            gate.wait();
+        };
+        ConversationDiskTier tier(std::move(o));
+        check(tier.open(e) && tier.put(image(400, 72, 9000)), "unbounded: a write held");
+        started.wait();
+        int beats = 0;
+        std::thread later([&release] {
+            std::this_thread::sleep_for(std::chrono::milliseconds(700));
+            release.set_value();
+        });
+        check(tier.wait_idle_for(-1, [&beats] { ++beats; }) && beats >= 2 && tier.stats().wait_timeouts == 0,
+              "unbounded: wait_idle_for(-1, beat) waits for the write, beating meanwhile");
+        later.join();
+        tier.clear();
+        std::promise<void> release2, entered2;
+        std::shared_future<void> gate2 = release2.get_future().share();
+        auto in_write2 = std::make_shared<std::promise<void>>(std::move(entered2));
+        std::future<void> started2 = in_write2->get_future();
+        auto o2 = opts(dir, GiB);
+        o2.wait_beat = [&take_beats] { ++take_beats; };
+        o2.before_write = [gate2, in_write2] {
+            try { in_write2->set_value(); } catch (...) {}
+            gate2.wait();
+        };
+        ConversationDiskTier tier2(std::move(o2));
+        check(tier2.open(e) && tier2.put(image(400, 73, 9500)), "unbounded take: a write held");
+        started2.wait();
+        const auto m = tier2.best(prompt_of(450, 9500), {}, false);
+        std::thread later2([&release2] {
+            std::this_thread::sleep_for(std::chrono::milliseconds(700));
+            release2.set_value();
+        });
+        SavedConversation back;
+        check(tier2.take(m.id, m.tokens, back, e) && same(back, image(400, 73, 9500)) && take_beats >= 2 &&
+                  tier2.stats().take_timeouts == 0,
+              "unbounded take: waits for the write, beating meanwhile, then reads the entry");
+        later2.join();
+        tier2.clear();
+    }
+
     fs::remove_all(dir);
     std::printf("conversation_disk_test: %d checks passed\n", checks);
     return 0;
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index d7d2811..bc35355 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -9251,6 +9251,23 @@ int main(int argc, char** argv) {
         disk_opts.dir = conversations.enabled() ? o.conversation_disk_dir : std::string();
         disk_opts.budget_bytes = (uint64_t) o.conversation_disk_mib << 20;
         disk_opts.min_free_bytes = (uint64_t) o.session_min_free_mib << 20;
+        // The disk tier's waits beat, so a slow drive no longer looks like a hang to the watchdog, and are bounded, so
+        // a write that truly hangs still ends the wait: a park that needs the RAM of queued writes waits at most
+        // STRATA_CONV_DISK_WAIT_MS (then the outgoing conversation is not parked: a later cache miss, not a stall), a
+        // read back of an entry still being written at most STRATA_CONV_DISK_RESTORE_WAIT_MS (then the prompt is
+        // read).  The defaults are far above what a working drive needs; -1: no limit (beating only).
+        auto env_i64 = [](const char* name, int64_t fallback) -> int64_t {
+            const char* v = std::getenv(name);
+            if (!v || !*v) return fallback;
+            char* end = nullptr;
+            const long long x = std::strtoll(v, &end, 10);
+            return (end && *end == '\0') ? (int64_t) x : fallback;
+        };
+        const int64_t disk_wait_ms = env_i64("STRATA_CONV_DISK_WAIT_MS", 10000);
+        // a restore that gives up reads the whole prompt again, so give it a longer bound than a park
+        const int64_t disk_restore_wait_ms = env_i64("STRATA_CONV_DISK_RESTORE_WAIT_MS", 60000);
+        disk_opts.take_wait_ms = disk_restore_wait_ms;
+        disk_opts.wait_beat = [] { strata::core::progress_beat(); };
         if (o.conversation_disk_persist != "off" && !disk_opts.dir.empty()) {
             // the session files' identity: a later start with the same model and settings takes the files in
             std::string id_err;
@@ -9288,7 +9305,7 @@ int main(int argc, char** argv) {
 #endif
         };
         auto make_ram = [&](uint64_t need, uint64_t floor, size_t& evicted,
-                            std::optional<strata::core::HostMemoryReading>& mem) -> bool {
+                            std::optional<strata::core::HostMemoryReading>& mem, int64_t wait_ms) -> bool {
             bool trimmed = false;
             for (;;) {
                 mem = strata::core::sample_host_memory();
@@ -9296,7 +9313,12 @@ int main(int argc, char** argv) {
                                                             need, floor))
                     return true;
                 if (!trimmed) { return_freed_heap(); trimmed = true; continue; }
-                if (conv_disk.pending()) { conv_disk.wait_idle(); trimmed = false; continue; }
+                if (conv_disk.pending()) {
+                    // beating and bounded: a slow disk held the request here as long as its queued writes took
+                    if (!conv_disk.wait_idle_for(wait_ms, [] { strata::core::progress_beat(); })) return false;
+                    trimmed = false;
+                    continue;
+                }
                 if (conversations.size() == 0 || evicted >= conversations.slots() || !conversations.evict_oldest())
                     return false;
                 ++evicted;
@@ -9419,7 +9441,7 @@ int main(int argc, char** argv) {
                 // (MemAvailable, the cgroup limits above the engine, --memory-limit-mib).
                 size_t evicted = 0;
                 std::optional<strata::core::HostMemoryReading> mem;
-                if (!make_ram(additional, floor, evicted, mem)) {
+                if (!make_ram(additional, floor, evicted, mem, disk_wait_ms)) {
                     std::fprintf(stderr, "strata serve: conversation cache: skip parking (physical RAM admission; need %zu MiB plus %lld MiB floor, %s; evicted %zu, %zu still parked)\n",
                                  additional >> 20, (long long) o.conversation_cache_min_free_mib, ram_note(mem).c_str(),
                                  evicted, conversations.size());
@@ -12522,7 +12544,7 @@ int main(int argc, char** argv) {
                     size_t evicted = 0;
                     auto admit = [&](uint64_t need, std::string& why) -> bool {
                         std::optional<strata::core::HostMemoryReading> mem;
-                        if (make_ram(need, floor, evicted, mem)) return true;
+                        if (make_ram(need, floor, evicted, mem, disk_restore_wait_ms)) return true;
                         why = "not enough RAM: " + ram_note(mem);
                         return false;
                     };
-- 
2.56.0

[G0-0000]

I have a follow-up that I would like to scope before opening a PR: versioned codecs for state stored in disk session snapshots.

The local implementation has two opt-in pieces:

• REF for prefix snapshots: when checkpoint state arrays are byte-identical to the live arrays, store one copy plus a reference. This is lossless.
• RHT+INT16 for ordinary base and delta records: encode GDN recurrent rows with a normalized RHT, per-row INT16 values, and FP32 scales. Keep convolution state RAW and runtime state FP32. Encode while writing, rather than writing an FP32 snapshot and converting it afterwards.

The records carry explicit codec descriptors; existing v1 files remain readable, and the default behavior is unchanged. Prefix snapshots are excluded from RHT. I have host round-trip/compatibility tests and a same-version restart restore test on a V100.

The RHT piece also covers delta records, so it needs an agreed delta representation as a prerequisite. Would you prefer this as one follow-up PR after the disk design and delta format are settled, or as two PRs (REF first, then RHT)? If useful, I can prepare a stacked draft, but I do not want to duplicate the base disk-tier work here.

[Pallav0099]

I have a follow-up that I would like to scope before opening a PR: versioned codecs for state stored in disk session snapshots.

The local implementation has two opt-in pieces:

* **REF for prefix snapshots:** when checkpoint state arrays are byte-identical to the live arrays, store one copy plus a reference. This is lossless.

* **RHT+INT16 for ordinary base and delta records:** encode GDN recurrent rows with a normalized RHT, per-row INT16 values, and FP32 scales. Keep convolution state RAW and runtime state FP32. Encode while writing, rather than writing an FP32 snapshot and converting it afterwards.

The records carry explicit codec descriptors; existing v1 files remain readable, and the default behavior is unchanged. Prefix snapshots are excluded from RHT. I have host round-trip/compatibility tests and a same-version restart restore test on a V100.

The RHT piece also covers delta records, so it needs an agreed delta representation as a prerequisite. Would you prefer this as one follow-up PR after the disk design and delta format are settled, or as two PRs (REF first, then RHT)? If useful, I can prepare a stacked draft, but I do not want to duplicate the base disk-tier work here.

Thanks, this helps with scoping. My numbers: RX 7900 XTX (HIP), --kv int8, 786 disk writes from three ~20 h agent soaks (median ~118K tokens). File size fits ~350-550 MiB fixed + ~17-19 KiB/token, so ~2.4-2.9 GB at 118K. The fixed part is 14-22% of bytes; the rest is int8 KV, which neither codec touches. That share caps what REF can save for us.

REF is the half I’d adopt, since it’s lossless. Two PRs, REF first, would be easiest to test. Two questions:

Is the fixed part the recurrent state plus checkpoint copies? On your workload, how many checkpoints per record does REF dedupe, and what are the bytes per record before and after?
Must RHT+INT16 reproduce the never-saved continuation exactly (greedy, same expert placement), or only closely? We compare restored answers exactly, so we’d keep it off unless it does.

I'll run a REF draft on HIP and report bytes per park, write time and restore equality.

One request: the codec write path uses the queued writes behind the two unbounded waits I raised (make_ram’s wait_idle(), take()’s in-flight wait; [link]). I know dropping queued writes was tried and reverted in 3fe88e6, so I’m not asking for that. Could the codec work stay inside a bounded wait, or leave room for the fix I offered?

[xrip]

@Pallav0099 thank you for the patch, the soak numbers and the clear write-up.

The maintainer has not yet chosen which disk design, if any, goes into the main branch. This PR is one of several, and it has had no review since 0.2.0. Until that choice is made, I don't think it makes sense to keep working on this PR or to take patches into it: work on a design that may never be merged is likely wasted, for you and for me. The same goes for follow-ups built on top of it, such as the snapshot codecs above.

If the maintainer picks this design, your bounded waits will be the first thing I look at. Until then, your patch stays here in the thread, and keeping it in your backport is the right place for it.

[G0-0000]

@Pallav0099 — thanks. Two answers, and one correction on where our codec work lives.

Where it lives: the codecs are not stacked on this PR. Our disk tier is an integration of #1271, #1269 and #1480 (plus the delta/LRU work), carried in #4226. So your REF draft here and our work there are independent of the design choice xrip described.

1. Fixed part and REF dedupe. Yes — the fixed part is the per-checkpoint running state (five GDN recurrent arrays plus the RAW convolution state); the int8 K/V grows with tokens and is what REF cannot touch. In our largest prefix sample (Flash-Next UD-Q4_K_XL, int8 K/V, V100): 278,223,720 B without REF → 160,160,728 B with REF. The 118,062,992 B saved is exactly one complete state copy (117,669,888 B, convolution included). Caveat: only the REF file was measured; the no-REF side is format arithmetic. REF only dedupes checkpoint arrays byte-identical to the live copy, so your 14–22% fixed share is the right ceiling — if your checkpoints differ from live even slightly, it saves less. We did not count matches per record beyond this sample.

2. RHT+INT16 is not exact. We have state-fidelity numbers only: relative L2 ≈ 1.9e-5 for RHT+INT16 vs ≈ 2.1e-4 for FP16 over 141,557,760 real FP32 values. We have no greedy, same-expert-placement A/B of restored answers, so by your test (exact equality) keep it off — we would too.

On the bounded waits: the 250 ms beat plus a bound is a good rule beyond this PR. Our disk-only save joins a background writer at a cache boundary with no bound (DiskSaveJob in generate.cpp), so a slow drive can block the request thread the same way. We are not porting your patch — the implementations differ — but if we add a bound we will follow the same rule.

[jaylfc]

@xrip thanks for this, we run it daily (Swift 1.5 IQ3_XXS on a 3090). We rebased it onto v0.2.7.1 for our own use; the conflicts were the park path in generate.cpp (vs the new ParkSource), quit_s vs engine_nice in server.py, and the conversation_cache.hpp include. If it saves you time, the resolved branch is https://github.com/jaylfc/strata-12gb/tree/port-1937-0271 (2 commits on v0.2.7.1, conversation_disk_test 69 checks pass). After a restart, a continuation reused 11524 of 11623 tokens from the disk: prompt 1.3 s vs 6.6 s cold.