Repository navigation
[Engine] --batch: parking an idle slot (#2526) drops the copy with the shared system-prompt checkpoint, so the next agent request re-reads 5K tokens #4420
Description
Area
Engine (the server's conversation cache with --batch)
Desktop app, Terminal or source?
Source build in Docker (upstream v0.2.7 + local patches, see Strata version), serve/server.py with strata --serve.
What happened
Our voice assistant starts every request with the same 5,156-token prefix (system prompt + 18 tool definitions) and adds a
short question. Since 0.2.7, when its requests overlap another user's answer (--batch 2), the cached prefix gets lost:
the third overlapping request of our probe reads the whole prompt again (prompt 5173 tokens = 0 reused + 5173 read in 4130 ms), where 0.2.6 on the same box borrows it (borrowed 5156 tokens from a parked conversation and 5156 reused + 17 read in 323 ms). Requests sent alone are not affected: after the first one they reuse 5156 tokens on both versions.
Expected: the shared prefix stays borrowable, as in 0.2.6.
| build | server starts | 8th agent request (3rd overlapping) | wall time of that request |
|---|---|---|---|
| 0.2.6 (engine 0.1.51) | 3 | 5156 reused + 17 read, 323-400 ms | 2.43-3.06 s |
| 0.2.7 (engine 0.1.52) | 3 | 0 reused + 5173 read, 4130-4172 ms | 6.12-6.46 s |
| 0.2.7 + the change below | 2 | 5156 reused + 17 read, 322-344 ms | 2.56-2.95 s |
Same box and config, alternating server starts. One of the three 0.2.7 starts ran without our --pcie-frac 0.38 (the
engine's own split); everything else was identical.
Smallest steps or command that reproduce it
Server (the relevant arguments): --serve --batch 2 --batch-mtp --mtp <dir> --spec 4 --spec-min-p 0.5 --kv q4_0 --max-context 524288 --conversation-cache-mib 1024 --conversation-cache-slots 6. One client process, every request
temperature 0, reasoning_effort none:
- Agent request: a system message (system prompt + tool instructions),
toolswith 18 functions, one user question,
max_tokens 120; 5,156 tokens come before the question. Send it 5 times alone, a different question each time. - Three rounds: start a long answer in another conversation (one 38-token user message,
max_tokens 600,
"strata_checkpoint": false); 3 s later send the agent request with a new question; wait for both. - Round 3's agent request reads all 5,173 tokens: 3 of 3 starts on 0.2.7, 0 of 3 on 0.2.6.
The prefix is our assistant's private prompt and is not attached. We have only run the probe with it.
What the log shows
0.2.7, from round 2's agent request to round 3's:
strata serve: prompt 5175 tokens = 5156 reused + 19 read in 357 ms (53.2 tok/s), 1 generated in 29 ms (33.9 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: conversation cache: parked 5175 tokens in 160.9 ms; parked=4 bytes=928080148 evictions=11 snapshot_bytes=400964236 reused_kv_bytes=0
strata batch: slot 0 gave back 199 tokens of this conversation (all it holds) in 90.3 ms
strata serve: prompt 200 tokens = 199 reused + 1 read in 266 ms (3.8 tok/s), 369 generated in 7166 ms (51.5 tok/s), drafts accepted 194 of 316, 0 checkpoints
strata serve: prompt 38 tokens = 0 reused + 38 read in 479 ms (79.3 tok/s), 125 generated in 2551 ms (49.0 tok/s), drafts accepted 61 of 108, 0 checkpoints
strata serve: conversation cache: dropped 1 superseded copy of this conversation; parked=3
strata serve: conversation cache: parked 5206 tokens from slot 1 in 61.2 ms; parked=4 bytes=810285000 evictions=11 snapshot_bytes=283169088 reused_kv_bytes=0
strata batch: slot 1 takes 163 tokens (copied in 169.5 ms)
strata serve: prompt 163 tokens = 0 reused + 163 read in 974 ms (167.4 tok/s), 1 generated in 26 ms (38.6 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: conversation cache: parked 199 tokens from slot 0 in 106.2 ms; parked=5 bytes=930157780 evictions=11 snapshot_bytes=119872780 reused_kv_bytes=0
strata batch: slot 0 takes 5173 tokens (copied in 245.5 ms)
strata serve: prompt 5173 tokens = 0 reused + 5173 read in 4130 ms (1252.7 tok/s), 1 generated in 36 ms (28.0 tok/s), drafts accepted 0 of 0, 2 checkpoints
Round 2's agent conversation is read in the main session (2 checkpoints), decoded in slot 1, and its main-session copy is
parked (5175 tokens). In round 3 the long answer goes to slot 1: park_slot(1) parks what slot 1 holds (round 2's agent
conversation with its reply, 5206 tokens) and drops the 5175-token copy as superseded. Nothing parked then has the
5156-token checkpoint, and round 3's agent request reads everything.
0.2.6, same point:
strata serve: prompt 5175 tokens = 5156 reused + 19 read in 349 ms (54.4 tok/s), 1 generated in 29 ms (35.1 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: conversation cache: parked 5175 tokens in 150.0 ms; parked=2 bytes=801891680 evictions=11 snapshot_bytes=400963908 reused_kv_bytes=0
strata batch: slot 0 gave back 184 tokens of this conversation (all it holds) in 105.9 ms
strata serve: prompt 185 tokens = 184 reused + 1 read in 276 ms (3.6 tok/s), 364 generated in 7378 ms (49.3 tok/s), drafts accepted 190 of 325, 0 checkpoints
strata serve: prompt 38 tokens = 0 reused + 38 read in 480 ms (79.1 tok/s), 123 generated in 2562 ms (48.0 tok/s), drafts accepted 62 of 114, 0 checkpoints
strata batch: slot 1 takes 161 tokens (copied in 82.6 ms)
strata serve: prompt 161 tokens = 0 reused + 161 read in 975 ms (165.1 tok/s), 1 generated in 29 ms (34.5 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: conversation cache: borrowed 5156 tokens from a parked conversation, which stays parked, in 41.8 ms; parked=2 bytes=801891680 borrowed=3
strata batch: slot 0 takes 5173 tokens (copied in 151.9 ms)
strata serve: prompt 5173 tokens = 5156 reused + 17 read in 323 ms (52.7 tok/s), 1 generated in 45 ms (22.2 tok/s), drafts accepted 0 of 0, 2 checkpoints
Why (our reading of the v0.2.7 code; not checked beyond the runs above)
- At a slot admission the slot's chain gets one checkpoint, the deepest one that is a strict prefix of the live tokens
(src/program/generate.cpparound L17102,sl.checks.push_back(*best)): for an agent request its turn checkpoint, not
the one at the end of the shared prefix that the next requests borrow. park_slot(L12603) goes throughpark_source, which callsconversations.drop_superseded(src.ids, src.imgs, src.checks, src.cvec)(L10470), andConversationCache::put()calls it again (conversation_cache.hppL493).
drop_superseded(Conversation cache slots are consumed by successive turns of the same subagent, evicting the parent conversation #342, L442) drops a parked entry when the outgoing chain holds that entry's deepest checkpoint. The
main-session copy of the same conversation qualifies, and its checkpoint at the end of the shared prefix goes with it.
The rule assumes the outgoing chain also carries the older checkpoints, which is true for the main session's chain but
not for a slot's.- The slot images also take room. With
--conversation-cache-mib 1024, an agent conversation's main copy is 401 MB, a
slot image 283 MB (agent) or 120-124 MB (long answer). With them parked the cache holds one main copy where 0.2.6 holds
two (parked=2 bytes=801891680above), so the older copies with the shared checkpoint had been evicted, and the
dropped one was the last. This part is reconstructed from theparked=,bytes=andevictions=fields; the log does
not say which entry an eviction removes.
The change we run
ConversationCache::supersedes_richer(ids, images, checkpoints, cvec) is true when drop_superseded with that chain
would drop an entry that has a checkpoint the chain does not hold. park_slot then returns early and says so; the slot is
overwritten as in 0.2.6, so a continuation of that conversation reads its reply again (about 30 tokens here). A slot with
no such copy (the long answers here) is still parked. With it, at the same point:
strata serve: prompt 5175 tokens = 5156 reused + 19 read in 327 ms (58.1 tok/s), 1 generated in 30 ms (33.8 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: conversation cache: parked 5175 tokens in 129.9 ms; parked=3 bytes=921446256 evictions=12 snapshot_bytes=400964236 reused_kv_bytes=0
strata batch: slot 0 gave back 188 tokens of this conversation (all it holds) in 118.8 ms
strata serve: prompt 189 tokens = 188 reused + 1 read in 258 ms (3.9 tok/s), 365 generated in 7397 ms (49.3 tok/s), drafts accepted 185 of 317, 0 checkpoints
strata serve: prompt 38 tokens = 0 reused + 38 read in 479 ms (79.4 tok/s), 131 generated in 2527 ms (51.8 tok/s), drafts accepted 69 of 118, 0 checkpoints
strata serve: conversation cache: slot 1 not parked before its overwrite (a parked copy of its conversation holds checkpoints the slot does not; it stays)
strata batch: slot 1 takes 169 tokens (copied in 132.7 ms)
strata serve: prompt 169 tokens = 0 reused + 169 read in 991 ms (170.5 tok/s), 1 generated in 29 ms (34.6 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: conversation cache: borrowed 5156 tokens from a parked conversation, which stays parked, in 44.6 ms; parked=3 bytes=921446256 borrowed=3
strata serve: conversation cache: parked 188 tokens from slot 0 in 51.0 ms; parked=4 bytes=1041216848 evictions=12 snapshot_bytes=119770592 reused_kv_bytes=0
strata batch: slot 0 takes 5173 tokens (copied in 223.1 ms)
strata serve: prompt 5173 tokens = 5156 reused + 17 read in 322 ms (52.7 tok/s), 1 generated in 28 ms (35.6 tok/s), drafts accepted 0 of 0, 2 checkpoints
Patch against v0.2.7 (3 files, +59)
diff --git a/include/strata/core/conversation_cache.hpp b/include/strata/core/conversation_cache.hpp
index 34898345..a4f03894 100644
--- a/include/strata/core/conversation_cache.hpp
+++ b/include/strata/core/conversation_cache.hpp
@@ -465,6 +465,33 @@ public:
superseded_ += dropped;
return dropped;
}
+ // A batch slot's chain holds one checkpoint (the one its admission copied in), while the main session's parked copy
+ // of the same conversation keeps the whole chain - the root checkpoint at the end of an agent's system prompt and
+ // tools among them, the one its next conversations borrow (borrows()). Parking the slot (#2526) would drop that
+ // copy as superseded (#342: the slot's chain holds its deepest checkpoint) and the root with it: every sibling then
+ // reads the shared prefix again. True when drop_superseded(ids, images, checkpoints, cvec) would drop an entry that
+ // has a checkpoint the given chain does not hold.
+ bool supersedes_richer(const std::vector<int32_t>& ids, const std::vector<ConversationImageKey>& images,
+ const std::vector<ConversationCheckpoint>& checkpoints, bool cvec) const {
+ auto held = [&](const ConversationCheckpoint& c) {
+ if (c.ids == ids && c.imgs == images) return true;
+ for (const auto& k : checkpoints)
+ if (k.ids == c.ids && k.imgs == c.imgs) return true;
+ return false;
+ };
+ for (size_t i = 0; i < entries_.size(); ++i) {
+ const auto& e = entries_[i];
+ const ConversationCheckpoint* deepest = nullptr;
+ for (const auto& c : e.checkpoints)
+ if (!deepest || c.ids.size() > deepest->ids.size()) deepest = &c;
+ if (e.cvec != cvec || !deepest || deepest->ids.empty() || !held(*deepest) ||
+ (guarded_ != 0 && serials_[i] == guarded_))
+ continue;
+ for (const auto& c : e.checkpoints)
+ if (!held(c)) return true;
+ }
+ return false;
+ }
size_t superseded() const { return superseded_; }
/// --conversation-cache-park-gain N: the longest parked entry this conversation's outgoing state supersedes, in
/// tokens, 0 when it has none. It is the entry drop_superseded() would remove, so the caller can tell whether a
diff --git a/src/core/conversation_cache_test.cpp b/src/core/conversation_cache_test.cpp
index 242e3760..6c2aec46 100644
--- a/src/core/conversation_cache_test.cpp
+++ b/src/core/conversation_cache_test.cpp
@@ -720,5 +720,30 @@ int main() {
check(tiny.put(std::move(p2)), "a pinned one fits an empty cache");
check(!tiny.put(image({7,8,9})) && tiny.size() == 1, "nothing else fits beside it: parking is refused, the pin stays");
}
+ {
+ // a batch slot's chain holds one checkpoint (its admission's): parking it must not drop (#342) the main
+ // session's copy of the same conversation that also holds the root a sibling borrows from (an agent's system
+ // prompt and tools) - supersedes_richer() names that case, so the slot is not parked
+ auto cp = [](std::vector<int32_t> ids) { ConversationCheckpoint c; c.ids = std::move(ids); return c; };
+ SavedConversation main_copy = image({1, 2, 3, 4, 10, 11, 12});
+ main_copy.checkpoints = {cp({1, 2, 3, 4}), cp({1, 2, 3, 4, 10, 11})};
+ ConversationCache cache(1 << 20, 4);
+ check(cache.put(std::move(main_copy)), "park the main session's copy with its root and turn checkpoints");
+ const std::vector<int32_t> slot_ids{1, 2, 3, 4, 10, 11, 12, 20, 21}; // the slot decoded the reply
+ const std::vector<ConversationCheckpoint> slot_chain{cp({1, 2, 3, 4, 10, 11})}; // the admission's only one
+ check(cache.supersedes_richer(slot_ids, {}, slot_chain, true), "the slot's park would drop the root checkpoint");
+ const std::vector<ConversationCheckpoint> full_chain{cp({1, 2, 3, 4}), cp({1, 2, 3, 4, 10, 11})};
+ check(!cache.supersedes_richer(slot_ids, {}, full_chain, true), "a chain that holds both loses nothing");
+ check(!cache.supersedes_richer(slot_ids, {}, slot_chain, false), "steering mode is isolated");
+ check(!cache.supersedes_richer(std::vector<int32_t>{1, 2, 3, 4, 30, 31}, {},
+ std::vector<ConversationCheckpoint>{cp({1, 2, 3, 4, 30})}, true),
+ "another conversation on the same root supersedes nothing");
+ check(cache.size() == 1 && cache.superseded() == 0, "the predicate changes nothing");
+ const auto sib = cache.best(std::vector<int64_t>{1, 2, 3, 4, 40}, {}, true);
+ check(sib.tokens == 4 && cache.borrows(sib), "a sibling still borrows the root");
+ // what drop_superseded does with the slot's chain (the park this rule prevents): the root goes
+ check(cache.drop_superseded(slot_ids, {}, slot_chain, true) == 1 && cache.size() == 0,
+ "the slot's chain supersedes the richer copy");
+ }
std::printf("conversation_cache_test: %d checks passed\n", checks);
}
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 824bd19d..644232f8 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -12609,6 +12609,13 @@ int main(int argc, char** argv) {
const bool in_main = live_ok && cvec_cached == sl.cvec && prefix_of(live) &&
imgs_below(live_imgs, (int64_t) sl.ids.size()).empty();
if (in_main || conversations.holds(sl.ids, {}, sl.cvec)) return;
+ if (conversations.supersedes_richer(sl.ids, {}, sl.checks, sl.cvec)) {
+ // its park would drop (as superseded) the main session's copy of this conversation, and with it the
+ // checkpoints the slot does not hold - an agent's system prompt root that the next requests borrow
+ std::fprintf(stderr, "strata serve: conversation cache: slot %d not parked before its overwrite (a parked "
+ "copy of its conversation holds checkpoints the slot does not; it stays)\n", b);
+ return;
+ }
if (use_mtp && !batch_mtp) { // no draft K/V of its own: an image without it would not restore beside the drafter
std::fprintf(stderr, "strata serve: conversation cache: slot %d not parked before its overwrite (its %zu tokens "
"have no per-slot draft K/V without --batch-mtp)\n", b, sl.ids.size());Ways that would keep the reply as well, not tried: copy the superseded entry's missing checkpoints into the slot's image,
or give a slot the conversation's whole chain at admission. sycl/src/program/generate.cpp has the same park_slot; the
patch does not touch it (we have no SYCL device).
Tests:
- Passed, on our tree (v0.2.7 + the local patches below; g++ 13 for the standalone tests, HIP build of
strata):
conversation_cache_test(4842 checks, with the new case),conversation_split_failure_test,
conversation_memory_test,conversation_file_test,session_save_reclaim_test,ple_residency_test,
arch_defaults_test;python -m unittestofserve.test_frontend,serve.test_answer_speed,serve.test_family_name,
serve.test_responses,serve.test_vision_elastic,serve.test_vision_elastic_http,serve.test_vision_start_error,
serve.test_cache_drop. The patch applies to plain v0.2.7 (git apply --cached --check); not built there. - Hardware: the 2 starts in the table, 6 of 6 overlapping agent requests reused 5156 tokens. Our A/B of the same starts
(4K prompt and decode, long answers, a 100K-token document 6/6, two users, pictures 4/4) stayed within run-to-run noise
of 0.2.6 and 0.2.7. - Not run: CUDA, SYCL, other agent prefixes or cache budgets.
Strata version
Strata 0.2.7 · engine 0.1.52: upstream v0.2.7 (3f6d9c28) + local patches (a conversation disk tier that only takes
conversations of 16,384+ tokens and did not read or write during these probes, extra /v1/models fields, an elastic CPU
image encoder, an opt-in Q4_0-KV WMMA prompt attention, a gfx1200 hipBLASLt table). The 0.2.6 runs: 0.2.6 · engine
0.1.51 with the same local patches.
Graphics card, memory and system
RX 9060 XT 16 GB (gfx1200, ROCm 7.2.3) · Core i7-12700H · 62 GB RAM, container limited to 52 GiB · Debian 12, kernel 6.18
Model, size and settings
Qwen3.8-Flash-Next IQ2_XS · 512K context · pictures on (CPU encoder) · --batch 2 --batch-mtp, MTP drafts
(--spec 4), --kv q4_0, conversation cache 1024 MiB / 6 slots
Reproduction status
Reproduced locally
Activity
Niko1221 commentedon Oct 29, 2026
Your reading was right, and
supersedes_richer()is the smallest change, so the fix carries it with your test (credited in the commit). Parking an idle--batchslot dropped the main session's copy with the shared system-prompt checkpoint, so the next agent request read the 6,028-token prefix again. Fixed on our side; the fix ships in the next update (0.2.8).On one RTX A6000 (Qwen IQ3_S,
--batch 2, a 6,052-token prefix) the time to first token from round 2 on was 3.3 s before and 0.66 s after. The trade-off you described stays: the slot's own reply is read again if that conversation comes back. Keeping the missing checkpoints (100+ MB each) is left open.drrck82 commentedon Oct 29, 2026
Same failure here on 0.2.7 / engine 0.1.52, but it loses a whole conversation's copy, not just a shared prefix.
Rig. Linux, 2x RTX 3090 (layer split auto: CUDA0 layers 0-21, CUDA1 22-47 + head/draft),
Qwen3.8-Flash-Next GSQ-RCO IQ3_S +
--mtp, engine 0.1.52 (tag v0.2.7,5e8a4cca).Relevant server args:
--batch 4 --batch-mtp --batch-groups auto --kv int8 --kv-resident 32768 --max-context 262144 --conversation-cache-mib 24576 --conversation-cache-slots 8 --conversation-cache-min-tokens 8192 --conversation-cache-min-free-mib 4096,--prompt-cacheand--prompt-cache-everyat their defaults (6 / 16384).STRATA_PIPED_MTP=1in the environment. Client: Claude Code, 8 sessions at once, byte-identicalappend-only prompts (
--batch-groups autoresolves to 2 groups of 2).What it costs. 43 turns since the last restart: 31 full hits (median prefill 0.65 s), 10 turns
restoring a stale checkpoint (median 4.3 s, 52 s in total), 2 cold.
evictions=0throughout and thecache sat at 7.1 GiB of its 24.5 GiB budget, so nothing was evicted for space.
Two conversations alternate, and the parked copy does not survive the other one's park.
Conversation A is at ~66 K, B at ~95 K. Note
parkedstays at 1 although both are served:A parks at 66,320 / 66,474 / 66,576 and B is restored live each time -- yet A's own next turn
restores 54,980, 11.5 K tokens behind the copy it had parked one turn earlier:
A second conversation does the same at 71,364 (5 turns, 4,237-4,539 tokens re-read, 3.5-4.1 s each),
plus one 19,543-token re-read (8.7 s).
Both restore points are on the
--prompt-cache-everygrid (16384): 54980 and 71364 are exactly16,384 apart (
mod 16384 = 5828for both). The previous session on this box showed the same pair at30,404 and 46,788 -- also exactly 16,384 apart. So the restore lands on a periodic checkpoint, not on
the turn-boundary state. Per #2228's diagnosis, the prompt is a pure append, so the deeper state
should have matched; the sibling conversation's grid checkpoint is winning because the requester's own
copy is gone.
The question. Your 0.2.8 note says the fix is
supersedes_richer()and that the trade-off is "theslot's own reply is read again if that conversation comes back". Does that fix also cover this shape,
where the other conversation's park makes
parkedstay at 1 and the next turn of the firstconversation falls back 16,384 tokens? Happy to test a build with the fix on this box (2x 3090,
--batch 4 --batch-mtp, 8 concurrent Claude Code sessions) and report the same breakdown.Conversation cache: parking an idle --batch slot no longer drops the …
Niko1221 commentedon Oct 29, 2026
Your reading was right and your test is credited in the commit. Fixed in 0.2.8: parking an idle
--batchslot no longer drops the main session's copy with the shared system-prompt checkpoint. On one RTX A6000 the time to first token from round 2 on went from 3.3 s to 0.66 s. Keeping the other checkpoints (100+ MB each) is still open. Runstrata update. Closing.Merge upstream v0.2.8 into memory-core: the Arc-pair and four-card ro…