Skip to content

[Engine] --batch: parking an idle slot (#2526) drops the copy with the shared system-prompt checkpoint, so the next agent request re-reads 5K tokens #4420

Closed

Description

@shenmintao

Area

Engine (the server's conversation cache with --batch)

Desktop app, Terminal or source?

Source build in Docker (upstream v0.2.7 + local patches, see Strata version), serve/server.py with strata --serve.

What happened

Our voice assistant starts every request with the same 5,156-token prefix (system prompt + 18 tool definitions) and adds a
short question. Since 0.2.7, when its requests overlap another user's answer (--batch 2), the cached prefix gets lost:
the third overlapping request of our probe reads the whole prompt again (prompt 5173 tokens = 0 reused + 5173 read in 4130 ms), where 0.2.6 on the same box borrows it (borrowed 5156 tokens from a parked conversation and 5156 reused + 17 read in 323 ms). Requests sent alone are not affected: after the first one they reuse 5156 tokens on both versions.

Expected: the shared prefix stays borrowable, as in 0.2.6.

build server starts 8th agent request (3rd overlapping) wall time of that request
0.2.6 (engine 0.1.51) 3 5156 reused + 17 read, 323-400 ms 2.43-3.06 s
0.2.7 (engine 0.1.52) 3 0 reused + 5173 read, 4130-4172 ms 6.12-6.46 s
0.2.7 + the change below 2 5156 reused + 17 read, 322-344 ms 2.56-2.95 s

Same box and config, alternating server starts. One of the three 0.2.7 starts ran without our --pcie-frac 0.38 (the
engine's own split); everything else was identical.

Smallest steps or command that reproduce it

Server (the relevant arguments): --serve --batch 2 --batch-mtp --mtp <dir> --spec 4 --spec-min-p 0.5 --kv q4_0 --max-context 524288 --conversation-cache-mib 1024 --conversation-cache-slots 6. One client process, every request
temperature 0, reasoning_effort none:

  1. Agent request: a system message (system prompt + tool instructions), tools with 18 functions, one user question,
    max_tokens 120; 5,156 tokens come before the question. Send it 5 times alone, a different question each time.
  2. Three rounds: start a long answer in another conversation (one 38-token user message, max_tokens 600,
    "strata_checkpoint": false); 3 s later send the agent request with a new question; wait for both.
  3. Round 3's agent request reads all 5,173 tokens: 3 of 3 starts on 0.2.7, 0 of 3 on 0.2.6.

The prefix is our assistant's private prompt and is not attached. We have only run the probe with it.

What the log shows

0.2.7, from round 2's agent request to round 3's:

strata serve: prompt 5175 tokens = 5156 reused + 19 read in 357 ms (53.2 tok/s), 1 generated in 29 ms (33.9 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: conversation cache: parked 5175 tokens in 160.9 ms; parked=4 bytes=928080148 evictions=11 snapshot_bytes=400964236 reused_kv_bytes=0
strata batch: slot 0 gave back 199 tokens of this conversation (all it holds) in 90.3 ms
strata serve: prompt 200 tokens = 199 reused + 1 read in 266 ms (3.8 tok/s), 369 generated in 7166 ms (51.5 tok/s), drafts accepted 194 of 316, 0 checkpoints
strata serve: prompt 38 tokens = 0 reused + 38 read in 479 ms (79.3 tok/s), 125 generated in 2551 ms (49.0 tok/s), drafts accepted 61 of 108, 0 checkpoints
strata serve: conversation cache: dropped 1 superseded copy of this conversation; parked=3
strata serve: conversation cache: parked 5206 tokens from slot 1 in 61.2 ms; parked=4 bytes=810285000 evictions=11 snapshot_bytes=283169088 reused_kv_bytes=0
strata batch: slot 1 takes 163 tokens (copied in 169.5 ms)
strata serve: prompt 163 tokens = 0 reused + 163 read in 974 ms (167.4 tok/s), 1 generated in 26 ms (38.6 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: conversation cache: parked 199 tokens from slot 0 in 106.2 ms; parked=5 bytes=930157780 evictions=11 snapshot_bytes=119872780 reused_kv_bytes=0
strata batch: slot 0 takes 5173 tokens (copied in 245.5 ms)
strata serve: prompt 5173 tokens = 0 reused + 5173 read in 4130 ms (1252.7 tok/s), 1 generated in 36 ms (28.0 tok/s), drafts accepted 0 of 0, 2 checkpoints

Round 2's agent conversation is read in the main session (2 checkpoints), decoded in slot 1, and its main-session copy is
parked (5175 tokens). In round 3 the long answer goes to slot 1: park_slot(1) parks what slot 1 holds (round 2's agent
conversation with its reply, 5206 tokens) and drops the 5175-token copy as superseded. Nothing parked then has the
5156-token checkpoint, and round 3's agent request reads everything.

0.2.6, same point:

strata serve: prompt 5175 tokens = 5156 reused + 19 read in 349 ms (54.4 tok/s), 1 generated in 29 ms (35.1 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: conversation cache: parked 5175 tokens in 150.0 ms; parked=2 bytes=801891680 evictions=11 snapshot_bytes=400963908 reused_kv_bytes=0
strata batch: slot 0 gave back 184 tokens of this conversation (all it holds) in 105.9 ms
strata serve: prompt 185 tokens = 184 reused + 1 read in 276 ms (3.6 tok/s), 364 generated in 7378 ms (49.3 tok/s), drafts accepted 190 of 325, 0 checkpoints
strata serve: prompt 38 tokens = 0 reused + 38 read in 480 ms (79.1 tok/s), 123 generated in 2562 ms (48.0 tok/s), drafts accepted 62 of 114, 0 checkpoints
strata batch: slot 1 takes 161 tokens (copied in 82.6 ms)
strata serve: prompt 161 tokens = 0 reused + 161 read in 975 ms (165.1 tok/s), 1 generated in 29 ms (34.5 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: conversation cache: borrowed 5156 tokens from a parked conversation, which stays parked, in 41.8 ms; parked=2 bytes=801891680 borrowed=3
strata batch: slot 0 takes 5173 tokens (copied in 151.9 ms)
strata serve: prompt 5173 tokens = 5156 reused + 17 read in 323 ms (52.7 tok/s), 1 generated in 45 ms (22.2 tok/s), drafts accepted 0 of 0, 2 checkpoints

Why (our reading of the v0.2.7 code; not checked beyond the runs above)

  • At a slot admission the slot's chain gets one checkpoint, the deepest one that is a strict prefix of the live tokens
    (src/program/generate.cpp around L17102, sl.checks.push_back(*best)): for an agent request its turn checkpoint, not
    the one at the end of the shared prefix that the next requests borrow.
  • park_slot (L12603) goes through park_source, which calls conversations.drop_superseded(src.ids, src.imgs, src.checks, src.cvec) (L10470), and ConversationCache::put() calls it again (conversation_cache.hpp L493).
    drop_superseded (Conversation cache slots are consumed by successive turns of the same subagent, evicting the parent conversation #342, L442) drops a parked entry when the outgoing chain holds that entry's deepest checkpoint. The
    main-session copy of the same conversation qualifies, and its checkpoint at the end of the shared prefix goes with it.
    The rule assumes the outgoing chain also carries the older checkpoints, which is true for the main session's chain but
    not for a slot's.
  • The slot images also take room. With --conversation-cache-mib 1024, an agent conversation's main copy is 401 MB, a
    slot image 283 MB (agent) or 120-124 MB (long answer). With them parked the cache holds one main copy where 0.2.6 holds
    two (parked=2 bytes=801891680 above), so the older copies with the shared checkpoint had been evicted, and the
    dropped one was the last. This part is reconstructed from the parked=, bytes= and evictions= fields; the log does
    not say which entry an eviction removes.

The change we run

ConversationCache::supersedes_richer(ids, images, checkpoints, cvec) is true when drop_superseded with that chain
would drop an entry that has a checkpoint the chain does not hold. park_slot then returns early and says so; the slot is
overwritten as in 0.2.6, so a continuation of that conversation reads its reply again (about 30 tokens here). A slot with
no such copy (the long answers here) is still parked. With it, at the same point:

strata serve: prompt 5175 tokens = 5156 reused + 19 read in 327 ms (58.1 tok/s), 1 generated in 30 ms (33.8 tok/s), drafts accepted 0 of 0, 2 checkpoints
strata serve: conversation cache: parked 5175 tokens in 129.9 ms; parked=3 bytes=921446256 evictions=12 snapshot_bytes=400964236 reused_kv_bytes=0
strata batch: slot 0 gave back 188 tokens of this conversation (all it holds) in 118.8 ms
strata serve: prompt 189 tokens = 188 reused + 1 read in 258 ms (3.9 tok/s), 365 generated in 7397 ms (49.3 tok/s), drafts accepted 185 of 317, 0 checkpoints
strata serve: prompt 38 tokens = 0 reused + 38 read in 479 ms (79.4 tok/s), 131 generated in 2527 ms (51.8 tok/s), drafts accepted 69 of 118, 0 checkpoints
strata serve: conversation cache: slot 1 not parked before its overwrite (a parked copy of its conversation holds checkpoints the slot does not; it stays)
strata batch: slot 1 takes 169 tokens (copied in 132.7 ms)
strata serve: prompt 169 tokens = 0 reused + 169 read in 991 ms (170.5 tok/s), 1 generated in 29 ms (34.6 tok/s), drafts accepted 0 of 0, 0 checkpoints
strata serve: conversation cache: borrowed 5156 tokens from a parked conversation, which stays parked, in 44.6 ms; parked=3 bytes=921446256 borrowed=3
strata serve: conversation cache: parked 188 tokens from slot 0 in 51.0 ms; parked=4 bytes=1041216848 evictions=12 snapshot_bytes=119770592 reused_kv_bytes=0
strata batch: slot 0 takes 5173 tokens (copied in 223.1 ms)
strata serve: prompt 5173 tokens = 5156 reused + 17 read in 322 ms (52.7 tok/s), 1 generated in 28 ms (35.6 tok/s), drafts accepted 0 of 0, 2 checkpoints
Patch against v0.2.7 (3 files, +59)
diff --git a/include/strata/core/conversation_cache.hpp b/include/strata/core/conversation_cache.hpp
index 34898345..a4f03894 100644
--- a/include/strata/core/conversation_cache.hpp
+++ b/include/strata/core/conversation_cache.hpp
@@ -465,6 +465,33 @@ public:
         superseded_ += dropped;
         return dropped;
     }
+    // A batch slot's chain holds one checkpoint (the one its admission copied in), while the main session's parked copy
+    // of the same conversation keeps the whole chain - the root checkpoint at the end of an agent's system prompt and
+    // tools among them, the one its next conversations borrow (borrows()).  Parking the slot (#2526) would drop that
+    // copy as superseded (#342: the slot's chain holds its deepest checkpoint) and the root with it: every sibling then
+    // reads the shared prefix again.  True when drop_superseded(ids, images, checkpoints, cvec) would drop an entry that
+    // has a checkpoint the given chain does not hold.
+    bool supersedes_richer(const std::vector<int32_t>& ids, const std::vector<ConversationImageKey>& images,
+                           const std::vector<ConversationCheckpoint>& checkpoints, bool cvec) const {
+        auto held = [&](const ConversationCheckpoint& c) {
+            if (c.ids == ids && c.imgs == images) return true;
+            for (const auto& k : checkpoints)
+                if (k.ids == c.ids && k.imgs == c.imgs) return true;
+            return false;
+        };
+        for (size_t i = 0; i < entries_.size(); ++i) {
+            const auto& e = entries_[i];
+            const ConversationCheckpoint* deepest = nullptr;
+            for (const auto& c : e.checkpoints)
+                if (!deepest || c.ids.size() > deepest->ids.size()) deepest = &c;
+            if (e.cvec != cvec || !deepest || deepest->ids.empty() || !held(*deepest) ||
+                (guarded_ != 0 && serials_[i] == guarded_))
+                continue;
+            for (const auto& c : e.checkpoints)
+                if (!held(c)) return true;
+        }
+        return false;
+    }
     size_t superseded() const { return superseded_; }
     /// --conversation-cache-park-gain N: the longest parked entry this conversation's outgoing state supersedes, in
     /// tokens, 0 when it has none.  It is the entry drop_superseded() would remove, so the caller can tell whether a
diff --git a/src/core/conversation_cache_test.cpp b/src/core/conversation_cache_test.cpp
index 242e3760..6c2aec46 100644
--- a/src/core/conversation_cache_test.cpp
+++ b/src/core/conversation_cache_test.cpp
@@ -720,5 +720,30 @@ int main() {
         check(tiny.put(std::move(p2)), "a pinned one fits an empty cache");
         check(!tiny.put(image({7,8,9})) && tiny.size() == 1, "nothing else fits beside it: parking is refused, the pin stays");
     }
+    {
+        // a batch slot's chain holds one checkpoint (its admission's): parking it must not drop (#342) the main
+        // session's copy of the same conversation that also holds the root a sibling borrows from (an agent's system
+        // prompt and tools) - supersedes_richer() names that case, so the slot is not parked
+        auto cp = [](std::vector<int32_t> ids) { ConversationCheckpoint c; c.ids = std::move(ids); return c; };
+        SavedConversation main_copy = image({1, 2, 3, 4, 10, 11, 12});
+        main_copy.checkpoints = {cp({1, 2, 3, 4}), cp({1, 2, 3, 4, 10, 11})};
+        ConversationCache cache(1 << 20, 4);
+        check(cache.put(std::move(main_copy)), "park the main session's copy with its root and turn checkpoints");
+        const std::vector<int32_t> slot_ids{1, 2, 3, 4, 10, 11, 12, 20, 21};          // the slot decoded the reply
+        const std::vector<ConversationCheckpoint> slot_chain{cp({1, 2, 3, 4, 10, 11})};   // the admission's only one
+        check(cache.supersedes_richer(slot_ids, {}, slot_chain, true), "the slot's park would drop the root checkpoint");
+        const std::vector<ConversationCheckpoint> full_chain{cp({1, 2, 3, 4}), cp({1, 2, 3, 4, 10, 11})};
+        check(!cache.supersedes_richer(slot_ids, {}, full_chain, true), "a chain that holds both loses nothing");
+        check(!cache.supersedes_richer(slot_ids, {}, slot_chain, false), "steering mode is isolated");
+        check(!cache.supersedes_richer(std::vector<int32_t>{1, 2, 3, 4, 30, 31}, {},
+                                       std::vector<ConversationCheckpoint>{cp({1, 2, 3, 4, 30})}, true),
+              "another conversation on the same root supersedes nothing");
+        check(cache.size() == 1 && cache.superseded() == 0, "the predicate changes nothing");
+        const auto sib = cache.best(std::vector<int64_t>{1, 2, 3, 4, 40}, {}, true);
+        check(sib.tokens == 4 && cache.borrows(sib), "a sibling still borrows the root");
+        // what drop_superseded does with the slot's chain (the park this rule prevents): the root goes
+        check(cache.drop_superseded(slot_ids, {}, slot_chain, true) == 1 && cache.size() == 0,
+              "the slot's chain supersedes the richer copy");
+    }
     std::printf("conversation_cache_test: %d checks passed\n", checks);
 }
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 824bd19d..644232f8 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -12609,6 +12609,13 @@ int main(int argc, char** argv) {
             const bool in_main = live_ok && cvec_cached == sl.cvec && prefix_of(live) &&
                                  imgs_below(live_imgs, (int64_t) sl.ids.size()).empty();
             if (in_main || conversations.holds(sl.ids, {}, sl.cvec)) return;
+            if (conversations.supersedes_richer(sl.ids, {}, sl.checks, sl.cvec)) {
+                // its park would drop (as superseded) the main session's copy of this conversation, and with it the
+                // checkpoints the slot does not hold - an agent's system prompt root that the next requests borrow
+                std::fprintf(stderr, "strata serve: conversation cache: slot %d not parked before its overwrite (a parked "
+                                     "copy of its conversation holds checkpoints the slot does not; it stays)\n", b);
+                return;
+            }
             if (use_mtp && !batch_mtp) {   // no draft K/V of its own: an image without it would not restore beside the drafter
                 std::fprintf(stderr, "strata serve: conversation cache: slot %d not parked before its overwrite (its %zu tokens "
                                      "have no per-slot draft K/V without --batch-mtp)\n", b, sl.ids.size());

Ways that would keep the reply as well, not tried: copy the superseded entry's missing checkpoints into the slot's image,
or give a slot the conversation's whole chain at admission. sycl/src/program/generate.cpp has the same park_slot; the
patch does not touch it (we have no SYCL device).

Tests:

  • Passed, on our tree (v0.2.7 + the local patches below; g++ 13 for the standalone tests, HIP build of strata):
    conversation_cache_test (4842 checks, with the new case), conversation_split_failure_test,
    conversation_memory_test, conversation_file_test, session_save_reclaim_test, ple_residency_test,
    arch_defaults_test; python -m unittest of serve.test_frontend, serve.test_answer_speed, serve.test_family_name,
    serve.test_responses, serve.test_vision_elastic, serve.test_vision_elastic_http, serve.test_vision_start_error,
    serve.test_cache_drop. The patch applies to plain v0.2.7 (git apply --cached --check); not built there.
  • Hardware: the 2 starts in the table, 6 of 6 overlapping agent requests reused 5156 tokens. Our A/B of the same starts
    (4K prompt and decode, long answers, a 100K-token document 6/6, two users, pictures 4/4) stayed within run-to-run noise
    of 0.2.6 and 0.2.7.
  • Not run: CUDA, SYCL, other agent prefixes or cache budgets.

Strata version

Strata 0.2.7 · engine 0.1.52: upstream v0.2.7 (3f6d9c28) + local patches (a conversation disk tier that only takes
conversations of 16,384+ tokens and did not read or write during these probes, extra /v1/models fields, an elastic CPU
image encoder, an opt-in Q4_0-KV WMMA prompt attention, a gfx1200 hipBLASLt table). The 0.2.6 runs: 0.2.6 · engine
0.1.51 with the same local patches.

Graphics card, memory and system

RX 9060 XT 16 GB (gfx1200, ROCm 7.2.3) · Core i7-12700H · 62 GB RAM, container limited to 52 GiB · Debian 12, kernel 6.18

Model, size and settings

Qwen3.8-Flash-Next IQ2_XS · 512K context · pictures on (CPU encoder) · --batch 2 --batch-mtp, MTP drafts
(--spec 4), --kv q4_0, conversation cache 1024 MiB / 6 slots

Reproduction status

Reproduced locally

Activity

  1. Niko1221 commented on Oct 29, 2026

    @Niko1221
    Owner

    Your reading was right, and supersedes_richer() is the smallest change, so the fix carries it with your test (credited in the commit). Parking an idle --batch slot dropped the main session's copy with the shared system-prompt checkpoint, so the next agent request read the 6,028-token prefix again. Fixed on our side; the fix ships in the next update (0.2.8).

    On one RTX A6000 (Qwen IQ3_S, --batch 2, a 6,052-token prefix) the time to first token from round 2 on was 3.3 s before and 0.66 s after. The trade-off you described stays: the slot's own reply is read again if that conversation comes back. Keeping the missing checkpoints (100+ MB each) is left open.

  2. drrck82 commented on Oct 29, 2026

    @drrck82

    Same failure here on 0.2.7 / engine 0.1.52, but it loses a whole conversation's copy, not just a shared prefix.

    Rig. Linux, 2x RTX 3090 (layer split auto: CUDA0 layers 0-21, CUDA1 22-47 + head/draft),
    Qwen3.8-Flash-Next GSQ-RCO IQ3_S + --mtp, engine 0.1.52 (tag v0.2.7, 5e8a4cca).
    Relevant server args: --batch 4 --batch-mtp --batch-groups auto --kv int8 --kv-resident 32768 --max-context 262144 --conversation-cache-mib 24576 --conversation-cache-slots 8 --conversation-cache-min-tokens 8192 --conversation-cache-min-free-mib 4096,
    --prompt-cache and --prompt-cache-every at their defaults (6 / 16384).
    STRATA_PIPED_MTP=1 in the environment. Client: Claude Code, 8 sessions at once, byte-identical
    append-only prompts (--batch-groups auto resolves to 2 groups of 2).

    What it costs. 43 turns since the last restart: 31 full hits (median prefill 0.65 s), 10 turns
    restoring a stale checkpoint (median 4.3 s, 52 s in total), 2 cold. evictions=0 throughout and the
    cache sat at 7.1 GiB of its 24.5 GiB budget, so nothing was evicted for space.

    Two conversations alternate, and the parked copy does not survive the other one's park.
    Conversation A is at ~66 K, B at ~95 K. Note parked stays at 1 although both are served:

    conversation cache: parked 66320 tokens in 701.7 ms; parked=1 bytes=1725097840 evictions=0 ...
    conversation cache: restored 93632 tokens (live) in 124.7 ms; parked=1 bytes=3154683352
    conversation cache: parked 94217 tokens in 349.4 ms; parked=1 bytes=2441321984 evictions=0 ...
    conversation cache: restored 54980 tokens (checkpoint) in 108.5 ms; parked=1 bytes=3453907880
    conversation cache: parked 66474 tokens in 301.2 ms; parked=1 bytes=1762157888 evictions=0 ...
    conversation cache: restored 94217 tokens (live) in 126.3 ms; parked=1 bytes=3370397900
    conversation cache: parked 95018 tokens in 349.9 ms; parked=1 bytes=2441454640 evictions=0 ...
    conversation cache: restored 54980 tokens (checkpoint) in 111.1 ms; parked=1 bytes=3491098752
    

    A parks at 66,320 / 66,474 / 66,576 and B is restored live each time -- yet A's own next turn
    restores 54,980, 11.5 K tokens behind the copy it had parked one turn earlier:

    conversation cache: restored 54980 tokens (checkpoint) in 108.5 ms; parked=1 bytes=3453907880
    prompt 66465 tokens = 54980 reused + 11485 read in 6463 ms (1777.1 tok/s), 8 generated in 90 ms (88.6 tok/s), drafts accepted 5 of 5, 5 checkpoints
    conversation cache: restored 54980 tokens (checkpoint) in 111.1 ms; parked=1 bytes=3491098752
    prompt 66567 tokens = 54980 reused + 11587 read in 6470 ms (1790.7 tok/s), 8 generated in 84 ms (95.4 tok/s), drafts accepted 5 of 5, 5 checkpoints
    conversation cache: restored 54980 tokens (checkpoint) in 111.8 ms; parked=1 bytes=3491164712
    prompt 66669 tokens = 54980 reused + 11689 read in 6474 ms (1805.6 tok/s), 12 generated in 163 ms (73.6 tok/s), drafts accepted 4 of 4, 5 checkpoints
    

    A second conversation does the same at 71,364 (5 turns, 4,237-4,539 tokens re-read, 3.5-4.1 s each),
    plus one 19,543-token re-read (8.7 s).

    Both restore points are on the --prompt-cache-every grid (16384): 54980 and 71364 are exactly
    16,384 apart (mod 16384 = 5828 for both). The previous session on this box showed the same pair at
    30,404 and 46,788 -- also exactly 16,384 apart. So the restore lands on a periodic checkpoint, not on
    the turn-boundary state. Per #2228's diagnosis, the prompt is a pure append, so the deeper state
    should have matched; the sibling conversation's grid checkpoint is winning because the requester's own
    copy is gone.

    The question. Your 0.2.8 note says the fix is supersedes_richer() and that the trade-off is "the
    slot's own reply is read again if that conversation comes back". Does that fix also cover this shape,
    where the other conversation's park makes parked stay at 1 and the next turn of the first
    conversation falls back 16,384 tokens? Happy to test a build with the fix on this box (2x 3090,
    --batch 4 --batch-mtp, 8 concurrent Claude Code sessions) and report the same breakdown.

  3. Niko1221 commented on Oct 29, 2026

    @Niko1221
    Owner

    Your reading was right and your test is credited in the commit. Fixed in 0.2.8: parking an idle --batch slot no longer drops the main session's copy with the shared system-prompt checkpoint. On one RTX A6000 the time to first token from round 2 on went from 3.3 s to 0.66 s. Keeping the other checkpoints (100+ MB each) is still open. Run strata update. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions