Repository navigation
Conversation
Per-turn memory retrieval results were pushed into the system prompt, which forks the context prefix at its first byte and defeats implicit prompt caching on OpenAI-protocol providers. Attach the retrieval output to the newest non-synthetic user message instead - the same message-stream position buildPostCompactMessages already uses for post-compact memory re-injection - so the system prompt and message history stay byte-stable across turns. Fall back to the system prompt when the conversation has no real user message to attach to. Closes OpenBMB#630.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per-turn memory retrieval results were pushed into the system prompt (
DefaultContextRuntime.prepareForModel), so the head of the context changed on every turn and OpenAI-protocol providers could never reuse their implicit prompt/KV cache — every "continuation" turn recomputed the whole prefix.This moves the retrieval output out of the system prompt and attaches it to the newest non-synthetic user message (prepended as its own text block). Dynamic content now rides at the tail of the context; the system prompt and the entire message history stay byte-stable across turns.
Closes #630.
Why this shape
buildPostCompactMessagespushesresult.attachments(the sameMemoryAttachmentBuilderoutput, wrapped in<memory-context>) into the message stream, not the system prompt.<date-update>notices, which are appended at the tail specifically to preserve the cached prefix.date_update) are skipped, so a date rollover cannot capture the memory block; injection then lands on the last real user message.<memory-context>wrapper fromMemoryAttachmentBuilder, so no second wrapper is added.Tests
New regression coverage in
tests/context/cache-runtime.spec.ts:memory attachments ride in the newest user message instead of the system prompt— system prompt (andsystemPromptParts) contain no memory text; the newest real user message starts with the<memory-context>block; earlier messages are untouched.memory injection keeps the system prompt byte-stable across turns— with retrieval output changing between turns,systemPromptis byte-identical, the shared history prefix is unchanged, and the new turn carries the new memory; also asserts the caller's message object is not mutated in place.memory injection skips synthetic user messages and falls back to the system prompt— synthetic-only conversations keep the old system-prompt delivery; a synthetic tail message does not capture the block.Validation (Windows, Node v24.15.0, run with
node --import tsx --test):1732d07, new tests only):tests/context/*.spec.ts→ 85 tests, 79 pass / 6 fail — the 3 new tests fail against the old code, reproducing the issue.tool text over token budget is persisted with expanded grep-first preview,large tool error references preserve error semantics for model replay,multibyte truncated tool result references advertise read_file access) fail identically on unmodified main on Windows — they compare/refs/...style paths against backslash paths and are pre-existing environment issues unrelated to this change.tests/agent/loop/model-override-defaults.spec.ts+tests/web/fork-compact-snapshot.spec.ts: 7/7 pass.pnpm exec tsc -p tsconfig.json --noEmit: clean.