Repository navigation
fix(dedupe): reject LLM output that introduces lines not in the original - #10
Merged
Merged
Conversation
semantic_dedupe_memory only guarded against too-short responses. But the
observed failure mode makes the file LONGER: the model ignores "return ONLY
the cleaned file" and prepends chat commentary ("No duplicates were found…",
or a flag about session context), which --apply would then write verbatim as
the opening lines of GLOBAL_MEMORY.md and load into every session.
A dedupe may only remove lines, never introduce them. Fail closed if any
non-blank output line was not present verbatim in the input. This is a
deterministic code-level guard (per PITFALLS #1: enforce in code, not prompt),
and makes the command safe to run unattended in a scheduled preview.
Tests: subset-removal passes; too-short still rejected; prepended and appended
commentary both rejected; unchanged-file passes. Also fixes the mock target in
the new tests (function-local `from memsync.llm import call_llm` must be patched
at memsync.llm, not memsync.sync).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EdgeCaser
added a commit
that referenced
this pull request
Jul 27, 2026
The hardcoded max_tokens=4096 in harvest_memory_content and refresh_memory_content silently skipped every session once the memory file grew past ~4k tokens — the model couldn't re-emit the full file within the ceiling, hit stop_reason=max_tokens, and got flagged as "truncated — skipped". Adds max_tokens to Config (default 16384, TOML-configurable via [core] max_tokens) and threads it through both API call sites. Updates PITFALLS #10 with the real root cause. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EdgeCaser
added a commit
that referenced
this pull request
Jul 27, 2026
fix(dedupe): reject LLM output that introduces lines not in the original
EdgeCaser
added a commit
that referenced
this pull request
Jul 27, 2026
… regen) Follow-up to #10. Validating semantic dedupe live surfaced that Sonnet at low effort keeps prepending a "No duplicates found" announcement and, worse, reads instruction-like text inside the memory (persona rules, security notes) and reacts to it — all of which #10's guard correctly rejected but which left the command erroring on every run, so the preview was never useful. Prompt wording alone can't stop this, and the claude_code backend only injects prefill as a soft system hint (not a real assistant-turn prefill), so that lever is out too. The durable fix is to change the contract: the model no longer regenerates the file. It returns only the verbatim text of the bullets to REMOVE (or NONE), and Python performs the removal. Stray chatter that matches no real bullet is ignored; the file's integrity no longer depends on model formatting. This supersedes #10's introduced-lines guard (the model can no longer introduce anything) with a sanity cap (never remove >half the bullets) and verbatim-match removal (a hallucinated/paraphrased line removes nothing). Also fixes a trailing-newline mismatch that produced a spurious 1-line diff on clean files, and a blank-line edge case ("" is spuriously `in` any string). Live-validated: `memsync dedup --semantic` on the real GLOBAL_MEMORY.md now returns "No semantic duplicates found." (exit 0), instead of erroring. Tests: mark-and-remove; NONE is byte-identical; non-matching commentary is a no-op; hallucinated removal is a no-op; bulk-removal cap; trailing-newline and blank-line spacing preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
memsync dedupe --semanticruns an LLM pass to catch same-policy bullets phrased differently — the only mechanism that can reliably distinguish real duplicates from distinct-but-similar rules (string similarity can't: today's real metered-spend near-dupes scored 0.727 while two genuinely-distinct constraints scored 0.912).But a live run against the real
GLOBAL_MEMORY.mdsurfaced a corruption path. The model ignored "Return ONLY the cleaned file content" and prepended chat commentary as file content:The dry-run reported 123 → 127 lines — it didn't remove anything, it added four lines. On
--apply, that paragraph would have been written as the opening lines ofGLOBAL_MEMORY.mdand loaded into every Claude Code session on every machine. The existing safety guard can't catch this: it only rejects output that's too short (<50%), and this pollution makes the file longer.The fix
A dedupe may only remove lines, never introduce them. Fail closed if any non-blank output line was not present verbatim in the input. Deterministic, code-level (per PITFALLS #1: enforce in code, not the prompt), and it makes the command safe to run unattended in a scheduled preview.
Tests
New
TestSemanticDedupeMemory:Also fixes the mock target:
semantic_dedupe_memorydoes a function-localfrom memsync.llm import call_llm, so tests must patchmemsync.llm.call_llm, notmemsync.sync.call_llm— otherwise they hit the real backend.Full suite: 458 passed, 1 skipped, coverage 82.58% (sync.py 97%).
ruff check memsync/clean.🤖 Generated with Claude Code