Fix #2456: Malformed extraction JSON silently becomes an empty successful /product/add resp - #2457
Open
Memtensor-AI wants to merge 2 commits into
Open
Memtensor-AI wants to merge 2 commits into
Memtensor-AI wants to merge 2 commits into
Conversation
`parse_json_result` previously returned `{}` when the LLM emitted a trailing
comma before `}` or `]` (e.g. `[{"value": "x"},]`). The empty dict
propagated through `multi_modal_struct._process_string_fine` — whose
exception-based fallback never fired because no exception was raised — and
surfaced as `POST /product/add` returning
`{"code":200, "message":"Memory added successfully", "data": []}` with no
memories stored. Related to MemTensor#1355 / fixed-by MemTensor#1977, but a distinct failure
path: the SimpleStructMemReader fallback-key fix does not cover this case.
Add a narrow last-chance repair that strips commas sitting immediately
before `}` or `]` **outside** quoted strings (respecting backslash-escaped
quotes), then reparses with `json.loads`. On success emit a
`[JSONParse] Repaired malformed JSON...` warning so the previously-silent
failure mode is observable. On repair failure keep returning `{}` to
preserve the historical contract for the 15 call sites.
Regression tests in `tests/mem_reader/test_parse_json_result.py` cover the
exact reproduction from the bug report plus safety cases (literal `,}`
inside strings, escaped quotes, genuinely-malformed JSON still returns
`{}`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Memtensor-AI
added
ai:generated
Collaborator
Author
🤖 Open Code ReviewTarget: PR #2457 ✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). Generated by cloud-assistant via Open Code Review. |
Collaborator
Author
🔧 Open Code Review requested Agent fixOpen Code Review found 3 issue(s). I have resumed the development Agent to fix them.
The Agent will push a new commit to this PR branch. OCR will recheck after the commit is pushed. |
…mma repair Three follow-up issues from the Open Code Review on PR MemTensor#2457: 1. tests/mem_reader/test_parse_json_result.py — the escaped-quote regression actually had its trailing ``,}`` *inside* the quoted value, so the raw string was already valid JSON and ``_strip_trailing_commas`` was never exercised. Move the trailing comma outside the string and assert the inner commas (and the escaped quotes around them) survive the repair. Also add a positive case for ``,,}`` runs so the fix below is pinned. 2. src/memos/mem_reader/utils.py — ``_strip_trailing_commas`` only remembered the *latest* candidate comma, so ``{"a":1,,}`` was repaired to ``{"a":1,}`` which still fails ``json.loads``. The outer ``logger.warning`` only fires when ``repaired == t``, so the silent fallthrough re-created the exact failure mode (MemTensor#2456) the repair was meant to kill. Track *all* pending comma indices and flush the whole run on a closer; a string open or any other non-whitespace token clears them. 3. src/memos/mem_reader/utils.py — when the repair modified the text but the result still failed ``json.loads``, the inner except silently swallowed the second error with no diagnostic. Emit a ``logger.debug`` with the repair-attempt error so future debugging has a breadcrumb; the outer WARNING still fires with the original decode error for operator visibility. Verified: ``PYTHONPATH=src pytest tests/mem_reader/test_parse_json_result.py`` 12 passed (adds 1 new case). The 2 pre-existing failures in ``tests/mem_reader/test_coarse_memory_type.py`` are unrelated (missing optional ``markitdown`` dep). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix issue #2456: malformed extraction JSON with a trailing comma before
}or](common output from GLM / Z.AI / Kimi style models) was silently converted to{}byparse_json_result, which downstream surfaced asPOST /product/addreturning{"code":200,"message":"Memory added successfully","data":[]}with zero memories persisted. Themulti_modal_struct._process_string_fineexception-based fallback never fired because no exception was raised. Related to #1355 (fixed by #1977) but a distinct failure path — that fix only correctedSimpleStructMemReader's fallback key.The fix adds a narrow last-chance repair in
src/memos/mem_reader/utils.py::parse_json_result: a_strip_trailing_commasscanner walks the payload character-by-character, tracks whether it is inside a quoted string (respecting backslash-escaped quotes), and removes commas that sit immediately before}or]outside strings. On successful repair it emits a[JSONParse] Repaired malformed JSON...warning so the previously-silent failure mode is observable; on genuine failure it still returns{}to preserve the historical contract for the 15 call sites. Zero signature change, zero API contract change.Testing: 11 new regression cases in
tests/mem_reader/test_parse_json_result.pycover the exact repro from the bug body, trailing commas before}/]/newlines/fenced code blocks, safety cases (literal,}inside strings must survive, escaped quotes must not confuse the scanner), and the preserved{}return for genuinely malformed JSON. All 11 pass. Adjacent suites (test_simple_structure,test_simple_struct_fallback) still green: 20/20. Fulltests/mem_reader/run: 80 passed, 2 pre-existing environment failures (test_doc_local_file,test_doc_mixed) requiring the optionalmarkitdownextra — confirmed reproducible on untouched dev-v2.0.36 HEAD. Ruff format + check clean. Archive synced and pushed to memos-autodev-specs main (commit 33ce785).Related Issue (Required): Fixes #2456
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Not run; documentation-only change.
Checklist
@WeiminLee please review this PR.
Reviewer Checklist