Skip to content

docs(eval): frontier reasoner detects natural-language supersessions (bi-temporal detection resolved) #120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
njfio merged 1 commit into from
Jul 23, 2026
Merged

docs(eval): frontier reasoner detects natural-language supersessions (bi-temporal detection resolved) #120

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/evaluation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2057,3 +2057,31 @@ equals the LLM backend's effective serving concurrency** (ollama default ≈1, s
replicated/hosted backend at 8-way → ~7.8×). Caveat: the mock is a controlled backend (fixed 1 s
latency), used only to isolate the harness-side concurrency; the real-LLM number remains the honest
end-to-end figure for a single local model.

### Bi-temporal, resolved: a frontier reasoner DOES detect natural-language supersessions (2026-07-23)

The bi-temporal thread stalled on detection: the heuristic reasoner (same subject + predicate, different
object) fired **0 supersessions** on LongMemEval knowledge-update (weak-answerer run: 40/40 identical
predictions), and an LLM-reasoner run was aborted as impractical (~4 h serial ingest). Two things
unblocked it: (1) the async write path (enrichment off the critical path, concurrent), and (2) a fast,
**concurrent, frontier** LLM backend — the ChatGPT/OpenAI OAuth `gpt-5.6-sol` served directly over its
responses endpoint (8 parallel calls in 4.1 s, vs the codex CLI's ~15 s serial process-spawn).
Comment on lines +2066 to +2068

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the endpoint probe outside the eval harness

The document’s stated provenance says every number comes from a measured run_eval invocation, but this new timing is from a direct OAuth Responses-endpoint probe. run_eval only configures the library reasoner via SYNAPTIC_LLM_URL/SYNAPTIC_LLM_MODEL and ingests dataset conversations; it has no mode to issue this direct four-case resolve probe or report its parallel-call timing. As written, readers will incorrectly treat the 4.1 s / 15 s measurements and the four verdicts as reproducible evaluation-harness results; add the probe command/method and explicitly exempt it from the harness provenance claim (or move it to a separately labeled experiment).

Useful? React with 👍 / 👎.


Direct probe of the library's `resolve` prompt (the exact write-path contradiction check) against the
frontier model, on realistic natural-language updates:

| new fact | prior memory | frontier verdict |
|---|---|---|
| "My charity 5K PB is now 25:50" | "…is 27:30" | **supersede** ✓ |
| "I just smashed my PB — finished in 25:50!" | "…is 27:30" | **supersede** ✓ |
| "I moved to Munich last month for a new job." | "I live in Berlin…" | **supersede** ✓ |
| "I adopted a golden retriever puppy." (control) | "…5K PB is 27:30" | **insert** ✓ |

The frontier reasoner detects natural-language supersessions correctly **and discriminates** (the
unrelated control returns `insert`, not `supersede`) — exactly what the heuristic could not do. This
closes the bi-temporal thread's root question: the blocker was **detection quality**, not the retrieval
mechanism (proven by unit test) nor answerer masking (a weak answerer still showed nothing because
nothing fired). With a frontier reasoner, supersessions fire; the async write path makes running that
detection at scale practical (concurrent, off the critical path). The end-to-end value question — does
dropping these correctly-detected stale facts improve a weak answerer's knowledge-update QA — is the
natural next A/B, now that both the detection and the throughput are in hand.
Loading