feat: async write-path enrichment — decouple ingest from LLM latency (630× on the hot path) by njfio · Pull Request #117 · njfio/rust-synaptic

njfio · 2026-07-23T03:47:15Z

Decouples the fast raw store() from slow per-turn LLM enrichment — the architectural bottleneck that made write-time distillation and LLM-supersession detection impractical (ingest gated on ~5.7s/turn LLM calls). Spec + plan under docs/superpowers/.

What

AgentMemory.state → Arc<RwLock<AgentState>> (behavior-preserving foundation).
MemoryConfig::enrichment_mode: Inline (default, unchanged) / Deferred / Background. In Deferred/Background, store() records the raw memory + enqueues; enrichment (extract → resolve → supersede → fact-store) runs later.
enrich_one engine (src/memory/enrichment.rs): the existing enrichment logic lifted onto shared Arc handles; feeds storage + state + the scoring provider so facts stay retrievable; preserves the failure-ordering invariant.
enrich_pending(): bounded-concurrent drain (default 8) → EnrichmentReport, best-effort.
Background worker + shutdown() / wait_for_enrichment() (in-flight-tracked for determinism).

Measurement (LoCoMo conv-0, 419 turns, local 7B reasoner)

path	time
synchronous (Inline) ingest	1433.9 s (~24 min)
deferred: raw write	2.3 s
deferred: enrich (concurrency 8)	1343.5 s

The write hot path drops 1434 s → 2.3 s (~630×) — store() is fully decoupled from LLM latency. In Background mode enrichment runs off the critical path entirely.

Honest caveat: the total enrichment speedup is 1.1× — a single 7B model on one GPU (ollama, default num_parallel ≈ 1–2) serves the 8 concurrent requests nearly serially, so the LLM backend, not the write path, is now the ceiling. The concurrency mechanism is proven (16 in-flight connections during the deferred enrich); realizing it on total time needs a parallel/batching-capable backend (OLLAMA_NUM_PARALLEL=8, replicas, or a hosted API). Documented in docs/evaluation.md.

Quality

6 TDD tasks, per-task spec+quality reviews. The per-task review found and fixed a real deadlock in retrieve() (an if let write-guard held across a promotion re-lock — passed 484 tests but hung on promotion-eligible retrieval; now regression-tested). The final whole-branch review found a shutdown() hang + a wait_for_enrichment() early-return (both fixed with regression tests). 493 lib tests pass (no hangs), clippy --all-targets clean, fmt clean, deadlock-free (all lock sites traced), Inline default byte-for-byte unchanged.

Documented divergence: Deferred/Background fact-memories are retrievable via storage/dense/sparse search and checkpointable, but (unlike Inline) don't get a KG node / temporal / analytics — noted in the spec and store_fact_shared.

🤖 Generated with Claude Code

Decouple the fast raw write from slow LLM enrichment: store() enqueues, enrich_pending()/background worker runs extract->resolve->supersede->fact-store with bounded concurrency on Arc-shared handles. state becomes Arc<RwLock> for full consistency. Default Inline (unchanged); Deferred/Background opt-in. Removes the ~5.7s/turn write-path bottleneck. Measure LoCoMo ingest speedup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

6 TDD tasks: Arc<RwLock<AgentState>> refactor, EnrichmentMode config + queue, enrich_one engine on shared handles, enrich_pending bounded-concurrent, background worker + shutdown/wait, and the LoCoMo ingest-speedup measurement. APIs verified against current code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…queue

…otal backend-bound) LoCoMo conv-0, 7B reasoner: synchronous ingest 1433.9s vs deferred raw-write 2.3s (630x on the write hot path — store() decoupled from LLM latency). Total enrich speedup 1.1x, backend-bound: ollama default parallelism serves the 8 concurrent requests serially; concurrency proven (16 in-flight) but realized only with a parallel-capable backend. Default Inline unchanged; Deferred/Background carry it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…light tracking + doc/dead-field cleanup - Worker loop now drains before checking the shutdown flag, so a store() racing shutdown() can no longer strand a coalesced Notify permit and park the worker forever. - wait_for_enrichment() now waits on an in-flight counter in addition to the queue, since drain_and_enrich empties the queue into a local batch before enrichment actually finishes. - Documented that Deferred/Background fact-memories lack a KG node, temporal tracking, and analytics events (unlike Inline), in both enrichment.rs and the design spec. - Removed the unused EnrichmentParams::exclude_raw_default field. - Added regression tests for both races. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3cb65b2f44

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-23T03:50:24Z

+                        self.enrichment_queue.lock().await.push_back(
+                            memory::enrichment::PendingEnrichment {
+                                key: key.to_string(),
+                                value: value.to_string(),
+                            },


Coalesce stale pending entries before enrichment

When a caller updates the same key more than once before a Deferred batch runs (or before the Background worker drains it), this appends a separate snapshot for each version. enrich_one later processes the old value but applies its extracted relations and ::fact0 data to the current key, whose raw storage entry has already been overwritten; the resulting graph/fact memory can therefore describe an obsolete version. Replace or version pending work per key, or verify the queued value is still current before applying enrichment.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-23T03:50:25Z

+    pub async fn stats(&self) -> MemoryStats {
+        let state_guard = self.state.read().await;


Keep existing query methods synchronous

This changes the public stats() API from returning MemoryStats immediately to returning a future; the same state refactor also changes public has_memory() to async. Existing downstream users of the 0.2.0 API (for example, memory.stats().total_size or if memory.has_memory(key)) will no longer compile even though the crate version was not bumped. Preserve compatible synchronous query access or make this a deliberate major-version API change.

Useful? React with 👍 / 👎.

njfio and others added 11 commits July 22, 2026 20:51

refactor(memory): state -> Arc<RwLock<AgentState>> (behavior-preserving)

c49a03b

feat(enrichment): EnrichmentMode config + pending queue + deferred en…

7195ed2

…queue

feat(enrichment): enrich_one engine on shared Arc handles

9ed120c

feat(enrichment): enrich_pending bounded-concurrent + EnrichmentReport

124bd73

feat(enrichment): background worker + shutdown/wait_for_enrichment

cdb4395

feat(eval): --deferred-enrichment ingest-speedup measurement

b440012

style: cargo fmt (drift across enrichment task commits)

4bf5476

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

njfio merged commit 97932d1 into main Jul 23, 2026
17 of 22 checks passed

Navigation Menu

Repository navigation

feat: async write-path enrichment — decouple ingest from LLM latency (630× on the hot path) - #117

feat: async write-path enrichment — decouple ingest from LLM latency (630× on the hot path)#117
njfio merged 11 commits into
mainfrom
spec/async-write-enrichment

njfio commented Jul 23, 2026

Uh oh!

greptile-apps Bot left a comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 23, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		pub async fn stats(&self) -> MemoryStats {
		let state_guard = self.state.read().await;

Conversation

njfio commented Jul 23, 2026

What

Measurement (LoCoMo conv-0, 419 turns, local 7B reasoner)

Quality

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant