feat(jev): the context engine — budget-greedy state assembly with paper-content taste cards

CONTEXT.md

@@ -21,3 +21,15 @@ before. No implementation detail lives here.
   writes its body; a written note's body feeds taste.
 - **inspect** — The read-only verification verb: it prints the checklist state
   and what the config actually sees, and exits nonzero when blocked.
+- **taste card** — One kept paper as the Jev state rides it under the
+  budget-greedy policy: the note's why plus the paper's own abstract and
+  conclusions excerpts, weighted by the note's decay and rating.
+- **budget-greedy** — The state-assembly policy where directives ride whole
+  and taste items fill the byte budget by recency-weighted priority; the
+  incumbent fixed-count policy is `fixed-15`.
+- **backfill** — The explicit, paced fetch of every corpus paper's PDF (plus
+  extracted text) into the library; the digest itself never fetches corpus
+  content, so the fold stays local and deterministic.
+- **finalist enrichment** — The candidate-half content arm: after the first
+  scoring pass, the top finalists are re-scored with their fetched
+  conclusions/outlook in the state; fail-open, cached, never in the library.

README.md

@@ -4,13 +4,49 @@ A daily arXiv digest engine that ranks against your **living Obsidian vault** 
 
 arJev serves **any field that posts to arXiv** — condensed matter, economics, quantitative biology, mathematics, any discipline with an arXiv category — and it sources nothing beyond arXiv: no PubMed, no RePEc, no journal feeds. If your field doesn't post to arXiv, arJev is not for you yet, and it will never silently rank a category you didn't choose.
 
+## The context engine: how your corpus becomes the ranking
+
+The design invariant: **your corpus can grow without bound; the model's window never does — and that's the mechanism, not a limitation.**
+
+Every digest runs the same fold:
+
+1. **Fold** — every note under your roots is scanned once; the papers you've kept become first-class records (title, your `why` line, your rating, tags, the vault's wikilink graph).
+2. **Taste cards** — each kept paper becomes a card: your why, the paper's own abstract, and its conclusions/outlook section (extracted from your local PDF library; your directives ride whole and first). Every card carries a 180-day half-life decay and your own rating verdict.
+3. **The budgeted window** — every candidate is judged against a fixed, byte-budgeted context assembled fresh for it: your directives, then cards, terms, and why-lines filled greedily by recency-weighted priority.
+
+Two phases fall out of that one mechanism:
+
+- **Small corpus → maximal fill.** When you've kept a handful of papers, everything you've ever kept rides the window — the early picks are judged against all of it.
+- **Growing corpus → recency-proportional width.** As the corpus grows past the window, the decay ranking gives recent work proportionally more of it. A paper read 300 days ago holds a quarter the width of one read 90 days ago. No reconfiguration — the window tracks your drift.
+
+The corpus never rides into the model wholesale: the fold is a local, linear scan, and the model's context is a fixed-budget summary regardless of vault size. A 10,000-note vault costs the same per digest as a 10-note one.
+
+### The numbers
+
+What the window holds, per budget (generic):
+
+| Taste budget | What rides |
+|---|---|
+| 2.5 KB (default) | full directives + ~15 weighted terms + recent why-lines |
+| 8 KB | full directives + ~4–6 recent taste cards (abstract + conclusions + why) + terms |
+| 12 KB | full directives + ~8–10 cards + terms — around 12% of the model's context window |
+
+What it costs (measured in daily use at [Harmoniqs](https://harmoniqs.ai), three feeds):
+
+- **~650–980 papers screened per day** — every item in the multi-feed union is scored; the full feed is never truncated.
+- **~470 model calls per digest** (every eligible item is screened by [Jev](https://typesafe.ai), the calibrated ranking model), **median 171 ms** per call, p90 212 ms.
+- **~$0.02/day** at the default budget; the largest arm (a 12 KB taste budget) is still ~$0.08/day. The model's own window is 32k tokens — the largest arm uses about an eighth of it.
+- **105 paper notes, a 64-PDF library**, and a median digest call already running at 99% of the old fixed budget — which is exactly why the budget became a knob.
+
+The honest caveat that shaped this design: the vendor documents *context rot* — accuracy falls as irrelevant state grows. So every enlargement is an **opt-in arm gated on measurement**: `arjev calibrate --replay-arms` re-scores your labeled candidates under each arm and reports Brier, precision@5, and the realized state size per arm, with honest n. Defaults flip only on a measured win, and the tool never edits its own config.
+
 ## The division of labor
 
 Three sentences are the product:
 
 - **The decision model judges.** [Jev](https://typesafe.ai), the ranking model, is text-free by design: it scores, rescues, and rates with calibrated confidence, and it can never write a sentence. Every call is a typed decision, logged as a receipt.
 - **Your agent writes — or your own hand does.** All the prose in the loop (note bodies, why-lines, takes on the picks) comes from an LLM agent or from you. The tool itself never writes prose and never edits an existing note.
-- **The vault stays yours.** Your Obsidian vault is the source of truth: the taste profile is folded from your paper notes (tags, `why` lines, note bodies, wikilink signals), your authored **taste directives** state in plain English what you want, and papers you keep become ground truth. Slack is an optional attention layer on top — the vault is the product.
+- **The vault stays yours.** Your Obsidian vault is the source of truth: the taste profile is folded from your paper notes, your authored **taste directives** state in plain English what you want, and papers you keep become ground truth. Slack is an optional attention layer on top — the vault is the product.
 
 ## Two ways to start
 
@@ -37,17 +73,18 @@ Point it at any Obsidian vault with literature notes (a frontmatter `type` value
 ## The loop, and the staged worklist
 
 1. **Digest** — every day, the full multi-feed union is screened and ranked against the vault's taste profile; each pick carries its why-line.
-2. **Keep** — a Slack reaction, `arjev keep`, or a checkbox scaffolds the paper into your vault as a note with `status: staged` — metadata only, zero taste contribution until prose exists.
+2. **Keep** — a Slack reaction, `arjev keep`, or a checkbox scaffolds the paper into your vault as a note with `status: staged` — metadata only, zero taste contribution until prose exists. The staged gate is exact: an untouched stub provably changes nothing (the fold audits this every digest).
 3. **Write** — `arjev staged` lists the staged notes awaiting prose; your agent (or your hand) writes each note's body and `why` line, then flips `status: staged` → `status: written` in the frontmatter.
-4. **Learn** — a written body feeds taste exactly like a hand-authored note's; `arjev calibrate` replays past digests against your labels and reports Brier scores, reliability, precision@5 — with honest n, never fake confidence.
+4. **Enrich the library** — `arjev fetch <arxiv-id>` downloads the paper's PDF into your library and emits the extracted text alongside it; `arjev backfill` does the whole corpus at once (paced, idempotent, arXiv-polite, converging on libraries that predate it). Library text is what gives each kept paper its abstract and conclusions inside the taste cards — a paper without it still counts, just with less to say. (A digest under the greedy policy also pays a missing PDF's extraction once, writing the `.txt` back — idempotent, same content fetch would write.)
+5. **Learn** — a written body feeds taste exactly like a hand-authored note's; `arjev calibrate` replays past digests against your labels and reports Brier scores, reliability, precision@5 — with honest n, never fake confidence.
 
 ## Why
 
 Keyword-based paper alerts miss everything that isn't spelled the same way ("jitter robustness" never matches "amplitude noise"), never learn from what you actually read, and can't be installed by anyone outside the lab that built them. arJev fixes all three:
 
 - **A decision model at the front line.** With a [Jev](https://typesafe.ai) key (the default), a cheap, calibrated model screens the **full feed** — every item, every day — so papers with zero keyword overlap still surface when they're relevant. Every call is a logged receipt.
 - **The vault still explains every pick.** Your taste profile supplies the matched terms on each why-line and the tiebreak — and it carries the whole digest without a key: no Jev, no outage, no problem; the ranking falls back to the deterministic lexical order and the `mode:` line says exactly which engine ran.
-- **A learning loop.** Keeps scaffold, writes feed taste, calibration reports the truth — the loop above, closing every day.
+- **A learning loop with a budget that scales.** The corpus grows, the context window doesn't — and every enlargement of that window earns its keep through measured calibration, not fashion.
 
 ## Seeding and taste
 
@@ -83,8 +120,12 @@ arJev's core is deliberately **text-free** — deterministic rendering, and the
 
 `arjev calibrate` joins the label ledger, the digest journal, and the decision receipts: Brier per primitive, reliability bins, precision@5. The report states its n, names its censoring (labels only exist for papers you were shown), and recommends thresholds — **you** apply them; the tool never tunes itself.
 
+`arjev calibrate --replay-arms` goes one further: it re-assembles each labeled candidate's context under every state arm — the incumbent fixed budget, the budget-greedy policy at 4/8/12 KB, each with and without candidate conclusions — re-scores them live (conclusions fetched once, paced, from the shared cache), and reports Brier, n, and the realized state size per arm. That report is the only road to changing a default in this tool.
+
+One network disclosure: corpus content resolves from local files only. The candidate-content arm is the one fetch a digest can make — bounded to the finalists, paced at 3 s, cached in the state dir, and fail-open: a dead network degrades that arm to abstract-only and the digest still ships, with the `mode:` line saying what ran.
+
 ## Status
 
-v0.3.0. v1 in daily use at [Harmoniqs](https://harmoniqs.ai). [Issue #1](https://github.com/harmoniqs/arJev/issues/1) is the plan of record (spec, compiled plan, obligation register).
+v0.3.0. v1 in daily use at [Harmoniqs](https://harmoniqs.ai). [Issue #1](https://github.com/harmoniqs/arJev/issues/1) is the plan of record (spec, compiled plan, obligation register); the context engine is [issue #75](https://github.com/harmoniqs/arJev/issues/75).
 
 Apache-2.0.

pyproject.toml

@@ -8,6 +8,7 @@ requires-python = ">=3.11"
 dependencies = [
   "PyYAML>=6.0",
   "httpx>=0.27",
+  "pypdf>=4.0",
 ]
 
 [project.optional-dependencies]

src/arjev/calibrate.py

@@ -14,6 +14,7 @@
 
 from .config import Config
 from .fold import FoldResult
+from .jev import StatePolicy
 from .ledger import Label, LabelLedger
 from .profile import staged_audit
 from .rerank import journal_path, state_dir
@@ -56,7 +57,7 @@ def render(self) -> str:
 def calibrate(cfg: Config, fold: FoldResult, runs: int = 30) -> Report:
     report = Report()
     journal = _load_journal(runs)
-    receipts = _load_receipts()
+    receipts = _latest_receipts(_load_receipts())
     ledger = LabelLedger()
 
     labels_by_id: dict[str, list[Label]] = {}
@@ -201,6 +202,120 @@ def keep_rate(ids: list[str]) -> tuple[float | None, int]:
     }
 
 
+# ── the arms instrument (issue #75: the A/B that gates every default flip) ───────
+
+
+def state_arms() -> list[tuple[str, StatePolicy]]:
+    """The A/B matrix. The hard cap scales WITH the arm — a bigger taste budget
+    under the incumbent 4096 cap would collapse every arm to the same backstopped
+    state and measure nothing (caught in review)."""
+    def greedy(budget: int, content: bool) -> StatePolicy:
+        return StatePolicy(policy="budget-greedy", taste_budget=budget, hard_cap=budget + 4096,
+                           candidate_conclusions=content)
+
+    return [
+        ("fixed-15/4KB", StatePolicy()),
+        ("fixed-15/4KB+content", StatePolicy(candidate_conclusions=True)),
+        ("greedy/4KB", greedy(2500, False)),
+        ("greedy/4KB+content", greedy(2500, True)),
+        ("greedy/8KB", greedy(8000, False)),
+        ("greedy/8KB+content", greedy(8000, True)),
+        ("greedy/12KB", greedy(12000, False)),
+        ("greedy/12KB+content", greedy(12000, True)),
+    ]
+
+
+def calibrate_arms(cfg: Config, fold: FoldResult, client, fetch_metadata, runs: int = 30) -> Report:
+    """Re-assemble and re-score journaled, LABELED candidates under each state arm —
+    the instrument that decides whether budget-greedy and candidate content earn a
+    default flip (context rot is real; evidence, not economics, decides). Honest n
+    is labeled candidates only; arm receipts land in a separate arms file, never
+    the production join. `fetch_metadata`, the client, and the conclusions warm-up
+    are injectable/seamed (CI offline)."""
+    from .jev import assemble_state, write_receipts
+    from .paper_content import corpus_content, paced_conclusions, read_sections_cache, sections_cache_dir
+    from .profile import build_profile
+    from .rerank import state_dir
+
+    report = Report()
+    journal = _load_journal(runs)
+    ledger = LabelLedger()
+    labels_by_id: dict[str, list[Label]] = {}
+    for label in ledger.rows():
+        labels_by_id.setdefault(label.arxiv_id, []).append(label)
+    # journaled candidates that carry a gradeable label — the honest n
+    journal_titles = {c["arxiv"]: c.get("title", "")
+                      for line in journal for c in line.get("candidates", []) if c.get("title")}
+    labeled = {arxiv: outcome for arxiv in journal_titles
+               if (outcome := _positive(labels_by_id.get(arxiv, []))) is not None}
+    if not labeled:
+        report.notes.append(
+            "replay-arms: n = 0 — no journaled candidate carries a gradeable label yet; "
+            "the arms are unmeasured, not refuted (collect keeps/skips first)."
+        )
+        report.metrics["replay_arms"] = {"n": 0}
+        return report
+    meta = fetch_metadata(sorted(labeled))
+    profile = build_profile(fold, cfg, _today(), content=corpus_content(cfg))
+    arms = state_arms()
+    # conclusions are arm-independent: warm the shared cache ONCE, paced — never a
+    # burst per arm
+    if any(policy.candidate_conclusions for _, policy in arms):
+        paced_conclusions(sorted(labeled), sections_cache_dir(), pace_s=cfg.candidate_pace_s)
+    arm_receipts: list = []
+    run_id = f"arms-{_today().isoformat()}"
+    for name, policy in arms:
+        pairs: list[tuple[float, int]] = []
+        state_bytes: list[int] = []
+        for arxiv, outcome in labeled.items():
+            m = meta.get(arxiv) or {}
+            item = _JournalCandidate(title=journal_titles.get(arxiv) or m.get("title") or "",
+                                     authors=[], abstract=m.get("abstract") or "", arxiv=arxiv)
+            conclusions = (read_sections_cache(arxiv).get("conclusions")
+                           if policy.candidate_conclusions else None)
+            assembly = assemble_state(profile, item, policy, conclusions)
+            if assembly.state is None:
+                continue
+            state_bytes.append(assembly.state_bytes)
+            answer = client.score_relevance(assembly.state, arxiv, run_id, arm_receipts)
+            if not answer.ok:
+                continue
+            pairs.append((_p_positive({"primitive": "score", "distribution": answer.distribution}), outcome))
+        report.metrics[f"arm_{name}"] = {
+            "brier": round(sum((p - o) ** 2 for p, o in pairs) / len(pairs), 4) if pairs else None,
+            "n": len(pairs),
+            "mean_state_bytes": round(sum(state_bytes) / len(state_bytes)) if state_bytes else None,
+        }
+    if arm_receipts:
+        write_receipts(state_dir() / "arms", arm_receipts)
+    report.notes.append(
+        f"replay-arms: {len(labeled)} labeled candidate(s) re-scored per arm from the CURRENT "
+        "fold (state assembly is the live corpus, not the historical one — a first "
+        "reading, not a verdict); arm receipts: arms/ under the state dir."
+    )
+    report.notes.append("defaults flip on a measured win here; the tool never edits config.")
+    return report
+
+
+class _JournalCandidate:
+    """assemble_state's candidate seam: it reads .item.{title, authors, abstract, arxiv}."""
+
+    def __init__(self, title: str, authors: list[str], abstract: str, arxiv: str):
+        from types import SimpleNamespace
+
+        self.item = SimpleNamespace(title=title, authors=authors, abstract=abstract, arxiv=arxiv)
+
+
+def _latest_receipts(receipts: list[dict]) -> list[dict]:
+    """The two-pass finalist enrichment appends a second score receipt for re-scored
+    candidates — the enriched call supersedes (it is the judgment the digest acted
+    on), so the calibration join keeps the LAST receipt per (run_id, candidate)."""
+    latest: dict[tuple[str, str], dict] = {}
+    for r in receipts:
+        latest[(r.get("run_id", ""), r["candidate_arxiv"])] = r
+    return list(latest.values())
+
+
 # ── loaders ─────────────────────────────────────────────────────────────────────
 
 def _load_journal(runs: int) -> list[dict]: