Repository navigation
feat(jev): the context engine — budget-greedy state assembly with paper-content taste cards - #76
Merged
Merged
Conversation
…er-content taste cards The corpus outgrew the fixed-15 state (production median: 4055 of 4096 bytes), while the papers' own words never reached the model — bodies were compressed to token frequencies. One mechanism replaces the fixed counts (issue #75): - budget-greedy assembly: directives whole, then taste cards (why + abstract + conclusions excerpts, decay-weighted) and terms fill the taste budget greedily — small corpus fits everything, growing corpus gives recent work proportionally more of the window - paper_content: tolerant section extraction over the library ladder (.txt → PDF → API abstract floor), candidate conclusions cached in the state dir (the library stays keep-only), finalist re-score under the content arm - arjev backfill: paced corpus PDF+text backfill, idempotent - calibrate: receipt dedup (the enriched call supersedes) + --replay-arms, the A/B that gates every default flip (context rot is real; evidence decides) — defaults unchanged until a measured win
…at scale, honest numbers Two criticals: a transport error in a candidate fetch crashed the digest (caught in candidate_conclusions now — a dead proxy degrades to None, never kills a digest); and the >4KB arms kept the 4096 hard cap, collapsing every arm to the same backstopped state — the A/B would have measured nothing (state_arms scales hard_cap with the budget; a 12KB arm joins the matrix). Importants: the arms test is hermetic again (cache pre-populated, warm-up paced and shared); fetch_pdf's exists-path pays extraction once so backfill converges on pre-feature libraries; README numbers now survive contact with the code (12KB ≈ 12% of the 32k-token window, ~$0.08/day, the 12KB arm real). Minors: a failed finalist re-score no longer supersedes the journaled receipt; slash-form ids slug to flat cache filenames with the real id inside the JSON; the mode line says finalists vs candidates per ranking mode; the paced_conclusions seam unifies politeness across digest and replay-arms; corpus-content and library write-back disclosures land in the README.
aarontrowbridge
marked this pull request as ready for review
October 8, 2026 10:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #75.
What
The corpus had outgrown the Jev state (production receipts show the median digest call at 4055 of 4096 bytes) while the papers' own words never reached the model. This PR replaces the fixed-15 counts with one mechanism: a budget-greedy, decay-ranked state assembly — directives whole, then per-paper taste cards (your why + the paper's abstract + its conclusions/outlook) and terms fill the configured budget by recency-weighted priority. Small corpus → everything rides; growing corpus → recent work claims proportionally more of the window.
Surface
paper_content.py: tolerant section extraction (Conclusion(s)/Summary/Outlook/Discussion/Perspectives, numbered or not) over the library ladder (.txt→ PDF extraction → arXiv API abstract floor). Fail-open everywhere; digest-time reads are local-only.arjev backfill: paced, idempotent fetch of every corpus paper's PDF + extracted text;arjev fetchnow emits the.txtat fetch time.arjev calibrate --replay-arms: the live A/B (fixed-15 · greedy@4KB · greedy@8KB × content on/off) over labeled candidates, Brier per arm with honest n; arm receipts land in a separate file, never the production join.state_policy,taste_budget_bytes,candidate_content. README rewritten around the context-management story (generic capacity math + the measured production block).Invariants kept
fixed-15, no candidate content) — every enlargement is opt-in untilcalibrate --replay-armsshows a measured win (the vendor documents context rot; evidence decides).pypdf(pure-Python, PDF text extraction at the fetch/backfill seams).Verification
test_paper_content.py+test_context_engine.py);ruff checkclean.