Repository navigation
fix(backfill): canonical API abstracts for the whole corpus; merge, not override - #80
Merged
Merged
Conversation
…ot override The live backfill exposed it: pypdf text usually drops the standalone 'Abstract' heading (a 12-text sample resolved 1 abstract against 7 conclusions), and corpus_content let a library resolution override the cached canonical abstract. The abstract of record is the arXiv API's: backfill stamps it into the sections cache for every corpus id still lacking one (one batched call — an idempotent re-run does zero network), and corpus_content now merges per field — the cache's abstract wins, the library's conclusions win, neither discards the other. Closes #79
This was referenced Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #79.
Found running the production backfill on v0.4.0 (103 library texts resolved to only 50 papers with content, 17 with abstracts):
Abstractheading.backfillstamps the API abstract into the sections cache for every corpus id still lacking one; an idempotent re-run does zero network (verified by test).corpus_contentpreviously let a library resolution replace the cache entry — a library hit with conclusions but no abstract silently discarded the canonical abstract. Per-field merge now: cache abstract wins, library conclusions win.Suite + ruff green (3 new tests: whole-corpus stamping, zero-network idempotence, merge precedence).