Skip to content

fix(assembly-strategies): stop reassembling whole doc for USFM-less book pairs - #399

Merged
linearcombination merged 5 commits into
WycliffeAssociates:doc-dev.walink.orgfrom
linearcombination:DOC/fm/doc-assembly-strategies-quadratic-spike
Nov 13, 2026
Merged

linearcombination merged 5 commits into
WycliffeAssociates:doc-dev.walink.orgfrom
linearcombination:DOC/fm/doc-assembly-strategies-quadratic-spike

Conversation

@linearcombination

@linearcombination linearcombination commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Closes #363

The PO previously asked by verse interleaving to be disabled for now, so this is not a bug that ever presented to a user. The bug was discovered as part of a larger performance audit directed toward computational complexity issues in the assembly logic.

Intent

Fix the duplicate-content bug found by the assembly_strategies audit (report at /data/ai-workspace/firstmate/data/doc-assembly-strategies-quadratic-spike/report.md, Finding 1): in backend/doc/domain/assembly_strategies/assemble_by_book.py's assemble_content_by_verse_book_at_a_time (lines 194-318, the recursive call at lines 299-317), the else branch - taken whenever a (lang_code, book_code) pair has no matching USFM resource, a routine case when a gateway-language-notes-only language (TN/TQ/BC) is requested alongside a fully-translated language for the same books - calls assemble_content_by_book with the full, unfiltered original resource lists instead of just this pair, causing the whole document to be reassembled from scratch. Nothing downstream dedupes the result, so book introductions and other content are duplicated in documents actually delivered to users. Only this specific fix is authorized; the report's Findings 2-4 (separate performance issues) are explicitly out of scope for this task and will be handled as follow-up work.

What Changed

  • In assemble_content_by_verse_book_at_a_time, replaced the else branch (hit when a (lang_code, book_code) pair has no matching USFM resource) that called assemble_content_by_book with the full, unfiltered resource lists — which reassembled the entire document from scratch and duplicated already-emitted content like book introductions.
  • Replaced it with targeted per-pair dispatch: assemble_tn_by_book when TN or TNC content exists for the pair, assemble_tq_by_book when only TQ exists, and assemble_tw_by_book when only TW, BC, or RG content exists — each scoped to just that book pair's resources instead of the whole resource set.
  • Added tests/unit/test_assemble_by_book.py covering the new branch selection logic for assemble_content_by_verse_book_at_a_time.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change is a minimal, well-scoped fix that replaces a buggy full-reassembly recursive call with direct calls to existing per-pair helper functions, exactly mirroring an already-established pattern in the sibling assemble_content_by_book function, with matching signatures and a behavioral regression test.

Testing

Ran the existing targeted regression test tests/unit/test_assemble_by_book.py, confirming it passes against the fix (target commit) and, after temporarily swapping in the pre-fix version of the source file, fails in exactly the way the report describes (duplicated book intro content) -- demonstrating this is a genuine before/after regression test for Finding 1, with no unrelated scope creep and a clean worktree afterward.

Evidence: pytest output on target commit (7f45b5d) - test passes
tests/unit/test_assemble_by_book.py::test_assemble_content_by_verse_book_at_a_time_does_not_duplicate_books_when_one_book_lacks_usfm PASSED [100%]
1 passed in 0.83s
Evidence: same test run against pre-fix code (base commit 1ce2c97) - reproduces the reported bug
AssertionError: assert 2 == 1
where 2 = count_parts_containing('Intro for gen')
(book intro duplicated because the else-branch reassembled the whole document from unfiltered lists)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • PYTHONPATH=backend /tmp/scoutvenv/bin/pytest tests/unit/test_assemble_by_book.py -q on target commit 7f45b5d7 -- passes
  • Same test run against the pre-fix version of backend/doc/domain/assembly_strategies/assemble_by_book.py (base commit 1ce2c973), temporarily swapped in and restored afterward -- fails with assert 2 == 1 on 'Intro for gen', reproducing the exact duplication described in the report before restoring the target file and confirming a clean worktree
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Developer rewrote one of the generated tests that was failing and run both backend and frontend test suites with success.

The fallback branch, taken whenever a (lang_code, book_code) pair had no
matching USFM resource (routine when a gateway-language notes-only
language is requested alongside a fully-translated one for the same
books), called assemble_content_by_book with the entire, unfiltered book
lists instead of just that one book/language. That reprocessed the whole
document from scratch on every such miss, duplicating already-assembled
book intros and other content in generated documents.

Dispatch on the already-selected tn_book/tq_book/tw_book/bc_book
for just this pair instead, matching the non-USFM branches
assemble_content_by_book already uses one function above.
tests/unit/test_assemble_by_book.py (added in the previous commit)
collided with the pre-existing tests/e2e/doc/test_assemble_by_book.py:
neither directory had an __init__.py, unlike tests/, tests/shared/, and
tests/e2e/, which do, so pytest assigned both files the same bare module
name and errored with "import file mismatch" during collection.

Add the two missing empty __init__.py files to match the existing
convention, rather than renaming the new test file.
Reviewer's guide was removed a while back from DOC after this branch
was originally drafted.
@linearcombination
linearcombination merged commit b9063ec into WycliffeAssociates:doc-dev.walink.org Nov 13, 2026
18 of 22 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Done in DOC Project Management Nov 13, 2026
@linearcombination
linearcombination deleted the DOC/fm/doc-assembly-strategies-quadratic-spike branch November 18, 2026 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants