Skip to content

fix(chunks): fire heading-aware split-before discount on glued headings; v0.6.2 - #7

Merged
emerose merged 1 commit into
mainfrom
chunking-boundary-tuning
May 29, 2026
Merged

emerose merged 1 commit into
mainfrom
chunking-boundary-tuning

Conversation

@emerose

@emerose emerose commented May 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • Latent bugfix in SPEC-CHUNK-322. The "encourage split before a heading" discount keyed on _is_heading (a standalone heading chunklet). Stage 2 almost always glues a heading to its first sentences, so that predicate is rarely true on real prose — the discount was effectively dead and chunks packed across section boundaries.
  • The split-before discount now keys on a new _starts_with_heading predicate (first block token is a heading). The split-after forbid stays on _is_heading, so a lone heading is still never stranded at a chunk's tail.
  • Bumps to 0.6.2 and updates the spec + changelog.

Before / after (Qasper coherence set)

Paper Before After Effect
1908.05925 12 chunks, 9 start at a heading 14 chunks, 13 start at a heading removes a stranded glued-heading tail; the 1646c equation chunk is preserved byte-identical
1909.13375 17 chunks 19 chunks fixes a mid-sentence tail and a chunk that packed past ## Model ::: Objective and Training; formula stays intact
1910.07601 16 chunks 16 (unchanged) n/a

Every new cut lands on a section boundary; smallest new chunk is 784c (above the ~410c general-term floor), so no tiny stubs.

What this deliberately does NOT do

  • No cross-heading penalty (over-split in testing — flat ## rendering can't distinguish major sections from siblings).
  • No under-target penalty — small chunks are fine when the document is genuinely structured that way; short-but-coherent sections are left intact.

Test plan

  • uv run pytest — 214 passed, 20 skipped
  • ruff + pyright clean
  • New test_split_before_glued_heading_with_body proven to fail on the old keying and pass with the change
  • Equation-keeping, stranded-heading prevention, tiny-stub avoidance all still hold

🤖 Generated with Claude Code

…gs; v0.6.2

SPEC-CHUNK-322's split-before discount keyed on _is_heading (a
standalone heading chunklet), which stage 2 almost never produces
because it groups a heading with its first sentences. The discount was
effectively dead on real prose, so chunks packed across section
boundaries. Key the split-before discount on a new _starts_with_heading
predicate; keep the split-after forbid on standalone headings so a lone
heading is still never stranded at a chunk's tail.

On the Qasper coherence set this aligns more chunk starts with section
boundaries (1908.05925: 9/12 -> 13/14 chunks) and removes glued-heading
tails, without forcing extra cuts inside a section, splitting equations,
or introducing tiny stubs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@emerose
emerose merged commit a73e267 into main May 29, 2026
2 checks passed

This branch was previously deployed

1 inactive deployment
pypi — ada9fa8b Deployed May 29, 2026 by emerose via publish #10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant