Repository navigation
Text-format experiment: is there a best layout for a record? - #3
Open
SwineCoder101 wants to merge 4 commits into
Open
SwineCoder101 wants to merge 4 commits into
SwineCoder101 wants to merge 4 commits into
Conversation
chunk_text splits on whitespace and joins with single spaces, so a file's line breaks are gone before it is embedded or stored. --whole-files stores each file verbatim as one chunk (ends trimmed), for corpora of one record per file where the layout is under test. It conflicts with --chunk-words and --overlap-words. Tests: chunk_whole keeps line breaks; a 450-word file ingests as three windows by default and one chunk with the flag.
The same 20,000 WikiBio records as templated prose, bullets, key-value, JSON and a table row (identical facts), plus the article's own paragraph and a key-value header over it, against 500 lookup and 500 descriptive queries from the infobox. For each format, model and query type: f32 nDCG@10, neighbour band and rank noise, re-score depth and nDCG behind binary-hamming and pq48, and scan cost per vector. summarise.py finds the best format within 10-100% of an exact f32 scan's time and tests whether noise / band predicts depth and hybrid nDCG on a held-out format.
With the facts held fixed, layout moves exact f32 nDCG@10 by up to 2x (MiniLM lookups: templated prose 0.179, table row 0.353), and templated prose is last of the five controlled formats in all four contexts. The best format depends on the model, not the query type: JSON on bge-small (0.441 lookup, 0.224 descriptive), the table row on MiniLM (0.353, 0.181). No accuracy-for-speed trade-off: within a context format changes the neighbour band by at most 1.3x, and a binary-hamming scan plus 100 f32 re-scores (0.22 ms against 4.1 ms) keeps 0.96-0.99 of the best format's nDCG, beating exact f32 on prose everywhere. Noise / band predicts the re-score depth on a held-out format within a factor of 1.16 at 95% (Spearman 0.985), and with f32 nDCG predicts hybrid nDCG within 0.003-0.007.
Adds the experiment page, the WikiBio reference (Lebret, Grangier and Auli, EMNLP 2016, checked against arXiv:1603.07771), and corrects the README's count of published experiments from four to ten.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The same 20,000 WikiBio records written seven ways. Five controlled formats hold identical facts: templated prose, bullets, key-value, JSON and a table row. Two references: the article's own paragraph, and a key-value header above it. Tested against 500 lookup and 500 descriptive queries, on bge-small and MiniLM.
Results
binary-hammingscan plus 100 f32 re-scores (0.22 ms against 4.1 ms) keeps 0.96–0.99 of the best format's nDCG. It beats exact f32 on prose in every context.Changes
hvec ingest --whole-files: one chunk per file, verbatim, line breaks kept. It conflicts with--chunk-wordsand--overlap-words. Covered by a unit test and a pipeline test.experiments/2026-11-text-format/: the WikiBio converter,run.sh,summarise.py, the write-up and the results (640 KB). The run took about an hour on an M4 Pro and used no API calls.docs/_experiments/text-format.mdand the WikiBio reference.Caveats