Skip to content

Text-format experiment: is there a best layout for a record? - #3

Open
SwineCoder101 wants to merge 4 commits into
mainfrom
experiment/text-format
Open

SwineCoder101 wants to merge 4 commits into
mainfrom
experiment/text-format

Conversation

@SwineCoder101

Copy link
Copy Markdown
Owner

The same 20,000 WikiBio records written seven ways. Five controlled formats hold identical facts: templated prose, bullets, key-value, JSON and a table row. Two references: the article's own paragraph, and a key-value header above it. Tested against 500 lookup and 500 descriptive queries, on bge-small and MiniLM.

Results

  • Layout matters more than any codec here. With the facts held fixed, it moves exact f32 nDCG@10 by up to 2× (MiniLM lookups: prose 0.179, table 0.353). Templated prose is last in every context.
  • The model decides the format, not the query type. JSON is best on bge-small, the table row on MiniLM.
  • There is no accuracy-for-speed trade-off. A binary-hamming scan plus 100 f32 re-scores (0.22 ms against 4.1 ms) keeps 0.96–0.99 of the best format's nDCG. It beats exact f32 on prose in every context.
  • Noise ÷ band predicts the re-score depth on a held-out format within a factor of 1.16 at 95% recall (Spearman 0.985).

Changes

  • hvec ingest --whole-files: one chunk per file, verbatim, line breaks kept. It conflicts with --chunk-words and --overlap-words. Covered by a unit test and a pipeline test.
  • experiments/2026-11-text-format/: the WikiBio converter, run.sh, summarise.py, the write-up and the results (640 KB). The run took about an hour on an M4 Pro and used no API calls.
  • Site: docs/_experiments/text-format.md and the WikiBio reference.
  • README: the count of published experiments, corrected from four to ten.
  • Roadmap: the entry for this experiment, plus two follow-ups (separate embedded and returned text, per-query scores for confidence intervals).

Caveats

  • With 500 queries per type, gaps under about 0.02 are ties.
  • The queries are templated and the judgments are exact-match, so absolute nDCG is low.
  • The domain is biographies only, and only retrieval is measured.

chunk_text splits on whitespace and joins with single spaces, so a file's line breaks are gone before it is embedded or stored. --whole-files stores each file verbatim as one chunk (ends trimmed), for corpora of one record per file where the layout is under test. It conflicts with --chunk-words and --overlap-words. Tests: chunk_whole keeps line breaks; a 450-word file ingests as three windows by default and one chunk with the flag.
The same 20,000 WikiBio records as templated prose, bullets, key-value, JSON and a table row (identical facts), plus the article's own paragraph and a key-value header over it, against 500 lookup and 500 descriptive queries from the infobox. For each format, model and query type: f32 nDCG@10, neighbour band and rank noise, re-score depth and nDCG behind binary-hamming and pq48, and scan cost per vector. summarise.py finds the best format within 10-100% of an exact f32 scan's time and tests whether noise / band predicts depth and hybrid nDCG on a held-out format.
With the facts held fixed, layout moves exact f32 nDCG@10 by up to 2x (MiniLM lookups: templated prose 0.179, table row 0.353), and templated prose is last of the five controlled formats in all four contexts. The best format depends on the model, not the query type: JSON on bge-small (0.441 lookup, 0.224 descriptive), the table row on MiniLM (0.353, 0.181). No accuracy-for-speed trade-off: within a context format changes the neighbour band by at most 1.3x, and a binary-hamming scan plus 100 f32 re-scores (0.22 ms against 4.1 ms) keeps 0.96-0.99 of the best format's nDCG, beating exact f32 on prose everywhere. Noise / band predicts the re-score depth on a held-out format within a factor of 1.16 at 95% (Spearman 0.985), and with f32 nDCG predicts hybrid nDCG within 0.003-0.007.
Adds the experiment page, the WikiBio reference (Lebret, Grangier and Auli, EMNLP 2016, checked against arXiv:1603.07771), and corrects the README's count of published experiments from four to ten.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant