Text-format experiment: is there a best layout for a record?

README.md

@@ -12,7 +12,7 @@ Blog and experiment write-ups: [swinecoder101.github.io/hvec](https://swinecoder
 > connectors, seven codec families (`f32`, `int8`, `binary`, `binary-centred`, `pq<m>`, `mrl<k>`
 > and `mrl<k>+<codec>`), `bench recall` against the f32 ranking and against relevance judgments, a
 > BEIR importer, `bench run` across the matrix with a payload stage, `report`, an LLM judge and an
-> agent memory-loop benchmark. Four retrieval experiments are published. The answer-level
+> agent memory-loop benchmark. Ten retrieval experiments are published. The answer-level
 > experiments (two chat models) are scripted and waiting on API access. Expect APIs and results to
 > change without notice. Contributions and experiments are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md).
 

ROADMAP.md

@@ -181,6 +181,17 @@ decide how deep a hybrid re-score must go, and to decide where an adaptive code
   cached buffer is 5.1× faster than the original row scan. Every asymmetric codec costs about
   what f32 costs per vector. `binary-hamming` is 20× cheaper, and with an f32 re-score it reaches
   95% recall@10 on FiQA about 19× faster than exact f32: the first speed claim hvec makes.
+- [x] **Text-format experiment** (`experiments/2026-11-text-format`), with `ingest --whole-files`
+  (one chunk per file, layout kept). The same WikiBio records as prose, bullets, key-value, JSON
+  and a table row: format moves f32 nDCG@10 by up to 2×, and templated prose is last in every
+  context. The best format depends on the model (JSON on bge-small, a table row on MiniLM), not
+  the query type. There is no accuracy-for-speed trade-off: a `binary-hamming` scan plus 100
+  f32 re-scores keeps 0.96–0.99 of the best format's nDCG at 1/19 of the time. Noise ÷ band
+  predicts the depth on a held-out format within a factor of 1.16.
+- [ ] Embed one format and return another (match on key-value, hand the model prose): a payload
+  text per chunk, separate from the text embedded.
+- [ ] Per-query scores from `bench recall`, so format and codec comparisons carry confidence
+  intervals.
 - [ ] Choose the codec automatically at ingest from noise ÷ band, then measure the re-score
   depth for a target recall on a sample of self-queries. The routing experiment showed that the
   curve chooses the codec well but predicts the depth badly.

crates/hvec-bench/src/chunk.rs

@@ -42,6 +42,17 @@ pub fn chunk_text(text: &str, opts: ChunkOptions) -> Vec<String> {
     out
 }
 
+/// The whole of `text` as one chunk, line breaks and spacing kept; only the
+/// leading and trailing whitespace is trimmed. Empty input yields no chunks.
+///
+/// For corpora whose files are already one record each, where the layout is
+/// the thing under test: `chunk_text` would join every line with single spaces.
+#[must_use]
+pub fn chunk_whole(text: &str) -> Vec<String> {
+    let t = text.trim();
+    if t.is_empty() { Vec::new() } else { vec![t.to_owned()] }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -69,4 +80,11 @@ mod tests {
         let chunks = chunk_text("a b c d e", ChunkOptions { words: 2, overlap: 5 });
         assert_eq!(chunks, vec!["a b", "b c", "c d", "d e"]);
     }
+
+    #[test]
+    fn whole_keeps_layout_and_trims_the_ends() {
+        let text = "\n- name: ada\n- born:  1815\n\n";
+        assert_eq!(chunk_whole(text), vec!["- name: ada\n- born:  1815"]);
+        assert!(chunk_whole(" \n\t").is_empty());
+    }
 }

crates/hvec-bench/src/lib.rs

@@ -14,7 +14,7 @@ pub mod record;
 pub mod report;
 pub mod rescore;
 
-pub use chunk::{ChunkOptions, chunk_text};
+pub use chunk::{ChunkOptions, chunk_text, chunk_whole};
 pub use judge::{JUDGE_PROMPT_VERSION, JUDGE_SYSTEM_PROMPT, Verdict, build_judge_prompt, parse_verdict};
 pub use payload::{Payload, PayloadStats};
 pub use prompt::{RAG_SYSTEM_PROMPT, build_user_prompt};

crates/hvec/src/cli.rs

@@ -416,6 +416,10 @@ pub struct IngestArgs {
     /// Overlapping words between consecutive chunks.
     #[arg(long, default_value_t = 40)]
     pub overlap_words: usize,
+    /// Store each file as one chunk, verbatim (line breaks kept), instead of word windows.
+    /// For corpora of one record per file, where the layout is under test.
+    #[arg(long, conflicts_with_all = ["chunk_words", "overlap_words"])]
+    pub whole_files: bool,
     /// File extensions to include when walking directories.
     #[arg(long, value_delimiter = ',', default_value = "md,txt,markdown,rst")]
     pub ext: Vec<String>,

crates/hvec/src/commands/ingest.rs

@@ -2,7 +2,7 @@ use std::path::{Path, PathBuf};
 use std::time::Instant;
 
 use anyhow::{Context as _, Result, bail};
-use hvec_bench::{ChunkOptions, chunk_text};
+use hvec_bench::{ChunkOptions, chunk_text, chunk_whole};
 use hvec_connect::Embedder;
 use hvec_core::{Metric, codec_by_name};
 use hvec_store::{NewChunk, NewCollection};
@@ -72,10 +72,10 @@ pub async fn run(config: &Option<PathBuf>, args: IngestArgs) -> Result<()> {
         codec = existing.codec()?;
     }
 
-    let opts = ChunkOptions {
+    let opts = (!args.whole_files).then_some(ChunkOptions {
         words: args.chunk_words,
         overlap: args.overlap_words,
-    };
+    });
     let batch = args.batch.max(1);
     let started = Instant::now();
     let mut total_chunks = 0usize;
@@ -189,11 +189,15 @@ pub async fn run(config: &Option<PathBuf>, args: IngestArgs) -> Result<()> {
     Ok(())
 }
 
-async fn read_chunks(file: &Path, opts: ChunkOptions) -> Result<Vec<String>> {
+/// Word windows under `opts`, or the whole file as one chunk when `opts` is `None`.
+async fn read_chunks(file: &Path, opts: Option<ChunkOptions>) -> Result<Vec<String>> {
     let text = tokio::fs::read_to_string(file)
         .await
         .with_context(|| format!("reading {}", file.display()))?;
-    Ok(chunk_text(&text, opts))
+    Ok(match opts {
+        Some(opts) => chunk_text(&text, opts),
+        None => chunk_whole(&text),
+    })
 }
 
 async fn embed_all(embedder: &dyn Embedder, pieces: &[String], batch: usize) -> Result<Vec<Vec<f32>>> {

crates/hvec/src/commands/shell.rs

@@ -294,6 +294,7 @@ impl Shell {
                 .unwrap_or_else(|| "f32".to_owned()),
             chunk_words: 200,
             overlap_words: 40,
+            whole_files: false,
             ext: ["md", "txt", "markdown", "rst"]
                 .iter()
                 .map(|s| (*s).to_owned())

crates/hvec/tests/pipeline.rs

@@ -1247,3 +1247,45 @@ fn bench_scan_times_paths_and_codecs_and_keeps_f32_exact() {
     assert!(codecs.iter().all(|c| c["ms_per_query"].as_f64().unwrap() >= 0.0));
     assert_eq!(r["bandwidth"].as_array().unwrap().len(), 2);
 }
+
+#[test]
+fn whole_files_stores_one_chunk_per_file() {
+    let env = Env::new();
+    // 450 words: three overlapping word windows by default, one chunk with --whole-files.
+    let long = (0..150)
+        .map(|i| format!("- field{i}: value{i}"))
+        .collect::<Vec<_>>()
+        .join("\n");
+    let path = env.write_doc("record.md", &long);
+    let path = path.to_str().unwrap();
+
+    let out = env.run(&["ingest", path, "--collection", "windows"]);
+    assert_ok(&out, "ingest windows");
+    assert!(
+        stdout(&out).contains("ingested 3 chunks from 1 files"),
+        "{}",
+        stdout(&out)
+    );
+
+    let out = env.run(&["ingest", path, "--collection", "whole", "--whole-files"]);
+    assert_ok(&out, "ingest whole");
+    assert!(
+        stdout(&out).contains("ingested 1 chunks from 1 files"),
+        "{}",
+        stdout(&out)
+    );
+
+    let out = env.run(&[
+        "ingest",
+        path,
+        "--collection",
+        "both",
+        "--whole-files",
+        "--chunk-words",
+        "50",
+    ]);
+    assert!(
+        !out.status.success(),
+        "--whole-files with --chunk-words should be refused"
+    );
+}

docs/_data/references.yml

@@ -314,6 +314,14 @@
   year: 2024
   url: https://arxiv.org/abs/2407.02883
   note: Code retrieval datasets in BEIR's layout. The source of hvec's CoSQA copy.
+- key: lebret2016wikibio
+  topic: Embeddings and RAG
+  authors: Rémi Lebret, David Grangier, Michael Auli
+  title: Neural Text Generation from Structured Data with Application to the Biography Domain
+  venue: Conference on Empirical Methods in Natural Language Processing (EMNLP)
+  year: 2016
+  url: https://arxiv.org/abs/1603.07771
+  note: WikiBio, Wikipedia infoboxes paired with each article's opening paragraph. The source of the text-format experiment's records.
 
 - key: morris2023vec2text
   topic: Embeddings and RAG

docs/_experiments/text-format.md

@@ -0,0 +1,90 @@
+---
+title: "Text format: is there a best layout for a record?"
+date: 2026-11-05 21:00:00 +0000
+result: "With the facts held fixed, layout moves f32 nDCG@10 by up to 2×, and templated prose is last in every context. The best format depends on the embedding model (JSON on bge-small, a table row on MiniLM), not the query type. Its advantage survives compression: a binary-hamming scan plus 100 f32 re-scores keeps 0.96–0.99 of it at 1/19 of the time."
+references: [lebret2016wikibio, xiao2023bge, reimers2019sbert, charikar2002simhash, jegou2011pq]
+---
+
+*Full write-up, raw results and a one-command reproduction are in the repository:*
+[experiments/2026-11-text-format](https://github.com/SwineCoder101/hvec/tree/main/experiments/2026-11-text-format).
+Follow-up to [neighbour spacing]({{ "/experiments/neighbour-spacing/" | relative_url }}) and
+[zero-copy scan]({{ "/experiments/zero-copy-scan/" | relative_url }}).
+
+Every earlier experiment held the text fixed and changed the codec. This one holds the facts
+fixed and changes only their layout. The question: for a given embedding model and kind of
+query, which format and fast path give the best nDCG@10 within a time budget?
+
+**Format can only change speed indirectly.** The scan never sees text. With one record per chunk,
+every format has the same vector count and the same scan cost. Format can only move time through
+neighbour spacing, which sets how many f32 re-scores a compressed scan needs. It also changes how
+many tokens a retrieved record costs the chat model.
+
+**Hypothesis.**
+1. The best format depends on the query type. Field-labelled layouts (key-value, JSON, table)
+   win attribute lookups at f32. But they repeat the same keys in every record, which packs
+   neighbours closer, so they lose at a tight time budget.
+2. Format acts on compressed retrieval only through f32 nDCG and noise ÷ band. The depth curve
+   fitted on some formats predicts the depth on a held-out one.
+
+**Setup.** 20,000 WikiBio [1] biographies, each written seven ways.
+- **Five controlled formats** with identical facts, the cleaned infobox: templated prose, bullets,
+  key-value, JSON, and a two-row table.
+- **Two references:** the article's own opening paragraph, and a key-value header above it.
+
+There are 500 lookup queries ("who was born on 25 april 1985 and played as guard / offensive
+tackle?") and 500 descriptive ones ("people who died in columbus , ohio"), the same for every
+format and judged from the infobox. Each record is one chunk, verbatim, using the new
+`ingest --whole-files`. The models are bge-small [2] and MiniLM [3], 384 dims. The fast path is
+a `binary-hamming` scan [4] (`pq48` [5] for comparison) with an f32 re-score, timed from measured
+per-vector costs on one thread. Apple M4 Pro.
+
+**Result.** Hypothesis 1 is refuted on both counts. Hypothesis 2 is confirmed.
+
+Exact f32 nDCG@10:
+
+| format | tokens | bge lookup | bge descriptive | MiniLM lookup | MiniLM descriptive |
+|---|---:|---:|---:|---:|---:|
+| templated prose (baseline) | 114 | 0.370 | 0.174 | 0.179 | 0.130 |
+| bullets | 75 | 0.402 | 0.180 | 0.336 | 0.161 |
+| key-value | 74 | 0.435 | 0.191 | 0.276 | 0.143 |
+| JSON | 114 | **0.441** | **0.224** | 0.240 | 0.142 |
+| table row | 119 | 0.415 | 0.211 | **0.353** | **0.181** |
+| Wikipedia paragraph (reference) | 74 | 0.181 | 0.113 | 0.098 | 0.091 |
+| key-value header + paragraph (reference) | 147 | 0.370 | 0.158 | 0.208 | 0.136 |
+
+Every structured layout beats prose in every context. The model, not the query type, decides
+which one wins: JSON on bge-small, the table row on MiniLM, where JSON is only fourth.
+
+The predicted speed penalty is too small to matter. Within a context, format changes the
+neighbour band by at most 1.3×, while the model changes it by about 2×. A `binary-hamming`
+scan plus 100 f32 re-scores takes 0.22 ms against about 4.1 ms for exact f32 over 20,000
+vectors.
+
+| context | best format | f32 nDCG | hybrid nDCG (share kept) | prose f32 nDCG | prose hybrid (share kept) |
+|---|---|---:|---:|---:|---:|
+| bge-small, lookup | JSON | 0.441 | 0.423 (0.959) | 0.370 | 0.335 (0.906) |
+| bge-small, descriptive | JSON | 0.224 | 0.219 (0.977) | 0.174 | 0.169 (0.976) |
+| MiniLM, lookup | table | 0.353 | 0.349 (0.987) | 0.179 | 0.170 (0.948) |
+| MiniLM, descriptive | table | 0.181 | 0.179 (0.992) | 0.130 | 0.126 (0.976) |
+
+The fast path on the best format beats exact f32 on prose in every context, at 1/19 of the time.
+
+With one format left out at a time, noise ÷ band ranks the re-score depth at Spearman 0.985. It
+predicts the held-out format's depth for 95% recall within a factor of 1.16, tighter than across
+corpora (1.25–1.41). Together with f32 nDCG, it predicts a hybrid's nDCG on the held-out format
+within 0.003–0.007.
+
+**Reading.** Choose the format for the model. Measure f32 nDCG on a sample in two or three
+layouts; the compressed cost follows from spacing. Key-value and bullets are about 35% cheaper in
+tokens than JSON and table, and within 0.02 of the best on lookups. Putting a key-value header
+above a prose document doubles its nDCG.
+
+Caveats:
+- With 500 queries per type, gaps under about 0.02 are ties.
+- The queries are templated, and the judgments match infobox values exactly, so absolute nDCG is
+  low.
+- The domain is biographies only.
+- Only retrieval is measured, not answers.
+- Hybrid times are computed from per-vector costs, with the candidates in memory.
+
+{% include references.html %}