Neighbour spacing, hybrid and adaptive quantization, routing, and zero-copy scans

README.md

@@ -99,12 +99,15 @@ limit:
 | Claim | What hvec measures | What that does not show |
 |---|---|---|
 | **Less memory at comparable quality** | Bytes per vector, and nDCG@k and document recall@k against the labels, next to the f32 ceiling | Anything about speed |
-| **Faster search** | Scan time of the compressed-domain score over the whole collection | Not yet claimable: hvec scores every codec asymmetrically against a float query in a brute-force scan, so binary is no faster than int8 here, and nothing in the harness is memory-bandwidth bound. A speed claim needs a symmetric (Hamming, lookup-table) path and a setting where bytes read is the bottleneck |
+| **Faster search** | Scan time per query over the whole collection, in place in a contiguous buffer (`bench scan`), against the machine's memory bandwidth | Only the symmetric `binary-hamming` scan is faster: every asymmetric codec costs about what f32 costs per vector, because its scoring, not its bytes, sets the time. Measured on one machine, brute force, with the hybrid's re-score cost computed rather than timed end to end |
 | **Useful operations directly on the compressed vectors** | Every codec's `score` runs without decoding, and is tested to equal the metric on the decoded vector | Whether the operation is cheap; see the speed row |
 
 Where the published numbers stand on the first claim: `int8` is 4× smaller and indistinguishable
 from f32 against the labels; `pq48` is 32× smaller at two points of nDCG@10; `binary` is 32×
-smaller at 4.4 points. On the second claim hvec makes no claim yet. The useful advantage a new
+smaller at 4.4 points. On the second claim: a `binary-hamming` scan plus an f32 re-score of 369
+candidates reaches 95% recall@10 on FiQA about 19× faster than an exact f32 scan on one thread,
+and is the only compressed path that is faster at all
+([zero-copy scan](experiments/2026-11-zero-copy-scan/)). The useful advantage a new
 codec has to demonstrate is over *those* numbers, on the same corpus, with the byte count stated.
 
 ### 2. Does the loss reach the answer?
@@ -339,6 +342,11 @@ parameters alongside the collection, and reuses them for every later ingest, so
 call a representative sample of the corpus rather than a single file. `mrl<k>` keeps the first
 `k` dimensions (what Matryoshka-trained models are built for) and composes with any other codec:
 `mrl256+int8`, `mrl128+pq16`.
+`binary-hamming` stores the same bits as `binary` but binarises the query too and scores by XOR
+and popcount: coarser, and about 20× faster to scan. `mixed<B>` is trained: it spends a fixed
+`B` bytes per vector with bits allocated per
+dimension by variance. `mixed<B>-pca` rotates into the corpus's principal components first, and
+`-w` weights each component by how much energy queries put into it (see below).
 
 With a question set that carries relevance judgments, `bench recall --questions` also reports
 nDCG@k and document recall@k against the labels, for every codec and for the f32 ceiling:
@@ -373,8 +381,67 @@ recovers only four or five points of recall: one bit per dimension cannot separa
 whose cosines differ by less than its resolution, whatever the origin is. Against the human
 relevance judgments the picture is gentler: binary loses 4.4 points of nDCG@10 (0.698 to 0.654),
 not 39, because the neighbours it reorders are near-ties that are rarely the judged abstract.
-The f32 scan is slow only because the baseline decodes each vector; the other codecs score the
-bytes directly. The experiments are written up under [`experiments/`](experiments/).
+The scan times in that table predate the in-place f32 kernel, when f32 decoded every vector into
+a new allocation. `bench scan` is now the place to time scans: it scores every codec in place in
+a contiguous buffer and reports GB/s against the machine's memory bandwidth
+([zero-copy scan](experiments/2026-11-zero-copy-scan/)). The experiments are written up under
+[`experiments/`](experiments/).
+
+#### Hybrid and adaptive quantization
+
+A small code scanned over every vector loses neighbours, and how many it loses is predictable:
+it is set by the code's score noise divided by how tightly the corpus packs a query's
+neighbours (the [neighbour-spacing experiment](experiments/2026-11-neighbour-spacing/)). There
+are two ways to win the loss back.
+
+**Hybrid.** Keep the small code in memory for the scan, shortlist its best N candidates, and
+re-score only those with int8 or f32 vectors fetched from disk:
+
+<p align="center">
+  <img src="docs/assets/animations/hybrid-rescoring.svg" width="820" alt="Animated schematic of two-stage retrieval: every vector is scored from a 48-byte code; the code's top 10 keeps 6 of the 10 true neighbours; the code's top 30 are re-scored with exact f32 vectors, lifting 9 of the 10 true neighbours into the top 10, while one ranked 31st by the code falls outside the shortlist and is missed.">
+</p>
+
+```sh
+hvec bench rescore --collection scifact --questions questions/scifact.jsonl \
+  --codecs binary,pq48 --rescore int8,f32 --depths 20,50,100
+```
+
+Across ten collections (five corpora, two embedding models), re-scoring 50 f32 candidates
+behind a 48-byte scan recovers 0.95 (`binary`) to 0.99 (`pq48`) of the exact top 10 and
+essentially all of the nDCG loss. That costs 3–4% of the bytes a full f32 scan reads. The depth
+needed follows the same noise-to-spacing ratio (Spearman ρ ≈ 0.93), so one f32 pass over a
+corpus tells you how deep to go:
+[two-stage re-scoring](experiments/2026-11-two-stage-rescoring/).
+
+**Adaptive depth.** A fixed N is too deep for some queries and too shallow for others. The
+`--margins` policy re-scores every candidate whose code score is within λσ of the code's own
+10th score, where σ is the code's noise, calibrated once per collection on self-queries:
+
+<p align="center">
+  <img src="docs/assets/animations/adaptive-depth.svg" width="820" alt="Animated schematic of adaptive re-scoring depth: two queries share the same noise-based margin below their 10th code score; a crowded query has 59 candidates inside the margin and a well-separated query 18, while a fixed shortlist of 30 would under-fetch for the first and over-fetch for the second.">
+</p>
+
+Measured exactly at the same mean depth, the margin policy adds only 0.3 to 0.6 points of
+recall@10 over a fixed depth, but up to 4 points for the worst decile of queries. A hindsight
+oracle shows about 6 points of headroom, almost none of which is visible from the scan's own
+scores ([routing experiment](experiments/2026-11-routing-oracle/)).
+
+**Adaptive bits.** Spend the scan code's bytes unevenly, where they buy the most. This is the
+idea behind mixed-precision quantization of mixture-of-experts models, where blocks and experts
+that matter more get more bits ([QuantMoE-Bench](https://github.com/UNITES-Lab/moe-quantization)).
+For embeddings, "matters more" means a principal component with more variance, or with more of
+the query's energy:
+
+<p align="center">
+  <img src="docs/assets/animations/adaptive-bits.svg" width="820" alt="Animated bar chart of how a 48-byte budget is spread over 384 principal components of SciFact's bge-small embeddings: uniform binary gives every component 1 bit (recall@10 0.611); variance-based allocation concentrates bits on the leading components and gives 97 none (0.694); query-weighted allocation gives 158 none and returns a few bits to the last components, which carry 28% of the corpus mean (0.703).">
+</p>
+
+On SciFact at 48 bytes with no re-scoring, `mixed48-pca-w` keeps 0.703 of the top 10, against
+0.611 for `binary` and 0.711 for `pq48`. The full comparison, every scan codec against every
+re-score policy, is the [hybrid × adaptive matrix](experiments/2026-11-hybrid-adaptive-matrix/).
+The animations are generated by
+[`docs/assets/animations/make_animations.py`](docs/assets/animations/make_animations.py); the
+bit-allocation chart uses SciFact's real PCA spectrum and allocations.
 
 ### 7. Run the matrix and report
 

ROADMAP.md

@@ -107,7 +107,9 @@ rotation is a `d × d` matrix fitted on the corpus.
   `rot+pq48`, `mrl256+rot+binary`. It is a trained codec in the existing sense: `fit` learns the
   rotation on the first ingest, `params` stores the matrix (or the seed) with the collection,
   and `score` rotates the f32 query once per search before the asymmetric scan, so the stored
-  side still scores on the bytes.
+  side still scores on the bytes. The hook for that now exists (`Codec::prepare_query` and
+  `score_prepared`, added in phase 8), and `mixed<B>-pca` already carries a PCA fit that
+  `rot:itq` can start from.
 - [ ] Two fits behind the same name. `rot` with a seed is a random orthogonal rotation
   (RaBitQ-style, a Haar-random matrix from the QR of a Gaussian), which needs no corpus pass and
   gives a baseline for how much the learned one buys. `rot:itq` is Iterative Quantization: PCA
@@ -127,6 +129,78 @@ rotation is a `d × d` matrix fitted on the corpus.
 Exit: Experiment 4 published, and a stated answer to whether rotation makes binary usable on
 this embedding model.
 
+## Phase 8 · Spacing, hybrid and adaptive quantization, routing, zero-copy (done; follow-ups open)
+
+Every codec number before this phase came from one corpus, so none of them said what a codec
+would cost elsewhere. This phase finds the quantity that sets the cost, then uses it twice: to
+decide how deep a hybrid re-score must go, and to decide where an adaptive code spends its bits.
+
+- [x] Neighbour spacing in `bench recall`: `neighbour_band` (exact score gap from rank k to 2k),
+  `boundary_gap` (k to k + 1) and `rank_noise` (the codec's score-error spread over the exact
+  top 2k, per-query bias removed).
+- [x] **Neighbour-spacing experiment** (`experiments/2026-11-neighbour-spacing`): SciFact,
+  NFCorpus, ArguAna, FiQA and CoSQA (code, via CoIR) × bge-small and MiniLM × nine codecs.
+  Noise ÷ band ranks recall@10 loss with ρ = 0.98 over 90 points (noise alone 0.64) and predicts
+  a held-out corpus within 0.024. Content type matters only through spacing, and the embedding
+  model moves spacing as much as the corpus does. Published with a site page and a post.
+- [x] `hvec bench rescore`: scan with one codec, re-score candidates with another (int8 or f32),
+  and report recall, worst-decile recall, nDCG, depth and bytes read per query. It also reports
+  the exact depth an f32 re-score needs for 90/95/99/100% recall.
+- [x] **Two-stage re-scoring experiment** (`experiments/2026-11-two-stage-rescoring`): the
+  depth needed tracks the same ratio (ρ ≈ 0.93; depth ≈ 4.8·k·r^1.29 for 95% recall). At 48
+  bytes, 50 f32 candidates recover 0.95 (`binary`) to 0.99 (`pq48`) of the top 10 and nearly all
+  of the nDCG loss, for 3–4% of an f32 scan's bytes. An int8 second stage stops at int8's own
+  0.990.
+- [x] Adaptive depth: a `--margins` policy that re-scores every candidate within λσ of the
+  scan's own 10th score, with σ calibrated on self-queries only.
+- [x] Adaptive bits: `mixed<B>[-pca][-w]`, a fixed byte budget with bits allocated per
+  dimension by greedy reverse water-filling and a Gaussian-optimal quantizer step. `-pca`
+  rotates into principal components first; `-w` weights each component by query energy, which
+  targets score error rather than reconstruction error. On SciFact at 48 bytes with no
+  re-score: `binary` 0.611, `mixed48` 0.653, `mixed48-pca` 0.694, `mixed48-pca-w` 0.703, `pq48`
+  0.711.
+- [x] `Codec::prepare_query` / `score_prepared`, so transformed-space codecs rotate the query
+  once per search; used by the store's search, `bench recall` and `bench rescore`.
+- [x] Animated README figures for hybrid re-scoring, adaptive depth and adaptive bits
+  (`docs/assets/animations/`, regenerated by `make_animations.py`).
+- [x] **Hybrid × adaptive matrix** (`experiments/2026-11-hybrid-adaptive-matrix`): 11 scan codecs
+  × none/fixed/margin × int8/f32 over ten collections. `mixed48-pca-w` beats `pq48` at equal
+  bytes on average (0.768 vs 0.747 recall@10; the gain is all on MiniLM, a tie on bge-small), and
+  leads at 24 and 96 bytes too. Adaptive depth gains 0.3–0.6 points of recall at equal mean
+  depth, and up to 4 points for the worst decile. (The first version said 1.5 points; that came
+  from interpolating between sparse fixed depths, and is corrected in the write-up.)
+- [x] Per-query traces (`bench rescore --trace`) and the **routing experiment**
+  (`experiments/2026-11-routing-oracle`). A hindsight oracle has 5.5–7 points of headroom over
+  fixed depth. The band visible in the scan's own scores tracks the true band only at Spearman
+  0.52–0.70, and no router built on it captures more than 6%. Per-collection routing picks the
+  right codec size in 10/10 collections, but its predicted depth misses a 95% target by up to
+  9 points.
+- [x] **Zero-copy** (`experiments/2026-11-zero-copy-scan`). Borrowed codes in the `Codec` trait,
+  `CodeBuffer` (a contiguous, cached per-collection buffer scanned in place, optionally over
+  threads), in-place f32 and single-pass cosine kernels, and `binary-hamming` (popcount). The
+  cached buffer is 5.1× faster than the original row scan. Every asymmetric codec costs about
+  what f32 costs per vector. `binary-hamming` is 20× cheaper, and with an f32 re-score it reaches
+  95% recall@10 on FiQA about 19× faster than exact f32: the first speed claim hvec makes.
+- [ ] Choose the codec automatically at ingest from noise ÷ band, then measure the re-score
+  depth for a target recall on a sample of self-queries. The routing experiment showed that the
+  curve chooses the codec well but predicts the depth badly.
+- [ ] Layered (residual) codes: one stored code read to different depths per candidate, so
+  hybrid retrieval needs no second full-precision copy on disk.
+- [ ] A faster `mixed<B>` scan (byte-aligned groups, or a lookup table per group as PQ uses):
+  today it is the best 48-byte code for recall and the slowest to scan, 2.2× f32 per vector.
+- [ ] Time the hybrid end to end, with the f32 candidates fetched from disk, not memory.
+- [ ] SIMD kernels for f32 and int8, so the asymmetric codecs are compared at their best.
+- [ ] Decide whether to allow one reviewed `unsafe` call for memory-mapped code files
+  (`unsafe_code` is forbidden workspace-wide today). A flat file already loads 110 MB in 9 ms, so
+  the case is memory footprint and start-up on much larger collections.
+- [ ] Per-vector adaptive bits, the analogue of MoE PTQ's outlier scorer: more bits for vectors
+  in crowded neighbourhoods. Needs a variable-size encoding or two tiers.
+- [ ] Per-cluster PCA (Sub-MoE's grouped subspaces), if `mixed<B>-pca` gets within reach of
+  `pq<m>` at equal bytes.
+- [ ] Spacing follow-ups the write-up flagged: a deduplicated code corpus (CoSQA is 70%
+  duplicates), FiQA subsampled to 8,000 chunks to separate corpus size from spacing, a third
+  embedding model, and a tail statistic for the aggressive codecs the curve misses.
+
 ## Ideas from MoE compression
 
 Weight compression for Mixture-of-Experts models is working on a problem shaped like this one:
@@ -160,13 +234,11 @@ Three lines of work, and what each would mean as an hvec codec or experiment:
 
 - Run Experiments 1 and 3 and the judge re-scoring as soon as two chat models are reachable;
   the scripts need only `ANTHROPIC_API_KEY` (or an Ollama server) and a built `hvec`.
-- Codec follow-ups the write-ups flagged: a per-vector scale for `binary-centred`, and a
-  rescoring stage (fetch 5-10× candidates by codec, re-rank with int8 or f32).
-- A second corpus with graded, denser judgments than SciFact's.
-- Done 2026-11-04: neighbour spacing across five corpora (`experiments/2026-11-neighbour-spacing`).
-  Codec noise ÷ neighbour band predicts recall@10 loss (ρ = 0.98). Follow-ups: rescoring depth
-  as a function of that ratio, a deduplicated code corpus, and FiQA subsampled to separate
-  corpus size from spacing.
+- Phase 8 follow-ups, starting with choosing the codec at ingest and layered codes.
+- Codec follow-ups the write-ups flagged: a per-vector scale for `binary-centred`. The
+  re-scoring stage is done (phase 8).
+- Graded relevance: NFCorpus ships graded judgments, but hvec's nDCG treats relevance as
+  binary. Use the grades, then re-score the five-corpus results.
 
 ## Always