Neighbour spacing, hybrid and adaptive quantization, routing, and zero-copy scans by SwineCoder101 · Pull Request #2 · SwineCoder101/hvec

SwineCoder101 · 2026-11-04T22:31:27Z

Five experiments and the tooling behind them. Every result has a write-up under experiments/, and all but the first (published earlier) get a site page here.

Experiments

Neighbour spacing (experiments/2026-11-neighbour-spacing): five BEIR-format corpora (SciFact, NFCorpus, ArguAna, FiQA, CoSQA code) × bge-small and MiniLM × nine codecs. Codec noise ÷ neighbour band ranks recall@10 loss with Spearman 0.98 over 90 points (noise alone: 0.64). Content type matters only through spacing.
Two-stage re-scoring (2026-11-two-stage-rescoring): the same ratio predicts the re-score depth (ρ ≈ 0.93). At 48 bytes, 50 f32 candidates recover 0.95–0.99 of the top 10 and nearly all of the nDCG loss.
Hybrid × adaptive matrix (2026-11-hybrid-adaptive-matrix): mixed48-pca-w (PCA + bits allocated by variance × query energy) beats pq48 at equal bytes on average (0.768 vs 0.747), all of it on MiniLM. Adaptive re-score depth gains 0.3–0.6 points at equal work, up to 4 for the worst decile. Includes a benefits-and-drawbacks table for each combination.
Routing (2026-11-routing-oracle): a hindsight oracle has 5.5–7 points of headroom over fixed depth, but the scan's own scores reveal almost none of it. Per collection, the noise-to-spacing curve picks the right codec size in 10/10 collections, but its predicted depth misses a 95% target by up to 9 points.
Zero-copy scan (2026-11-zero-copy-scan): the cached in-place buffer searches 5.1× faster than the original row scan. Every asymmetric codec costs about what f32 costs per vector; binary-hamming plus an f32 re-score reaches 95% recall@10 on FiQA about 19× faster than exact f32 on one thread.

Code

Codec: prepare_query / score_prepared; decode and score borrow &[u8] (Encoded derefs to [u8], so existing callers are unchanged).
New codecs: mixed<B>[-pca][-w] (adaptive bits) and binary-hamming (symmetric popcount).
In-place f32 scoring and a single-pass cosine, both bit-identical to before (tested).
hvec-store: CodeBuffer, a contiguous per-collection buffer scanned in place, optionally over threads. Store::search uses a cached buffer and reads text for the hits only. The cache is invalidated on writes and on other connections' commits.
bench recall reports neighbour_band, boundary_gap and rank_noise.
New commands: bench rescore (fixed and margin policies, worst-decile recall, depth for target recall, --trace per-query output) and bench scan (storage paths, codecs, threads, bandwidth ceiling).

Docs

README: a hybrid and adaptive section with three animated SVGs (CSS keyframes, light/dark, reduced-motion safe), and the "Faster search" row now makes its first claim.
Roadmap phase 8 and a section mapping MoE compression work (QuantMoE-Bench, MoE-SVD, Sub-MoE) onto hvec codecs.
Site: four experiment pages and a post, live once this merges to main.

Corrections made along the way

The matrix write-up first reported +1.5 points for adaptive depth. Interpolating between too few fixed depths inflated that. Measured exactly it is 0.3–0.6, and the write-up says so.

For review

Memory-mapped code files are not included, because unsafe_code is forbidden workspace-wide. Reading a flat file is already fast (110 MB in 9 ms). It's listed on the roadmap as a decision.
Five references were written from memory: the dataset papers, Max (1960) and Huang–Schultheiss (1963). QuantMoE-Bench and Sub-MoE were checked against arXiv. MoE-SVD is mentioned in the roadmap but not cited, because no source could be confirmed.
The site wasn't built locally (no Jekyll installed). The pages follow the existing pages' front matter and Liquid tags.

Testing

cargo test --workspace passes, and cargo clippy --all-targets and cargo fmt --check are clean.
Each intermediate commit was built and tested on its own.
New tests cover traces replaying the computed cells, buffered search equalling the row scan, cache invalidation across connections, the popcount distance, and the bench scan end to end.

Three CSS-keyframe SVGs that GitHub plays inside <img>: two-stage re-scoring (schematic), adaptive re-scoring depth by a per-query margin (schematic), and adaptive bit allocation at 48 bytes on SciFact's real PCA spectrum (uniform binary 0.611, mixed48-pca 0.694, mixed48-pca-w 0.703 recall@10). Light and dark steps follow prefers-color-scheme; under prefers-reduced-motion they rest on the final labelled frame. make_animations.py regenerates all three from the stored allocation.

Records the neighbour-spacing and two-stage re-scoring results, the margin policy, the mixed<B> codec, the query-preparation hook and the README animations as done; the hybrid x adaptive matrix as running; and the follow-ups: automatic re-score depth at ingest, per-vector adaptive bits, per-cluster PCA, and the spacing caveats. Phase 7 notes that its query hook now exists. Next drops the finished re-scoring item and adds graded relevance for NFCorpus.

bench rescore --trace <file> writes one JSON line per scan codec with, for every query, the scan's rank of each exact top-k neighbour, the exact band and the band read from the scan's own scores, and the margin policy's depth on a 0.25-sigma grid. With an exact second stage that is enough to replay any depth policy offline; a test checks the replay against the computed fixed and margin cells. evaluate now returns an Evaluation struct carrying the traces.

Replays fixed depth, the margin policy, a ratio router (depth from noise over the scan's own band) and a hindsight oracle from per-query traces, compared at equal mean depth: headroom, share captured, worst-decile recall. Also tests per-collection routing, predicting each collection's cheapest scan codec and depth for a recall target with the curve fitted on the other corpora. Results follow when the run completes.

Codec::decode, score and score_prepared take &[u8] instead of &Encoded, so a codec can score codes in place in a contiguous buffer or a database page. Encoded stays as encode's owned output and derefs to [u8], so existing callers compile unchanged. F32Codec::score reads the stored floats in place in one fused pass, with no Vec<f32> per vector; distance::cosine is one pass instead of three. Both are bit-identical to the previous results (tested). binary-hamming stores the same sign bits as binary but binarises the query once in prepare_query and scores by XOR and popcount over 64-bit words, estimating cosine as cos(pi * hamming / d). It is the symmetric form a scan-speed claim can be made for; its error model is documented as the one codec whose score is not the metric on its decoded vector.

CodeBuffer holds a collection's codes back to back in one allocation, with ids and sources as side arrays, and scans them in place: the query is prepared once, each code is a borrowed slice, a k-sized heap keeps the best hits, and the scan can split over scoped threads with identical results. It can be written to a flat code file and read back with one read. A memory-mapped file is left out: it needs unsafe, which the workspace forbids. Store::search now scans a cached buffer per collection and reads source, ordinal and text for the top k only. The cache is dropped on this handle's writes and when another connection commits (PRAGMA data_version). load_codes builds a buffer from borrowed blobs (one copy in all); search_borrowed scans SQLite's rows in place; set_mmap_size turns on SQLite's own memory map; search_by_rows keeps the original path for measurement. Tests: buffered search equals the row scan, writes and other connections' commits are seen.

hvec bench scan times one f32 collection three ways: the storage paths (SQLite rows copied, SQLite rows borrowed with and without SQLite's memory map, the cached buffer) and the cost of building the buffer from SQLite or a flat file; every codec scanned in place at each thread count (ms/query, GB/s, recall@k against exact f32, self excluded); and a sequential-read memory-bandwidth ceiling at the same thread counts. experiments/2026-11-zero-copy-scan has run.sh and summarise.py; results follow from a quiet machine.

Ten collections, eleven scan codecs, none/fixed/margin policies, int8 and f32 second stages. Adaptive bits: mixed48-pca-w keeps 0.768 of the exact top 10 with no re-scoring against 0.747 for pq48 and 0.669 for binary, and leads at 24 and 96 bytes too; the gain is all on MiniLM, a tie on bge-small; without the PCA rotation it is no better than binary-centred. Adaptive depth beats fixed depth at equal mean depth in 522 of 526 cells, +1.5 points of recall and +3.7 of worst-decile recall, at a p90 depth 1.3-2.1x the mean. Includes a benefits-and-drawbacks table for each combination.

38,124 query traces over six scan codecs and ten collections. A hindsight oracle beats fixed re-score depth by 5.5-7 points of recall@10 at equal mean depth, but the band visible in the scan's own scores tracks the true band only at Spearman 0.52-0.70: the margin policy captures 0-6% of the headroom and a ratio router does worse than fixed depth up to 50 candidates. Per-collection routing picks the right scan size in 10/10 collections, but its predicted depth misses a 95% target by up to 9 points, so depth should be measured, not predicted. Traces are gzipped; analyse.py reads either form.

On an otherwise idle M4 Pro: the cached contiguous buffer searches 5.1x faster than the original row scan on FiQA (14.7 vs 74.7 ms) and SQLite's memory map alone gives 2.7x. Every asymmetric codec costs 200-490 ns per vector against 207 for f32, under 2% of memory bandwidth, so fewer bytes do not mean a faster scan. binary-hamming costs 10 ns per vector; with an f32 re-score of 369 candidates it reaches 95% recall@10 on FiQA about 19x faster than exact f32 on one thread. mixed48-pca-w is the slowest scan, 2.2x f32 per vector.

The matrix write-up reported +1.5 points of recall for adaptive depth at equal mean depth; that came from interpolating between fixed depths of only 20, 50 and 100, which runs below the true curve. Scored exactly from the routing traces the gain is 0.3-0.6 points (ahead in 156 of 180), with up to 4 points for the worst decile. The write-up now says so and why. README: the Faster search row makes its first claim (binary-hamming plus an f32 re-score), binary-hamming is listed, the stale note about f32 decoding is replaced, and the adaptive-depth section states the measured effect. Roadmap: phase 8 items done, and new follow-ups (codec choice at ingest, layered codes, a faster mixed scan, end-to-end hybrid timing from disk, SIMD kernels, the unsafe decision for memory-mapped files).

…riments Four experiment pages, each with its hypothesis, setup, result table and caveats, linking to the full write-up, and a post that ties them together: hybrid retrieval gets the neighbours back, adaptive bits beat PQ at equal bytes on average, adaptive depth helps only a little (with the 1.5-point figure corrected), and only symmetric popcount scoring is faster. The README animations are embedded from docs/assets.

SwineCoder101 added 12 commits November 4, 2026 21:46

SwineCoder101 merged commit 7667e9f into main Nov 4, 2026
2 checks passed

Navigation Menu

Repository navigation

Neighbour spacing, hybrid and adaptive quantization, routing, and zero-copy scans - #2

Neighbour spacing, hybrid and adaptive quantization, routing, and zero-copy scans#2
SwineCoder101 merged 12 commits into
mainfrom
experiment/spacing-rescoring-adaptive

SwineCoder101 commented Nov 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

SwineCoder101 commented Nov 4, 2026

Experiments

Code

Docs

Corrections made along the way

For review

Testing

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant