Experiment/spacing rescoring adaptive

ROADMAP.md

@@ -127,13 +127,46 @@ rotation is a `d × d` matrix fitted on the corpus.
 Exit: Experiment 4 published, and a stated answer to whether rotation makes binary usable on
 this embedding model.
 
+## Ideas from MoE compression
+
+Weight compression for Mixture-of-Experts models is working on a problem shaped like this one:
+many similar objects (experts, vectors) sharing structure, where some matter more than others.
+Three lines of work, and what each would mean as an hvec codec or experiment:
+
+- **MoE post-training quantization** (QuantMoE-Bench, UNITES-Lab; arXiv:2406.08155,
+  [code](https://github.com/UNITES-Lab/moe-quantization)). Fine-grained, structure-aware mixed
+  precision: blocks, experts and linear layers get different bit widths, chosen by importance
+  scorers (expert usage frequency, a linear-weight outlier score) tuned to MoE's sparse
+  activation. *In hvec:* `mixed<B>` allocates a byte budget's bits per dimension by variance,
+  and `-w` weights that variance by how much energy queries put into the dimension, the analogue
+  of weighting an expert by how often it is routed to. The per-vector analogue of an outlier
+  scorer, giving more bits to vectors in crowded neighbourhoods, is not built yet.
+- **MoE-SVD** (citation not yet verified; we have only the description so far). Training-free:
+  SVD decomposes each expert's weights into low-rank factors, the V matrices are shared across
+  experts, and the U matrices are trimmed to their top-k columns. *In hvec:* PCA is the SVD of
+  the centred corpus. `mixed<B>-pca` with zero bits on the trailing components is a low-rank
+  codec, and its single rotation per collection is the shared V. Top-k U trimming corresponds to
+  truncating components, which is `mrl<k>` applied after the rotation instead of on raw
+  dimensions.
+- **Sub-MoE, subspace expert merging** (arXiv:2506.23266,
+  [code](https://github.com/siruihan2024/Sub-MoE)). Experts are clustered by cosine similarity,
+  each group gets a joint SVD, and the V matrices are merged by expert frequency to resolve
+  parameter conflicts. *In hvec:* one PCA per k-means cluster instead of one per collection
+  (closest to locally optimised PQ), with each cluster's bit allocation weighted by how often
+  queries land in it. This is the natural next codec if `mixed<B>-pca` beats `pq<m>` at equal
+  bytes.
+
 ## Next
 
 - Run Experiments 1 and 3 and the judge re-scoring as soon as two chat models are reachable;
   the scripts need only `ANTHROPIC_API_KEY` (or an Ollama server) and a built `hvec`.
 - Codec follow-ups the write-ups flagged: a per-vector scale for `binary-centred`, and a
   rescoring stage (fetch 5-10× candidates by codec, re-rank with int8 or f32).
 - A second corpus with graded, denser judgments than SciFact's.
+- Done 2026-11-04: neighbour spacing across five corpora (`experiments/2026-11-neighbour-spacing`).
+  Codec noise ÷ neighbour band predicts recall@10 loss (ρ = 0.98). Follow-ups: rescoring depth
+  as a function of that ratio, a deduplicated code corpus, and FiQA subsampled to separate
+  corpus size from spacing.
 
 ## Always
 

crates/hvec-bench/src/lib.rs

@@ -12,6 +12,7 @@ pub mod questions;
 pub mod recall;
 pub mod record;
 pub mod report;
+pub mod rescore;
 
 pub use chunk::{ChunkOptions, chunk_text};
 pub use judge::{JUDGE_PROMPT_VERSION, JUDGE_SYSTEM_PROMPT, Verdict, build_judge_prompt, parse_verdict};

crates/hvec-bench/src/recall.rs

@@ -53,16 +53,64 @@ pub struct RecallReport {
     pub ndcg_at_k: Option<f64>,
     /// Mean fraction of relevant documents among the codec's top-k.
     pub doc_recall_at_k: Option<f64>,
+    /// Median over queries of the exact score gap between rank k and rank
+    /// k + 1: how far the last true neighbour sits above the first outsider.
+    /// A property of the corpus and embedding model; the same for every codec.
+    #[serde(default)]
+    pub boundary_gap: f64,
+    /// Median over queries of the exact score gap between rank k and rank
+    /// 2k: the width of the score window holding the next k competitors.
+    /// Codec-independent, like `boundary_gap`.
+    #[serde(default)]
+    pub neighbour_band: f64,
+    /// Mean over queries of the standard deviation of the codec's score error
+    /// over the exact top-2k. Per-query bias is removed, since a constant
+    /// shift does not change a ranking: this is the noise that reorders.
+    #[serde(default)]
+    pub rank_noise: f64,
 }
 
-fn top_k(scores: &[(i64, f32)], k: usize, metric: Metric) -> Vec<i64> {
-    let mut v: Vec<(i64, f32)> = scores.to_vec();
+/// Indices into `scores`, best first, ties broken by id.
+pub(crate) fn ranked(scores: &[(i64, f32)], metric: Metric) -> Vec<usize> {
+    let mut order: Vec<usize> = (0..scores.len()).collect();
+    let by_id = |a: usize, b: usize| scores[a].0.cmp(&scores[b].0);
     if metric.higher_is_better() {
-        v.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
+        order.sort_by(|&a, &b| scores[b].1.total_cmp(&scores[a].1).then(by_id(a, b)));
     } else {
-        v.sort_by(|a, b| a.1.total_cmp(&b.1).then(a.0.cmp(&b.0)));
+        order.sort_by(|&a, &b| scores[a].1.total_cmp(&scores[b].1).then(by_id(a, b)));
+    }
+    order
+}
+
+fn top_k(scores: &[(i64, f32)], k: usize, metric: Metric) -> Vec<i64> {
+    ranked(scores, metric)
+        .into_iter()
+        .take(k)
+        .map(|i| scores[i].0)
+        .collect()
+}
+
+pub(crate) fn median(mut v: Vec<f64>) -> f64 {
+    if v.is_empty() {
+        return 0.0;
+    }
+    v.sort_by(f64::total_cmp);
+    let m = v.len() / 2;
+    if v.len() % 2 == 1 {
+        v[m]
+    } else {
+        (v[m - 1] + v[m]) / 2.0
+    }
+}
+
+/// Standard deviation of `approx − exact` over the given indices.
+pub(crate) fn centred_noise(exact: &[(i64, f32)], approx: &[(i64, f32)], idx: &[usize]) -> f64 {
+    if idx.len() < 2 {
+        return 0.0;
     }
-    v.into_iter().take(k).map(|(id, _)| id).collect()
+    let errs: Vec<f64> = idx.iter().map(|&i| f64::from(approx[i].1 - exact[i].1)).collect();
+    let mean = errs.iter().sum::<f64>() / errs.len() as f64;
+    (errs.iter().map(|e| (e - mean).powi(2)).sum::<f64>() / errs.len() as f64).sqrt()
 }
 
 /// Evaluate `codec` on `(id, f32 vector)` pairs with the given queries.
@@ -101,6 +149,9 @@ pub fn evaluate(
     let mut labelled = 0usize;
     let mut ndcg_sum = 0f64;
     let mut doc_recall_sum = 0f64;
+    let mut boundary_gaps: Vec<f64> = Vec::with_capacity(queries.len());
+    let mut bands: Vec<f64> = Vec::with_capacity(queries.len());
+    let mut noise_sum = 0f64;
 
     for q in queries {
         let t = Instant::now();
@@ -113,11 +164,12 @@ pub fn evaluate(
         baseline_scan_ms += t.elapsed().as_millis() as u64;
 
         let t = Instant::now();
+        let prepared = codec.prepare_query(&q.vector)?;
         let approx: Vec<(i64, f32)> = ids
             .iter()
             .zip(&encoded)
             .filter(|(id, _)| Some(**id) != q.exclude)
-            .map(|(id, e)| codec.score(&q.vector, e, metric).map(|s| (*id, s)))
+            .map(|(id, e)| codec.score_prepared(&prepared, e, metric).map(|s| (*id, s)))
             .collect::<Result<_, _>>()?;
         scan_ms += t.elapsed().as_millis() as u64;
 
@@ -127,7 +179,17 @@ pub fn evaluate(
             err_max = err_max.max(e);
             pairs += 1;
         }
-        let truth = top_k(&exact, k, metric);
+        // `exact` and `approx` are index-aligned: both walk `ids` in order.
+        let order = ranked(&exact, metric);
+        let at = |rank: usize| order.get(rank - 1).map(|&i| f64::from(exact[i].1));
+        if let (Some(sk), Some(sk1)) = (at(k), at(k + 1)) {
+            boundary_gaps.push((sk - sk1).abs());
+        }
+        if let (Some(sk), Some(s2k)) = (at(k), at(2 * k)) {
+            bands.push((sk - s2k).abs());
+        }
+        noise_sum += centred_noise(&exact, &approx, &order[..order.len().min(2 * k)]);
+        let truth: Vec<i64> = order.iter().take(k).map(|&i| exact[i].0).collect();
         let got = top_k(&approx, k, metric);
         recall_sum += recall_at_k(&truth, &got, k);
         if !truth.is_empty() && truth.first() == got.first() {
@@ -169,6 +231,9 @@ pub fn evaluate(
         labelled_queries: labelled,
         ndcg_at_k: labelled_mean(ndcg_sum),
         doc_recall_at_k: labelled_mean(doc_recall_sum),
+        boundary_gap: median(boundary_gaps),
+        neighbour_band: median(bands),
+        rank_noise: noise_sum / nq,
     })
 }
 
@@ -238,6 +303,35 @@ mod tests {
         assert_eq!(r.vectors, 200);
         assert_eq!(r.labelled_queries, 0);
         assert_eq!(r.ndcg_at_k, None);
+        assert_eq!(r.rank_noise, 0.0);
+        assert!(r.neighbour_band >= r.boundary_gap && r.boundary_gap > 0.0);
+    }
+
+    #[test]
+    fn spacing_is_codec_independent_and_noise_is_not() {
+        let (ids, vecs) = clustered();
+        let queries = self_queries(&ids, &vecs, 20);
+        let i8 = evaluate(&ids, &vecs, &queries, &Int8Codec::new(64), Metric::Cosine, 10, None).unwrap();
+        let bin = evaluate(&ids, &vecs, &queries, &BinaryCodec::new(64), Metric::Cosine, 10, None).unwrap();
+        assert_eq!(i8.boundary_gap, bin.boundary_gap);
+        assert_eq!(i8.neighbour_band, bin.neighbour_band);
+        assert!(
+            bin.rank_noise > 10.0 * i8.rank_noise,
+            "{} vs {}",
+            bin.rank_noise,
+            i8.rank_noise
+        );
+    }
+
+    #[test]
+    fn centred_noise_ignores_a_constant_shift() {
+        let exact = vec![(1, 0.9), (2, 0.8), (3, 0.7)];
+        let shifted: Vec<(i64, f32)> = exact.iter().map(|(i, s)| (*i, s - 0.25)).collect();
+        assert!(centred_noise(&exact, &shifted, &[0, 1, 2]) < 1e-6);
+        let noisy = vec![(1, 0.8), (2, 0.9), (3, 0.7)];
+        assert!(centred_noise(&exact, &noisy, &[0, 1, 2]) > 0.05);
+        assert_eq!(median(vec![3.0, 1.0, 2.0]), 2.0);
+        assert_eq!(median(vec![4.0, 1.0, 2.0, 3.0]), 2.5);
     }
 
     #[test]