Measure accuracy after merging m shards by zzylol · Pull Request #131 · ProjectASAP/sketch-bench

zzylol · 2026-10-01T13:10:10Z

Summary

Adds accuracy after merging to sketch-bench. Before this PR, --operations merge --metrics accuracy failed with "nothing measures the accuracy of merge, for any sketch".

--operations merge --metrics accuracy --merge-shards m: splits the N-item stream into m contiguous shards, builds one sketch per shard, and merges them into one. The merged sketch is scored with the same probes, comparator and full-stream ground truth as the single-sketch query, and reported as merge_accuracy. Sketches without merge fail with the registry's existing message.
scripts/study_saturation.py --merge-shards-list 1,4,16,64: writes saturation_merge_curve.csv, giving error vs N per shard count.
Tests: CMS merged accuracy equals single-sketch accuracy exactly; KLL merged accuracy is finite.

Findings

Bounded run: first config per sketch, N ≤ 1e7, 3 seeds, θ ∈ {0.5, 1.0, 1.5}, K ∈ {1e3, 1e5, 1e7}, α ∈ {1.1, 2}.

CMS, CountSketch, HLL, DDSketch: merged error is identical to the single sketch in every cell (0 of 1479 differ). Their saturation curves apply directly to merged windows.
KLL (k = 50): within seed noise up to 16 shards; 5–16% higher at 64 shards.
CMS-heap top-k (3×256): unchanged at K = 1e3. At K ≥ 1e5 and θ ≥ 1, precision@k drops by 0.03–0.09 (8–25%). Single-sketch curves overstate merged top-k quality.

Follow-up

The run covers one config per sketch. scripts/recommend_config.py (in #130) therefore still flags top-k/KLL recommendations without a matching merge curve as possibly optimistic.

Validation

Before the rebase: cargo fmt --check, cargo test for aqpbm-core/sketch-bench/aqpbm-cli, and cargo clippy --all-targets -D warnings all passed.
After the rebase: Python unittests pass, and the merge_accuracy integration test passes.

🤖 Generated with Claude Code

Update: full grids

Merged accuracy now covers:

all 8 CMS-heap top-k configs (3 seeds);
KLL k 50/200/800 and DDSketch α_dd 0.005–0.05 (10 seeds).

Results are in docs/figures/saturation/merge_grid/:

Sketch	Merged vs single sketch
DDSketch	exact everywhere
Top-k	median precision unchanged; drops of up to 40% at large K, for every width
KLL	merged error grows with k: ≈ 1.0–1.1× at k = 50/200, 1.12–1.32× at k = 800 at N = 1e7, and up to 3–4× at N = 1e4–1e5 (k = 800, m = 4)

recommend_config.py --merge-curves now has a merged curve for every top-k and KLL config:

No recommendation is flagged "single-sketch curve optimistic" any more; before this there were 43.
6 recommendations changed. For example, p99 latency over 5m windows now needs KLL k = 200, not k = 50.

The math behind the KLL result is in #130's docs/saturation_conclusions.md.

milindsrivastava1997 · 2026-10-04T20:25:43Z

                    *merge_supported,
                    "merge_supported"
                );
+                keep_owned!(out.merge.accuracy, accuracy.clone(), "accuracy");


Merge accuracy can't be flattened together with any other merge metric. Every merge record stamps merge_shards/merge_supported (main.rs:440, bench_report.rs:52), and keep_owned! on merge_shards (line 169) rejects the second value.

Repro on this branch:

--operations merge --metrics throughput,accuracy --flat Error: flatten_record: cms-fastpath-vector2d/lib merge has two values for merge_shards

Same failure from approxbench flatten joining a cost pass (merge throughput) with an accuracy pass that now includes merge accuracy, which is the join flatten_cmd.rs exists for. Throughput+latency on merge already hit this before, but this PR makes it the common case. These two fields probably want keep_shared! (equal values allowed) rather than keep_owned!.

milindsrivastava1997 · 2026-10-04T20:25:43Z

        curve_file.flush()
+        if shard_list:
+            merge_path = os.path.join(args.out, "saturation_merge_curve.csv")
+            append = args.resume and os.path.exists(merge_path)


With --resume, saturation_merge_curve.csv is only appended to, while saturation_curve.csv is rewritten with complete points only.

Duplicates: if a run is killed mid-point after the 8 KB buffer flushed some of that point's merge rows, those rows stay; the resumed run re-runs the point and appends them again. plot_saturation_merge.py then plots repeated (n, error) per m, garbling lines/bands and the ratio at max N.

Gaps: a point whose curve is complete but whose merge rows were never written (kill between curve_file.flush() and merge_file.flush(), or an earlier run without --merge-shards-list) is skipped and never gets merge rows.

Suggest rewriting the merge CSV on resume the same way as the curve CSV: keep only rows for points that are complete in both, and re-run points missing merge rows.

this makse sense

milindsrivastava1997 · 2026-10-04T20:25:43Z

+            ys = [p[1] for p in pts]
+            ses = [p[2] for p in pts]
+            ax.fill_between(ns, [y - 2 * s for y, s in zip(ys, ses)], [y + 2 * s for y, s in zip(ys, ses)],
+                            color=SHARD_COLORS[m], alpha=0.12, lw=0)


SHARD_COLORS only has keys 1/4/16/64, so any other --merge-shards-list (e.g. 1,2,8) dies with KeyError here. The summary plot also hardcodes (4, 16, 64), so other shard counts would silently be dropped there even with a color. Suggest deriving the shard counts from the data and falling back to a palette cycle for unknown m.

this makes sense. you can extend.

With --merge-shards m, the stream is split into m contiguous shards, one sketch is built per shard, the sketches are folded into one, and that sketch is asked the query's probes and scored with the same comparator against the whole-stream truth, as merge_accuracy. Every mergeable row supplies the asked-after-fold body next to its merge and merge-step bodies; the plain query is the one-shard case. Rows without merge keep the registry error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

For each point and N, also score the sketch merged from m contiguous shards for every m>1 in the list, into saturation_merge_curve.csv (m=1 repeats the single-sketch curve). Without the flag nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Replace the "Accuracy after merging" gap with results from a bounded --merge-shards-list 1,4,16,64 run: linear sketches, HLL and DDSketch merge exactly; KLL drifts within seed noise up to 64 shards; the top-k heap loses precision once there are more keys than its heap holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Error vs N per shard count for every sketch and distribution, and a merged/single ratio summary at the largest N, with a reading guide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

All top-k configs and KLL k 50/200/800 and DDSketch with 10 seeds, merged at 4/16/64 shards; plots per config, a merged/single summary, and the recommendations rerun with these merge curves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- flatten_record: merge_shards/merge_supported may repeat across merge squares when equal, so throughput+accuracy on merge flattens. - study_saturation --resume rewrites saturation_merge_curve.csv with only complete points and reruns points whose merge curve is missing. - plot_saturation_merge: colours and summary follow the shard counts in the data; exits when the m=1 baseline is missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

GordonYuanyc

Wow! LGTM! I'm convinced this is the right way to measure merge accuracy.

#131 merged into main as bd644fe with content identical to the merged-accuracy head the merge curves came from, so drop the not-merged caveat. The table and its recorded revision are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…#136) * Export the AutoSketch vs ASAP evaluation table from saturation curves Adds scripts/export_autosketch_eval_table.py and the generated rqe-optimizer/data/autosketch-eval/table.json (W2: alibaba_v2022, boom, google_2011). Every config lists AutoSketch's single-sketch error at max_N and ASAP's error after merging m = 1/4/16/64 shards, with the saturation flag ASAP requires. recommend_config.py's grid rounding moves into grid_point() so both scripts share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Note that sketch-bench #131 has merged #131 merged into main as bd644fe with content identical to the merged-accuracy head the merge curves came from, so drop the not-merged caveat. The table and its recorded revision are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

- rqe-optimizer/results/autosketch-vs-asap-synthetic: 66 runs of the ProjectASAP/ASAPQuery#777 workload grid (default point, every dimension alone, replicas x series, windows x card), model A and model B per family, every SLA. 0 sanity violations, the peak bound exact everywhere, no solve or job timed out. Raw results as sweep-json.tar.gz; sweep.csv, summary, figures and per-shard provenance (34 runs at ba52fd8 with verified identical tables, 32 at 0a45c48). - rqe-optimizer/data/profiling-time.json: ASAP's one-time profiling, 35.6 h of sketch-bench runs (#130, #131, #140), from record timestamps. - The synthetic summary and figures order each dimension's values by the grid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol mentioned this pull request Oct 1, 2026

Add saturation study: error vs N, N_saturation and Pareto value data #130

Merged

zzylol force-pushed the merged-accuracy branch 3 times, most recently from 18cada3 to 219c9c9 Compare October 2, 2026 03:11

milindsrivastava1997 reviewed Oct 4, 2026

View reviewed changes

milindsrivastava1997 marked this pull request as ready for review October 4, 2026 21:09

milindsrivastava1997 mentioned this pull request Oct 4, 2026

Add saturation study: error vs N, N_saturation and Pareto value data #133

Merged

milindsrivastava1997 force-pushed the merged-accuracy branch from 875600f to 05eec26 Compare October 4, 2026 21:18

milindsrivastava1997 changed the base branch from study-saturation to saturation-study October 4, 2026 21:18

milindsrivastava1997 force-pushed the merged-accuracy branch from 05eec26 to 792aaf8 Compare October 4, 2026 21:26

milindsrivastava1997 changed the base branch from saturation-study to main October 4, 2026 21:26

milindsrivastava1997 mentioned this pull request Oct 4, 2026

CI: build for generic x86-64, not the runner's CPU #134

Merged

zzylol and others added 6 commits October 4, 2026 17:49

Plot accuracy after merging

8f5db92

Error vs N per shard count for every sketch and distribution, and a merged/single ratio summary at the largest N, with a reading guide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

milindsrivastava1997 force-pushed the merged-accuracy branch from 792aaf8 to a0b8b42 Compare October 4, 2026 21:49

milindsrivastava1997 requested a review from GordonYuanyc October 4, 2026 22:06

This was referenced Oct 4, 2026

Export the AutoSketch vs ASAP evaluation table from saturation curves #136

Merged

eval: AutoSketch vs. ASAP on the synthetic mixed set (cost by use; frontier and SLA versions) #138

Merged

GordonYuanyc approved these changes Oct 5, 2026

View reviewed changes

milindsrivastava1997 merged commit bd644fe into main Oct 5, 2026
2 checks passed

zzylol added a commit that referenced this pull request Oct 5, 2026

Merge the #131-merged note from feat/autosketch-eval-table

2d6f580

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol added a commit that referenced this pull request Oct 5, 2026

Drop the #131-still-open caveat from the results README

1174edd

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol added a commit that referenced this pull request Oct 5, 2026

Merge the #131 notes from eval/autosketch-vs-asap

3ee220f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol mentioned this pull request Oct 5, 2026

Saturation curves at label cardinalities 1e1–1e6 for the synthetic planner workload #140

Closed

This was referenced Oct 6, 2026

Evaluate the synthetic PromQL workload of ASAPQuery#777 #139

Closed

rqe-optimizer: merged KLL and top-k answers read the merge curves (#158) #179

Merged

This was referenced Oct 9, 2026

rqe-optimizer: let mergeable finer-group deployments serve coarser RAQEs #187

Closed

rqe-optimizer: let mergeable finer-group deployments serve coarser RAQEs #190

Merged

Navigation Menu

Repository navigation

Measure accuracy after merging m shards - #131

Measure accuracy after merging m shards#131
milindsrivastava1997 merged 6 commits into
mainfrom
merged-accuracy

zzylol commented Oct 1, 2026 •

edited by milindsrivastava1997

Loading

Uh oh!

milindsrivastava1997 Oct 4, 2026

Uh oh!

zzylol Oct 4, 2026

Uh oh!

milindsrivastava1997 Oct 4, 2026

Uh oh!

zzylol Oct 4, 2026

Uh oh!

milindsrivastava1997 Oct 4, 2026

Uh oh!

zzylol Oct 4, 2026

Uh oh!

GordonYuanyc left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

zzylol commented Oct 1, 2026 • edited by milindsrivastava1997 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Findings

Follow-up

Validation

Update: full grids

Uh oh!

milindsrivastava1997 Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

zzylol Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

milindsrivastava1997 Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

zzylol Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

milindsrivastava1997 Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

zzylol Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

GordonYuanyc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zzylol commented Oct 1, 2026 •

edited by milindsrivastava1997

Loading