You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(asap-tools): benchmark ASAPQuery against ClickHouse precompute strategies (sketches, materialized views, and both) #491
Today the netflow benchmark compares ASAPQuery (sketchdb) against a single ClickHouse arm: exact aggregation over the raw MergeTree table (baseline). This is a useful correctness reference, but it is not a fair performance comparison — ClickHouse is doing full-scan exact work while ASAP serves results from pre-computed sketches.
To make a credible apples-to-apples claim, we need to compare ASAPQuery against ClickHouse configured to also precompute, using the mechanisms ClickHouse actually offers. There are three distinct ClickHouse strategies worth measuring, and we currently support none of them.
Comparisons to support
Comparison
ClickHouse strategy
Question it answers
A. ASAP vs ClickHouse with sketches
Approximate sketch functions at query time over raw data
Does ASAP's ingest-time precompute beat ClickHouse's query-time sketches?
B. ASAP vs ClickHouse with materialized views
Pre-aggregated (exact) state maintained at ingest
Does ASAP beat ClickHouse's classic incremental pre-aggregation?
C. ASAP vs ClickHouse with materialized views + sketches
Pre-aggregated sketch state maintained at ingest
The fairest match: both sides precompute approximate state — who wins on latency/resources at equal accuracy?
The exact baseline arm remains as the shared ground truth for accuracy across all three.
Goals / what this should achieve
Run all comparisons over the same netflow workload and queries
Produce, for each comparison, the metrics we already track: query latency, speedup vs ASAP, and CPU/memory of the monitored backend process.
Make the comparisons reproducible from a single experiment run where practical (one data load, multiple backend strategies), so results are directly comparable and not skewed by separate loads.
Slot cleanly into the existing post-experiment analysis and plotting so the three ClickHouse strategies appear as additional series alongside ASAP.
Acceptance criteria
Each of the three comparisons (A, B, C) can be run against the netflow workload.
Latency, speedup, and resource results are generated for each, comparable to current baseline/sketchdb outputs.
Per-query accuracy vs exact baseline is reported for every approximate arm.
Results are visualized alongside the existing ASAP-vs-baseline plots.
Known per-query semantic caveats are documented with the results.
Activity