From e5f0eabd7b5c652976e02bfeed0c2cc45b555525 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 5 Oct 2026 17:37:27 +0000 Subject: [PATCH 1/6] docs(asap-tools): evaluation plan for data plane vs ClickHouse Plan for the paper's data-plane experiment: deploy the planner's plan set on the ASAP data plane and on ClickHouse (MVs with exact and sketch states), and compare cost, latency and accuracy over the synthetic Zipf/Pareto workload. Co-Authored-By: Claude Opus 5.5 --- .../DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md | 311 ++++++++++++++++++ 1 file changed, 311 insertions(+) create mode 100644 asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md diff --git a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md new file mode 100644 index 00000000..b3d1320c --- /dev/null +++ b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md @@ -0,0 +1,311 @@ +# Evaluation plan: ASAP data plane vs ClickHouse with MVs and summaries + +Status: **draft for discussion**, no results yet. Backs the paper subsection +"ASAP's data plane". Companion to the planner-level plan in +`docs/evaluation/autosketch-vs-planner.md` (branch +`docs/autosketch-vs-planner-eval`), whose synthetic workload this plan reuses. + +## 1. Question + +Fix a dataset and a query workload. Run the planner once and get its set of +summary-based plans. Deploy that **same** plan set two ways: + +1. **ASAP data plane**: the backend precompute runtime, SummaryStore and query + adapters serve the plan natively. +2. **ClickHouse**: the closest equivalent ClickHouse can express, which is + materialized views (MVs) over the raw table that store ClickHouse's built-in + aggregate states. + +How much cheaper and faster is (1) than (2), at what accuracy? + +The planner is held fixed, so this isolates the data plane. ClickHouse cannot +deploy MVs or summaries automatically, so the harness translates each plan by +hand. Where ClickHouse cannot express a planned summary, the translation records +the reason. That list becomes the paper's "ClickHouse may not support it" point: + +- (a) the sketch is not implemented: Count-Min, KLL, keyed KLL; +- (b) the sketch's parameters are not configurable: `quantilesState`'s + reservoir is fixed at 8192, `uniqHLL12` is fixed at 12 bits, and `topK`'s + load factor is fixed. + +## 2. What exists today + +### 2.1 ASAPQuery (this repo): `experiment_run_clickhouse.py` + +Checked on `main` @ `52821e0`. This is a single-node runner (`num_nodes=1`; +`node_offset` picks the node). + +1. rsync the JSONL dataset to the node and start ClickHouse in Docker. +2. **Load the data once**, under an ingest monitor that samples the + `clickhouse` process. With `dataset.init_sql_file` set, that SQL owns all + DDL, *including MVs*. It runs before the load, so the MVs fill on insert. +3. Loop over the modes in `experiment_params.experiment`: + - **`sketchdb`**: + - Run `asap-planner` in SQL mode, using the **hardcoded generator, not + the cost optimizer**. + - Start the legacy `query_engine_rust`. It ingests the same JSONL itself + and forwards unsupported queries to ClickHouse. + - Wait `steady_state_wait`, then query `:8088`. + - **Any other mode** (`baseline`, `baseline_mv`, `baseline_sketch` or + `baseline_mv_sketch`, defined in `constants.py`): query ClickHouse + `:8123` with that mode's SQL file. `query_groups[i].sql_file` is a dict + keyed by mode. + +The mode names, per-mode SQL and MV-capable init SQL already exist. Nothing +generates the MV DDL or the MV-targeted SQL. + +### 2.2 ASAPQuery-backend: the current data plane + +The backend has the precompute runtime, SummaryStore and PromQL, MetricsQL and +ClickHouse SQL adapters, each with exact fallback. Its families include KLL, +DDSketch, HLL, CountMin, CountMin with heap and TopK. + +The relevant prior ClickHouse work is in two places. + +**`tools/o11y-sql-main-eval/`** +- A **27-query o11y corpus** (`corpus.json`): each query appears as PromQL, + MetricsQL and a ClickHouse SQL translation over + `raw_samples(metric, labels Map, ts_ms, value)`. +- Matched-budget runners: 2 pinned cores, 4 GiB, a pinned ClickHouse image, + restart per trial, query cache off, `max_threads=2`. +- Reported results: + - Only q05, q06 and q23 run warm, all three on exact MinMax. + - 18 queries fall back to exact ClickHouse. + - 6 queries OOM on both sides. + - The SQL frontend rejects ClickHouse-specific syntax: lambdas, Map access, + `mapConcat`. + +**`tools/clickhouse-benefit/`** +- A single-query warm-path probe against ClickHouse. +- A matched-trial runner. + +**Limit that shapes this plan.** The backend's *automatic* ClickHouse-SQL +binder supports only bounded scalar reductions (sum, count, min, max) today. +Quantile and top-k plans are **not** warm through the SQL adapter. They are +through the PromQL adapter (`quantile_over_time`, `sum by`, `rate`). + +### 2.3 Planner + +The cost-optimal planner used in the paper is sketch-bench `rqe-optimizer`, a +MILP. It reads the measured `AtomicCostTable` and implements the Model A and +Model B cost models from the AutoSketch plan. `asap-planner-rs/src/optimizer` +is still offline and PromQL-only, and the SQL-mode `asap-planner` uses the +hardcoded generator. + +### 2.4 Gaps + +| Gap | Where | +|---|---| +| `config/experiment_type/clickhouse.yaml` is used in the runner docstring but was never committed | ASAPQuery | +| `generate_queries.py` (SQL workload generator) was deleted in #404 | ASAPQuery | +| The MV init SQL examples (`netflow_init.sql`, `quantile_demo/init.sql`) are referenced but absent | ASAPQuery | +| No Zipf/Pareto **JSONL** generator. The fake exporter's Zipf is Prometheus-only, with a fixed α=1.01 | ASAPQuery | +| Data loads once per run, so MV maintenance cost would be charged to every ClickHouse mode | ASAPQuery runner | +| `compare_costs.py` keys on the Prometheus process | ASAPQuery post-processing | +| No plan → ClickHouse MV translator | both | +| SQL adapter is not warm for quantile/top-k plans | backend | +| The setup scripts assume `/scratch`; these nodes use `/mydata` | ASAPQuery `cloudlab_setup` | + +## 3. Workload + +Reuse the **synthetic workload from `autosketch-vs-planner.md` §6** so both +evaluation sections share data, queries and plans. Add the real o11y corpus as +a secondary point. + +### 3.1 Synthetic (main figure) + +**Data** +- One metric, `data`, with labels `label_0` (`C` groups) and `instance` + (`s` series per group). +- Each series emits 100 samples/s. +- Frequency and top-k weights follow Zipf θ. Quantile values follow Pareto a. + +**Templates** +- The 10 PromQL templates from the AutoSketch plan: `sum by`, `topk by`, + `quantile by`, `sum_over_time`, `quantile_over_time`, `rate`, their spatial + aggregations, and the `quantile_over_time` ratio. +- Each one is paired here with a hand-checked **ClickHouse SQL** translation + over a flat table `data(ts_ms, label_0, instance, value)`. +- A flat table is used because Map columns are what break the backend SQL + frontend in the corpus results. + +**Grid** (one dimension at a time around the bold defaults, as in that plan) + +| Dimension | Values | +|---|---| +| Query mix | **all 10**, frequency only, quantile only, top-k only | +| `C` | 1e2, **1e3**, 1e4, 1e5 | +| `s` | 1, 10, **100** | +| θ / a | θ ∈ {0, 0.5, **1.0**, 1.5}; a ∈ {1.1, **2**, 3} | +| Window set `W` | {1m}, {1m, 1h}, **{1m, 10m, 1h}** | +| Accuracy target | 90%, **95%**, 99% | + +The defaults are capped at `W` ≤ 1h and `C·s` = 1e5 series × 100 samples/s +(1e7 samples/s). That is too much for one ClickHouse node at real time, so the +data plane runs on **replayed time**: + +- generate a fixed span (default: 2 h of event time); +- ingest it as fast as each system accepts; +- run each RQE at every `T` over that span. + +The rate is a knob, not a fixed point: 1e5 samples/s by default, with ingest +throughput reported separately. + +### 3.2 Real o11y corpus (secondary) + +The backend's 27-query corpus already has a correctness matrix. Use only the +queries whose plans actually install. Re-run the corpus audit first, since +today that is q05, q06 and q23. Report them as "real workload" bars. + +## 4. Arms + +All arms see the same rows and the same plan set. + +| Arm | What is deployed | Role | +|---|---|---| +| `ch_raw` | ClickHouse raw `MergeTree`, no MVs | Exact reference and accuracy ground truth | +| `ch_mv_exact` | ClickHouse with MVs per plan, exact states (`sumState`, `countState`, `min/maxState`, `quantilesExactState`, per-key `count` for top-k) | "MVs without summaries" | +| `ch_mv_sketch` | ClickHouse with MVs per plan, closest summary state (table below) | **Main ClickHouse point** | +| `asap` | Backend data plane with the same plan set | **Main ASAP point** | + +Each MV is keyed by `(toStartOfInterval(ts, x), group-by labels)`, where `x` +is the plan's window/pane. Queries merge `S / x` panes with the `-Merge` +combinators, mirroring the plan's `(x, y)` and merge pattern. + +Plan → ClickHouse mapping for `ch_mv_sketch`: + +| Planned summary | ClickHouse state | Gap reported | +|---|---|---| +| Sum / count | `sumState` / `countState` | none: an exact parity control | +| MinMax | `minState` / `maxState` | none: a control (matches corpus q05/q06/q23) | +| KLL (per group or per series) | `quantilesState(levels)(v)` (reservoir 8192) or `quantilesTDigestState` | No KLL. Size/accuracy cannot be set: a reservoir is either over-sized or misses the target | +| DDSketch | `quantilesDDState(relative_accuracy, …)` | Closest match. Parameterized, so expected near-parity. Report it honestly | +| CountMin (frequency) | none: exact `sumState` per key | Not implemented. Cost scales with key cardinality, so state is `C·s` rows, not `d×w` | +| CountMin with heap / TopK | `topKWeightedState(k)` (Filtered Space-Saving) | No CMS. The load factor cannot be set | +| HLL | `uniqHLL12State` / `uniqCombinedState(p)` | `uniqHLL12` is fixed at 12 bits; `uniqCombined` lets you set the precision | + +The translator emits every row of this table. A plan that falls into a "none" +row becomes an exact fallback and is counted. + +## 5. Running ASAP on the same plan + +1. Run `rqe-optimizer` on the workload, with the model A cost and the default + accuracy target. Its output is the plan set. +2. Translate it into a backend `PhysicalPlanInstallRequest` / `PlanEnvelope` + publication. The compiler path already exists for Planner DAGs. **Check + with the backend owners whether an `rqe-optimizer` plan can be published + directly, or only through ASAPPlanner.** +3. Ingest: + - **ASAP**: Prometheus Remote Write (or the backfill reader from the + ClickHouse raw table, `storage_engines/sketch_db/backfill/clickhouse_reader.rs`). + - **ClickHouse**: JSONL bulk insert. + + Report the ingest cost of each path separately. Do not charge ASAP for the + ClickHouse raw load unless fallback is needed. +4. Queries: + - **ASAP**: PromQL against the backend. Quantile and top-k are not warm + through the SQL adapter yet (§2.2). + - **ClickHouse**: the paired SQL. + + Correctness is checked per query, as the corpus runner does it: compare the + decoded results against `ch_raw` within the accuracy target. +5. Require **0 fallbacks** on the ASAP side for every RQE counted as served. + Count and report any fallback separately, as the corpus report does. + +If the team prefers the legacy ASAPQuery `query_engine_rust` path in +`experiment_run_clickhouse.py`, which is SQL end to end, steps 2 and 4 change. +That path cannot take an `rqe-optimizer` plan either (decision D1). + +## 6. Metrics + +| Metric | ClickHouse | ASAP | +|---|---|---| +| Ingest CPU (core-s), including MV/summary maintenance | `clickhouse` process during load (existing ingest monitor) | backend precompute threads | +| Retained state bytes | `system.parts.bytes_on_disk` + `primary_key_bytes_in_memory` for the MV target tables (raw table reported separately) | SummaryStore state directory + in-memory instances | +| RSS (steady and peak) | process | process | +| Query latency p50/p95/p99 | client-side per request, warm (10 warm-ups discarded) | same | +| Query CPU | process CPU delta over the query phase | same | +| Accuracy | vs `ch_raw`: rank error for quantiles (as in #784), relative error for sums, recall@k for top-k | same | + +**Cost.** Use the AutoSketch plan's **model A** +(`$/h = a·avg vCPU + b·GiB`, with `a ≈ 0.0368` and `b ≈ 0.00364` from the +2026-10-04 EC2 fit) over measured ingest + query CPU and retained state memory. +Report model B (peak-provisioned, staggered) in the appendix. This keeps the +data-plane and planner sections in the same units. + +**Fairness** +- Same node type: `c6320` (2× E5-2683 v3, 56 threads, 251 GB). +- Both systems are pinned to the same 2 cores and 4 GiB, matching the + backend's matched-budget runners. +- ClickHouse image pinned by digest. Query cache off, `max_threads=2`. +- ClickHouse restarted per trial. +- 3 trials per point, alternating route order. Report the median, with + min/max error bars. + +## 7. Open decisions + +- **D0: which data plane is measured.** The ASAPQuery-backend (recommended: its + README describes it as the summary storage, precompute runtime and query + adapters, and its ClickHouse harness already exists), or the legacy `query_engine_rust` in this repo. +- **D1: the source of the plan.** `rqe-optimizer` (recommended, and the paper + calls the plan optimal) or the `asap-planner` SQL-mode generator, which works + with today's runner but is not cost-optimal. +- **D2: the ASAP query interface.** PromQL now (recommended: quantile and top-k + are warm there), or wait until the SQL adapter can bind quantile and top-k. +- **D3:** keep `ch_mv_exact` as an extra bar, or show only `ch_mv_sketch`. +- **D4:** the ClickHouse quantile state: `quantilesState`, `quantilesTDigest` + or `quantilesDD`, per workload. Proposal: report the one closest to ASAP in + accuracy and list the others in the appendix. +- **D5:** where this harness lives. The backend already has the matched runners. + This repo has the Hydra/CloudLab orchestration. + +## 8. Work items (each a small PR) + +1. Data generator: seeded Zipf (θ) weights and Pareto (a) values over + `(C, s, r)`, written as JSONL for ClickHouse plus Remote Write or the + OpenMetrics replay for ASAP from the same seed. Include a test on the + empirical rank-frequency slope. +2. Template set: the 10 synthetic templates as PromQL + ClickHouse SQL pairs, + with a differential test against `ch_raw` on a small `C, s`. +3. Plan → ClickHouse translator. It reads an `rqe-optimizer` plan and emits + `init.sql` (raw table + MVs per §4) and the MV-targeted query SQL. It logs + every unsupported mapping. +4. Plan → backend publication (or confirm the existing path), then the install + and backfill driver. +5. Runner. Either extend `experiment_run_clickhouse.py` with `ch_mv_exact`, + `ch_mv_sketch` and an `asap_backend` mode, or adapt the backend's + `run_matched.sh` (D5). In both cases, **one ClickHouse load per arm**, each + with its own `init.sql`. +6. Post-processing: ClickHouse and backend paths in the cost script, the model A + and model B pricing, and the figure script. + +## 9. Cluster and run procedure + +CloudLab experiment `scratch2.cloudmigration-pg0.clemson.cloudlab.us`. +`node1`–`node3` run 3 configs in parallel (`node_offset` 1/2/3). `node1` is +also the orchestrator. + +Setup: +1. `/mydata` (845 GB ext4) is already mounted, world-writable, on node1–3. The + setup scripts expect `/scratch`, so symlink it or make the root volume + configurable. +2. Docker, the Rust toolchain, release builds of the backend (and of + `asap-query-engine` / `asap-planner-rs` if D0 picks the legacy path), and + `rqe-optimizer`. +3. Generate datasets once into `/mydata/datasets/`, and rsync them to the nodes. + +Order: +1. **Pilot**: the default workload with `C=1e3`, `s=10`, all 4 arms, 1 trial. + Check: + - 0 ASAP fallbacks; + - MV aggregates equal `ch_raw` exactly for exact states; + - accuracy within target; + - corpus q05/q06/q23 reproduce the backend report's speedups (sanity). +2. **Sweep**: the §3.1 dimensions one at a time + (≈ 3+3+2+5+2+2 = 17 non-default points + default = 18 workloads), + × 4 arms × 3 trials = **216 runs**. Allowing about 15 min per run, that is + about 18 h on 3 nodes. +3. **Figure**: grouped bars of model-A $/h and p95 latency per workload, with + ASAP ÷ `ch_mv_sketch` annotated (the "up to X×" numbers). Each bar carries + its accuracy, and the "not supported in ClickHouse" plans are listed under + the figure. From 60410cd45707242b63fb262a469e9c67d62e1a6a Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 5 Oct 2026 17:42:49 +0000 Subject: [PATCH 2/6] docs(asap-tools): use ASAPQuery query engine as the data plane in eval plan Co-Authored-By: Claude Opus 5.5 --- .../DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md | 435 +++++++++--------- 1 file changed, 221 insertions(+), 214 deletions(-) diff --git a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md index b3d1320c..a1f512f1 100644 --- a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md +++ b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md @@ -1,17 +1,21 @@ -# Evaluation plan: ASAP data plane vs ClickHouse with MVs and summaries +# Evaluation plan: ASAPQuery data plane vs ClickHouse with MVs and summaries Status: **draft for discussion**, no results yet. Backs the paper subsection "ASAP's data plane". Companion to the planner-level plan in `docs/evaluation/autosketch-vs-planner.md` (branch -`docs/autosketch-vs-planner-eval`), whose synthetic workload this plan reuses. +`docs/autosketch-vs-planner-eval`), whose synthetic data model this plan reuses. + +The data plane under test is **ASAPQuery's** precompute engine and query +engine (`asap-query-engine`, run as `query_engine_rust`), driven by +`asap-tools/experiments/experiment_run_clickhouse.py`. ## 1. Question Fix a dataset and a query workload. Run the planner once and get its set of summary-based plans. Deploy that **same** plan set two ways: -1. **ASAP data plane**: the backend precompute runtime, SummaryStore and query - adapters serve the plan natively. +1. **ASAPQuery**: the precompute engine builds the planned summaries, and the + query engine answers SQL from them. 2. **ClickHouse**: the closest equivalent ClickHouse can express, which is materialized views (MVs) over the raw table that store ClickHouse's built-in aggregate states. @@ -19,293 +23,296 @@ summary-based plans. Deploy that **same** plan set two ways: How much cheaper and faster is (1) than (2), at what accuracy? The planner is held fixed, so this isolates the data plane. ClickHouse cannot -deploy MVs or summaries automatically, so the harness translates each plan by -hand. Where ClickHouse cannot express a planned summary, the translation records -the reason. That list becomes the paper's "ClickHouse may not support it" point: +deploy MVs or summaries automatically, so the harness translates each plan. +Where ClickHouse cannot express a planned summary, the translation records the +reason. That list becomes the paper's "ClickHouse may not support it" point: -- (a) the sketch is not implemented: Count-Min, KLL, keyed KLL; +- (a) the sketch is not implemented: Count-Min, KLL; - (b) the sketch's parameters are not configurable: `quantilesState`'s reservoir is fixed at 8192, `uniqHLL12` is fixed at 12 bits, and `topK`'s load factor is fixed. ## 2. What exists today -### 2.1 ASAPQuery (this repo): `experiment_run_clickhouse.py` - -Checked on `main` @ `52821e0`. This is a single-node runner (`num_nodes=1`; -`node_offset` picks the node). - -1. rsync the JSONL dataset to the node and start ClickHouse in Docker. -2. **Load the data once**, under an ingest monitor that samples the - `clickhouse` process. With `dataset.init_sql_file` set, that SQL owns all - DDL, *including MVs*. It runs before the load, so the MVs fill on insert. -3. Loop over the modes in `experiment_params.experiment`: - - **`sketchdb`**: - - Run `asap-planner` in SQL mode, using the **hardcoded generator, not - the cost optimizer**. - - Start the legacy `query_engine_rust`. It ingests the same JSONL itself - and forwards unsupported queries to ClickHouse. - - Wait `steady_state_wait`, then query `:8088`. - - **Any other mode** (`baseline`, `baseline_mv`, `baseline_sketch` or - `baseline_mv_sketch`, defined in `constants.py`): query ClickHouse - `:8123` with that mode's SQL file. `query_groups[i].sql_file` is a dict - keyed by mode. +Checked on `main` @ `52821e0`. + +### 2.1 `experiment_run_clickhouse.py` + +This is a single-node runner (`num_nodes=1`; `node_offset` picks the node). + +1. rsync the JSONL dataset to the node and start ClickHouse in Docker + (`ClickHouseService`). +2. **Load the data once** (`ClickHouseDataLoaderService`), under an ingest + monitor that samples the `clickhouse` process. With + `dataset.init_sql_file` set, that SQL owns all DDL, *including MVs*. It runs + before the load, so the MVs fill on insert. +3. Loop over the modes in `experiment_params.experiment` (`[{mode, server}]`). + + **`sketchdb`:** + - `generate_sql_planner_input` builds the planner input. + - `asap-planner` runs in SQL mode and writes `streaming_config.yaml` + + `inference_config.yaml`. + - `query_engine_rust` starts. It ingests the **same JSONL file** + (`ingest_json_config`), builds summaries, and forwards unsupported + queries to ClickHouse. + - After `flow.steady_state_wait`, the query client runs against `:8088` + under one continuous monitor that covers ingest + query, with per-thread + attribution (`pc-worker` threads = precompute). + + **Any other mode** (`baseline`, `baseline_mv`, `baseline_sketch` or + `baseline_mv_sketch`, defined in `constants.py`): the query client runs + against ClickHouse `:8123` with that mode's SQL file. +4. rsync the results back. + +`query_groups[i].sql_file` is a dict keyed by mode. `sketchdb` always reuses +`baseline`'s SQL text so the engine can pattern-match it. The mode names, per-mode SQL and MV-capable init SQL already exist. Nothing generates the MV DDL or the MV-targeted SQL. -### 2.2 ASAPQuery-backend: the current data plane - -The backend has the precompute runtime, SummaryStore and PromQL, MetricsQL and -ClickHouse SQL adapters, each with exact fallback. Its families include KLL, -DDSketch, HLL, CountMin, CountMin with heap and TopK. +### 2.2 SQL shapes ASAPQuery serves -The relevant prior ClickHouse work is in two places. +From `sql_utilities` (`sqlpattern_matcher.rs`, `sqlhelper.rs`): +- `SUM`, `COUNT`, `AVG`, `MIN`, `MAX` and `QUANTILE`, with `GROUP BY` labels + over a `WHERE ts BETWEEN …` window; +- `COUNT(DISTINCT col)`, normalised to `CARDINALITY` and served by HLL; +- top-k as `ORDER BY SUM|COUNT(..) DESC LIMIT k`, served by + `CountMinSketchWithHeap`; +- nested subqueries (spatial-of-temporal). -**`tools/o11y-sql-main-eval/`** -- A **27-query o11y corpus** (`corpus.json`): each query appears as PromQL, - MetricsQL and a ClickHouse SQL translation over - `raw_samples(metric, labels Map, ts_ms, value)`. -- Matched-budget runners: 2 pinned cores, 4 GiB, a pinned ClickHouse image, - restart per trial, query cache off, `max_threads=2`. -- Reported results: - - Only q05, q06 and q23 run warm, all three on exact MinMax. - - 18 queries fall back to exact ClickHouse. - - 6 queries OOM on both sides. - - The SQL frontend rejects ClickHouse-specific syntax: lambdas, Map access, - `mapConcat`. +There is no `rate`/`increase` and no binary operator between aggregates. -**`tools/clickhouse-benefit/`** -- A single-query warm-path probe against ClickHouse. -- A matched-trial runner. +Example of the shape the engine matches (from the deleted +`clickhouse_quantile_queries.sql`): -**Limit that shapes this plan.** The backend's *automatic* ClickHouse-SQL -binder supports only bounded scalar reductions (sum, count, min, max) today. -Quantile and top-k plans are **not** warm through the SQL adapter. They are -through the PromQL adapter (`quantile_over_time`, `sum by`, `rate`). +```sql +SELECT quantile(0.95)(v1) FROM h2o_groupby +WHERE timestamp BETWEEN '2024-01-01 00:00:00' AND '2024-01-01 00:00:10' +GROUP BY id1, id2; +``` ### 2.3 Planner -The cost-optimal planner used in the paper is sketch-bench `rqe-optimizer`, a -MILP. It reads the measured `AtomicCostTable` and implements the Model A and -Model B cost models from the AutoSketch plan. `asap-planner-rs/src/optimizer` -is still offline and PromQL-only, and the SQL-mode `asap-planner` uses the -hardcoded generator. +- The `sketchdb` mode uses `asap-planner`'s **hardcoded SQL-mode generator**. +- The cost optimizer in `asap-planner-rs/src/optimizer` is offline-only and + PromQL-only (`.design_docs/optimizer-v1-implementation-plan.md`). +- The cost-optimal planner in the AutoSketch plan is sketch-bench + `rqe-optimizer`, a MILP. + +See D1. ### 2.4 Gaps -| Gap | Where | +| Gap | Impact | |---|---| -| `config/experiment_type/clickhouse.yaml` is used in the runner docstring but was never committed | ASAPQuery | -| `generate_queries.py` (SQL workload generator) was deleted in #404 | ASAPQuery | -| The MV init SQL examples (`netflow_init.sql`, `quantile_demo/init.sql`) are referenced but absent | ASAPQuery | -| No Zipf/Pareto **JSONL** generator. The fake exporter's Zipf is Prometheus-only, with a fixed α=1.01 | ASAPQuery | -| Data loads once per run, so MV maintenance cost would be charged to every ClickHouse mode | ASAPQuery runner | -| `compare_costs.py` keys on the Prometheus process | ASAPQuery post-processing | -| No plan → ClickHouse MV translator | both | -| SQL adapter is not warm for quantile/top-k plans | backend | -| The setup scripts assume `/scratch`; these nodes use `/mydata` | ASAPQuery `cloudlab_setup` | +| `config/experiment_type/clickhouse.yaml` is used in the runner docstring but was never committed | Can't run from a clean checkout | +| `execution-utilities/benchmark/generate_queries.py` was deleted in #404 | No SQL workload generator | +| The MV init SQL examples (`netflow_init.sql`, `quantile_demo/init.sql`) are referenced in comments but absent | No template for the MV arms | +| No Zipf/Pareto **JSONL** generator. The fake exporter's Zipf is Prometheus-only, with a fixed α=1.01 | Can't sweep skew or cardinality | +| Data loads once per run, then the modes loop | MV maintenance would be charged to *every* ClickHouse mode, and `baseline` inserts would slow down. Run each arm separately (§4) | +| `sketchdb` requires the ClickHouse raw load for fallback | Report ASAP's cost without it, and require 0 fallbacks | +| No way to feed an externally chosen plan to `sketchdb` | Needed if D1 = `rqe-optimizer` | +| `post_experiment/single_experiment/compare_costs.py` keys on the Prometheus process | Needs ClickHouse and precompute paths | +| `cloudlab_setup/single_node/constants.sh` assumes `/scratch` | These nodes use `/mydata` (§9) | ## 3. Workload -Reuse the **synthetic workload from `autosketch-vs-planner.md` §6** so both -evaluation sections share data, queries and plans. Add the real o11y corpus as -a secondary point. +### 3.1 Data + +The data model is the one in `autosketch-vs-planner.md` §6, written as JSONL +for both systems. ASAPQuery's JSON ingest and ClickHouse's loader read the +**same file**. -### 3.1 Synthetic (main figure) +- Rows are `{ts, label_0, instance, value}`. +- `label_0` takes `C` values (the groups). `instance` takes `s` values per + group. +- Each series emits 100 samples/s of event time. +- Frequency and top-k: the weight of each key follows Zipf θ. +- Quantiles: values follow Pareto a. +- Generation is seeded, so every arm and trial reads identical bytes. -**Data** -- One metric, `data`, with labels `label_0` (`C` groups) and `instance` - (`s` series per group). -- Each series emits 100 samples/s. -- Frequency and top-k weights follow Zipf θ. Quantile values follow Pareto a. +Event time is replayed, not real time: generate a fixed span (default 2 h of +event time) and let each system ingest as fast as it can. Ingest throughput is +reported separately. -**Templates** -- The 10 PromQL templates from the AutoSketch plan: `sum by`, `topk by`, - `quantile by`, `sum_over_time`, `quantile_over_time`, `rate`, their spatial - aggregations, and the `quantile_over_time` ratio. -- Each one is paired here with a hand-checked **ClickHouse SQL** translation - over a flat table `data(ts_ms, label_0, instance, value)`. -- A flat table is used because Map columns are what break the backend SQL - frontend in the corpus results. +### 3.2 Query templates (SQL) -**Grid** (one dimension at a time around the bold defaults, as in that plan) +These are the AutoSketch plan's templates that ASAPQuery's SQL path can serve +(§2.2), plus count-distinct, which SQL adds. `W` is the window, and queries +repeat every `T` over the span. + +| # | SQL (per window) | Planned summary | Mirrors AutoSketch template | +|---|---|---|---| +| 1 | `SUM(value) … GROUP BY label_0` | Sum / CountMin | 1, 8 | +| 2 | `label_0, SUM(value) … GROUP BY label_0 ORDER BY 2 DESC LIMIT 3` | CountMin with heap | 2 | +| 3 | `quantile(q)(value) … GROUP BY label_0`, q ∈ {0.5, 0.75, 0.9, 0.95, 0.99} | KLL per group | 3 | +| 4 | `SUM(value) … GROUP BY label_0, instance` | Sum / CountMin per series | 4 | +| 5 | `quantile(q)(value) … GROUP BY label_0, instance`, same q | KLL per series | 5 | +| 6 | `COUNT(DISTINCT instance) … GROUP BY label_0` | HLL | none (SQL-only) | +| 7 | `MAX(value) … GROUP BY label_0` | MinMax | control | + +Dropped, because ASAPQuery SQL cannot express them: AutoSketch templates 6, 7 +and 9 (`rate`), and 10 (a ratio of two quantiles). They can come back if the +SQL path gains them. + +Templates 1 and 7 are exact on both systems. They are **parity controls**, +not wins. + +### 3.3 Grid + +One dimension at a time around the bold defaults: | Dimension | Values | |---|---| -| Query mix | **all 10**, frequency only, quantile only, top-k only | +| Query mix | **all 7**, frequency only {1, 4}, quantile only {3, 5}, top-k only {2} | | `C` | 1e2, **1e3**, 1e4, 1e5 | | `s` | 1, 10, **100** | | θ / a | θ ∈ {0, 0.5, **1.0**, 1.5}; a ∈ {1.1, **2**, 3} | -| Window set `W` | {1m}, {1m, 1h}, **{1m, 10m, 1h}** | +| Window `W` / repeat `T` | {10 s}, **{1 m}**, {10 m}; `T = W` (tumbling) | | Accuracy target | 90%, **95%**, 99% | -The defaults are capped at `W` ≤ 1h and `C·s` = 1e5 series × 100 samples/s -(1e7 samples/s). That is too much for one ClickHouse node at real time, so the -data plane runs on **replayed time**: - -- generate a fixed span (default: 2 h of event time); -- ingest it as fast as each system accepts; -- run each RQE at every `T` over that span. - -The rate is a knob, not a fixed point: 1e5 samples/s by default, with ingest -throughput reported separately. - -### 3.2 Real o11y corpus (secondary) - -The backend's 27-query corpus already has a correctness matrix. Use only the -queries whose plans actually install. Re-run the corpus audit first, since -today that is q05, q06 and q23. Report them as "real workload" bars. - ## 4. Arms -All arms see the same rows and the same plan set. +All arms see the same rows and the same plan set. **Each arm is a separate +runner invocation** (`experiment=[]`), with its own `init_sql_file`, +so ingest cost isn't cross-charged. -| Arm | What is deployed | Role | +| Arm (runner mode) | Init SQL | Role | |---|---|---| -| `ch_raw` | ClickHouse raw `MergeTree`, no MVs | Exact reference and accuracy ground truth | -| `ch_mv_exact` | ClickHouse with MVs per plan, exact states (`sumState`, `countState`, `min/maxState`, `quantilesExactState`, per-key `count` for top-k) | "MVs without summaries" | -| `ch_mv_sketch` | ClickHouse with MVs per plan, closest summary state (table below) | **Main ClickHouse point** | -| `asap` | Backend data plane with the same plan set | **Main ASAP point** | +| `baseline` | raw `MergeTree` only | Exact reference and accuracy ground truth | +| `baseline_mv` | raw + MVs with exact states (`sumState`, `countState`, `maxState`, `quantilesExactState`, `uniqExactState`, per-key `sumState` for top-k) | "MVs without summaries" | +| `baseline_mv_sketch` | raw + MVs with the closest summary state (table below) | **Main ClickHouse point** | +| `sketchdb` | raw only (for fallback) | **ASAPQuery**. Must report 0 fallbacks | -Each MV is keyed by `(toStartOfInterval(ts, x), group-by labels)`, where `x` -is the plan's window/pane. Queries merge `S / x` panes with the `-Merge` -combinators, mirroring the plan's `(x, y)` and merge pattern. +Each MV is keyed by `(toStartOfInterval(ts, W), group-by labels)`. The +`baseline_mv*` SQL files query the MV target tables with the `-Merge` +combinators. Every mode's `sql_file` entry in `query_groups[i].sql_file` +covers the same logical query. -Plan → ClickHouse mapping for `ch_mv_sketch`: +Plan → ClickHouse mapping for `baseline_mv_sketch`: | Planned summary | ClickHouse state | Gap reported | |---|---|---| -| Sum / count | `sumState` / `countState` | none: an exact parity control | -| MinMax | `minState` / `maxState` | none: a control (matches corpus q05/q06/q23) | -| KLL (per group or per series) | `quantilesState(levels)(v)` (reservoir 8192) or `quantilesTDigestState` | No KLL. Size/accuracy cannot be set: a reservoir is either over-sized or misses the target | -| DDSketch | `quantilesDDState(relative_accuracy, …)` | Closest match. Parameterized, so expected near-parity. Report it honestly | -| CountMin (frequency) | none: exact `sumState` per key | Not implemented. Cost scales with key cardinality, so state is `C·s` rows, not `d×w` | -| CountMin with heap / TopK | `topKWeightedState(k)` (Filtered Space-Saving) | No CMS. The load factor cannot be set | -| HLL | `uniqHLL12State` / `uniqCombinedState(p)` | `uniqHLL12` is fixed at 12 bits; `uniqCombined` lets you set the precision | - -The translator emits every row of this table. A plan that falls into a "none" -row becomes an exact fallback and is counted. - -## 5. Running ASAP on the same plan - -1. Run `rqe-optimizer` on the workload, with the model A cost and the default - accuracy target. Its output is the plan set. -2. Translate it into a backend `PhysicalPlanInstallRequest` / `PlanEnvelope` - publication. The compiler path already exists for Planner DAGs. **Check - with the backend owners whether an `rqe-optimizer` plan can be published - directly, or only through ASAPPlanner.** -3. Ingest: - - **ASAP**: Prometheus Remote Write (or the backfill reader from the - ClickHouse raw table, `storage_engines/sketch_db/backfill/clickhouse_reader.rs`). - - **ClickHouse**: JSONL bulk insert. - - Report the ingest cost of each path separately. Do not charge ASAP for the - ClickHouse raw load unless fallback is needed. -4. Queries: - - **ASAP**: PromQL against the backend. Quantile and top-k are not warm - through the SQL adapter yet (§2.2). - - **ClickHouse**: the paired SQL. - - Correctness is checked per query, as the corpus runner does it: compare the - decoded results against `ch_raw` within the accuracy target. -5. Require **0 fallbacks** on the ASAP side for every RQE counted as served. - Count and report any fallback separately, as the corpus report does. - -If the team prefers the legacy ASAPQuery `query_engine_rust` path in -`experiment_run_clickhouse.py`, which is SQL end to end, steps 2 and 4 change. -That path cannot take an `rqe-optimizer` plan either (decision D1). +| Sum / count | `sumState` / `countState` | none: parity control | +| MinMax | `maxState` | none: parity control | +| KLL (`DatasketchesKLL` / `HydraKLL`) | `quantilesState(levels)(v)` (reservoir 8192) or `quantilesTDigestState` | No KLL. Size/accuracy cannot be set | +| CountMin (`CountMinSketch`) | none: exact `sumState` per key | Not implemented. State is one row per key, not `d×w` | +| CountMin with heap | `topKWeightedState(k)` (Filtered Space-Saving) | No CMS. The load factor cannot be set | +| HLL | `uniqHLL12State` or `uniqCombinedState(p)` | `uniqHLL12` is fixed at 12 bits. `uniqCombined` lets you set p | + +The translator emits every row of this table. A "none" row becomes an exact +state, and it is counted and listed under the figure. + +## 5. Plan source and deployment + +The plan source depends on D1: + +- **D1 = `asap-planner` (works today).** `sketchdb` already runs it. + 1. Run `sketchdb` first. + 2. Take `controller_output/streaming_config.yaml`: the summary type, + parameters, window and group-by of each deployment. + 3. Generate the ClickHouse MV arms from it. +- **D1 = `rqe-optimizer`.** + 1. Translate its plan into ASAPQuery's `streaming_config.yaml` + + `inference_config.yaml`. + 2. Add a runner option that skips `asap-planner` and rsyncs the given + configs. + 3. Generate the ClickHouse arms from the same plan. + +Either way, the translator reads **one** plan file and writes both the ASAPQuery +configs and the ClickHouse `init.sql`, so the two systems cannot drift apart. + +`sketchdb` ingests by itself from the JSONL. Its cost excludes the ClickHouse +raw load, which exists only for fallback. Every RQE counted as served must show +0 forwards to ClickHouse, taken from the query-engine logs. ## 6. Metrics -| Metric | ClickHouse | ASAP | +| Metric | ClickHouse arms | `sketchdb` | |---|---|---| -| Ingest CPU (core-s), including MV/summary maintenance | `clickhouse` process during load (existing ingest monitor) | backend precompute threads | -| Retained state bytes | `system.parts.bytes_on_disk` + `primary_key_bytes_in_memory` for the MV target tables (raw table reported separately) | SummaryStore state directory + in-memory instances | +| Ingest CPU (core-s), including MV/summary maintenance | `monitor_output_ingest.json` (`clickhouse` process) | `pc-worker` threads in `monitor_output.json` | +| Retained state bytes | `system.parts.bytes_on_disk` + `primary_key_bytes_in_memory` for the MV target tables (raw table reported separately) | query-engine store size (to add if not already exported) | | RSS (steady and peak) | process | process | -| Query latency p50/p95/p99 | client-side per request, warm (10 warm-ups discarded) | same | -| Query CPU | process CPU delta over the query phase | same | -| Accuracy | vs `ch_raw`: rank error for quantiles (as in #784), relative error for sums, recall@k for top-k | same | +| Query latency p50/p95/p99 | prometheus-client per query (`analyze_latencies.py`) | same | +| Query CPU | monitor during the query phase | same | +| Accuracy | vs `baseline` (`calculate_fidelity.py --exact_experiment_mode baseline`): rank error for quantiles (as in #784), relative error for sums, recall@k for top-k, relative error for distinct counts | same | **Cost.** Use the AutoSketch plan's **model A** (`$/h = a·avg vCPU + b·GiB`, with `a ≈ 0.0368` and `b ≈ 0.00364` from the 2026-10-04 EC2 fit) over measured ingest + query CPU and retained state memory. -Report model B (peak-provisioned, staggered) in the appendix. This keeps the -data-plane and planner sections in the same units. +This keeps the planner and data-plane sections in the same units. Report raw +CPU-seconds and bytes as well. **Fairness** - Same node type: `c6320` (2× E5-2683 v3, 56 threads, 251 GB). -- Both systems are pinned to the same 2 cores and 4 GiB, matching the - backend's matched-budget runners. -- ClickHouse image pinned by digest. Query cache off, `max_threads=2`. -- ClickHouse restarted per trial. -- 3 trials per point, alternating route order. Report the median, with - min/max error bars. +- ClickHouse and `query_engine_rust` get the same cores and memory cap + (`docker --cpuset-cpus/--memory` and `taskset`/cgroup). +- ClickHouse image pinned by tag or digest (`dataset.clickhouse_image_tag`), + not `latest`. +- Query cache off (`use_query_cache=0`). The first query pass is a discarded + warm-up. +- 3 trials per point. Report the median, with min/max error bars. ## 7. Open decisions -- **D0: which data plane is measured.** The ASAPQuery-backend (recommended: its - README describes it as the summary storage, precompute runtime and query - adapters, and its ClickHouse harness already exists), or the legacy `query_engine_rust` in this repo. -- **D1: the source of the plan.** `rqe-optimizer` (recommended, and the paper - calls the plan optimal) or the `asap-planner` SQL-mode generator, which works - with today's runner but is not cost-optimal. -- **D2: the ASAP query interface.** PromQL now (recommended: quantile and top-k - are warm there), or wait until the SQL adapter can bind quantile and top-k. -- **D3:** keep `ch_mv_exact` as an extra bar, or show only `ch_mv_sketch`. -- **D4:** the ClickHouse quantile state: `quantilesState`, `quantilesTDigest` - or `quantilesDD`, per workload. Proposal: report the one closest to ASAP in - accuracy and list the others in the appendix. -- **D5:** where this harness lives. The backend already has the matched runners. - This repo has the Hydra/CloudLab orchestration. +- **D0: data plane.** Decided: ASAPQuery `query_engine_rust` + (`experiment_run_clickhouse.py` `sketchdb`). +- **D1: plan source.** `asap-planner`'s SQL generator works today, but it is + not cost-optimal, so the paper should not call it "optimal". `rqe-optimizer` + matches the paper's claim but needs the plan translator and a runner option. + Proposal: run the pilot with `asap-planner`, and switch the sweep to + `rqe-optimizer` once the translator lands. +- **D2:** keep `baseline_mv` as an extra bar, or show only `baseline_mv_sketch`. +- **D3:** the ClickHouse quantile state, `quantilesState` or + `quantilesTDigestState`. Report the one closest to ASAP in accuracy, and put + the other in the appendix. +- **D4:** whether to add `rate` or a quantile ratio to the ASAPQuery SQL path, + to cover all 10 AutoSketch templates. ## 8. Work items (each a small PR) -1. Data generator: seeded Zipf (θ) weights and Pareto (a) values over - `(C, s, r)`, written as JSONL for ClickHouse plus Remote Write or the - OpenMetrics replay for ASAP from the same seed. Include a test on the - empirical rank-frequency slope. -2. Template set: the 10 synthetic templates as PromQL + ClickHouse SQL pairs, - with a differential test against `ch_raw` on a small `C, s`. -3. Plan → ClickHouse translator. It reads an `rqe-optimizer` plan and emits - `init.sql` (raw table + MVs per §4) and the MV-targeted query SQL. It logs - every unsupported mapping. -4. Plan → backend publication (or confirm the existing path), then the install - and backfill driver. -5. Runner. Either extend `experiment_run_clickhouse.py` with `ch_mv_exact`, - `ch_mv_sketch` and an `asap_backend` mode, or adapt the backend's - `run_matched.sh` (D5). In both cases, **one ClickHouse load per arm**, each - with its own `init.sql`. -6. Post-processing: ClickHouse and backend paths in the cost script, the model A - and model B pricing, and the figure script. +1. Data generator: seeded JSONL with Zipf θ weights and Pareto a values over + `(C, s)`, plus a test on the empirical rank-frequency slope. +2. SQL workload generator for templates 1–7 over the grid (replaces the + deleted `generate_queries.py`), with a differential test against ClickHouse + on a small `C, s`. +3. Plan translator: one plan file in; the ASAPQuery configs and the ClickHouse + `init.sql` + per-mode SQL out. It logs every unsupported mapping. +4. Runner: + - commit `config/experiment_type/clickhouse.yaml`; + - add an option to use a given plan instead of running `asap-planner` (D1); + - collect MV bytes from `system.parts` after the load. +5. Post-processing: ClickHouse and precompute paths in `compare_costs.py`, + model-A pricing, and the figure script. +6. Sweep driver that fans out (workload, arm, trial) across node1–3. ## 9. Cluster and run procedure CloudLab experiment `scratch2.cloudmigration-pg0.clemson.cloudlab.us`. -`node1`–`node3` run 3 configs in parallel (`node_offset` 1/2/3). `node1` is -also the orchestrator. +`node1`–`node3` run 3 configs in parallel (`node_offset` 1/2/3, +`num_nodes=1`). `node1` is also the orchestrator. Setup: 1. `/mydata` (845 GB ext4) is already mounted, world-writable, on node1–3. The setup scripts expect `/scratch`, so symlink it or make the root volume configurable. -2. Docker, the Rust toolchain, release builds of the backend (and of - `asap-query-engine` / `asap-planner-rs` if D0 picks the legacy path), and - `rqe-optimizer`. -3. Generate datasets once into `/mydata/datasets/`, and rsync them to the nodes. +2. `deploy_from_scratch.sh` without its storage step: Docker, the Rust + toolchain, and `cargo build --release` for `asap-query-engine` and + `asap-planner-rs`. +3. The orchestrator needs SSH to `node{1,2,3}.`, including itself. +4. Generate datasets once into `/mydata/datasets/`. The runner rsyncs them. Order: 1. **Pilot**: the default workload with `C=1e3`, `s=10`, all 4 arms, 1 trial. Check: - - 0 ASAP fallbacks; - - MV aggregates equal `ch_raw` exactly for exact states; - - accuracy within target; - - corpus q05/q06/q23 reproduce the backend report's speedups (sanity). -2. **Sweep**: the §3.1 dimensions one at a time - (≈ 3+3+2+5+2+2 = 17 non-default points + default = 18 workloads), + - 0 fallbacks in `sketchdb`; + - exact MV results equal `baseline`; + - accuracy is within target. +2. **Sweep**: the §3.3 dimensions one at a time + (3+3+2+5+2+2 = 17 non-default points + default = 18 workloads), × 4 arms × 3 trials = **216 runs**. Allowing about 15 min per run, that is about 18 h on 3 nodes. 3. **Figure**: grouped bars of model-A $/h and p95 latency per workload, with - ASAP ÷ `ch_mv_sketch` annotated (the "up to X×" numbers). Each bar carries - its accuracy, and the "not supported in ClickHouse" plans are listed under - the figure. + `sketchdb` ÷ `baseline_mv_sketch` annotated (the "up to X×" numbers). Each + bar carries its accuracy, and the plans ClickHouse can't express are listed + under the figure. From 29ba3ce7b8cc00cc7dc986fa6c80cb450917a463 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 5 Oct 2026 17:50:32 +0000 Subject: [PATCH 3/6] docs(asap-tools): take eval plans from sketch-bench rqe-optimizer Co-Authored-By: Claude Opus 5.5 --- .../DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md | 167 ++++++++++++------ 1 file changed, 111 insertions(+), 56 deletions(-) diff --git a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md index a1f512f1..71730d12 100644 --- a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md +++ b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md @@ -91,15 +91,40 @@ WHERE timestamp BETWEEN '2024-01-01 00:00:00' AND '2024-01-01 00:00:10' GROUP BY id1, id2; ``` -### 2.3 Planner - -- The `sketchdb` mode uses `asap-planner`'s **hardcoded SQL-mode generator**. -- The cost optimizer in `asap-planner-rs/src/optimizer` is offline-only and - PromQL-only (`.design_docs/optimizer-v1-implementation-plan.md`). -- The cost-optimal planner in the AutoSketch plan is sketch-bench - `rqe-optimizer`, a MILP. +### 2.3 Planner: sketch-bench `rqe-optimizer` + +The plan comes from **sketch-bench `rqe-optimizer`**, the MILP used in the +AutoSketch plan (checked at sketch-bench `964ccb5`). The `sketchdb` mode +currently runs `asap-planner`'s hardcoded SQL generator instead, so the runner +needs to bypass it (§5). + +What `rqe-optimizer` provides today: +- **A library, not a CLI.** RQEs and label sets are built in Rust + (`examples/small_problem.rs`), and `milp::minimize_cost` solves them. +- **Input:** + - `Rqe {capability, lookback S, interval T, labels, accuracy metric/tolerance}`; + - `LabelSetInfo {cardinality, arrival_rate}`; + - the measured `AtomicCostTable` from `scripts/export_rqe_optimizer_costs.sh`. +- **Output:** a `Mapping` from each RQE to a + `Deployment {capability, labels, config: AtomicCostEntry, window x, slide y}`. + `config.sketch` is a sketch-bench variant, and `config.sketch_config` holds + its parameters. It is not written to a file. +- **Capabilities:** `Freq`, `Quantile`, `Cardinality` and `TopK`. Exact + Sum/MinMax are not capabilities. + +Mapping its variants to ASAPQuery `aggregationType`s: + +| `rqe-optimizer` variant | ASAPQuery `aggregationType` | Status | +|---|---|---| +| `cms-fastpath-vector2d` `{rows, cols}` | `CountMinSketch` `{depth, width}` | ok | +| `cms-heap-topk-fastpath-vector2d` `{rows, cols}` | `CountMinSketchWithHeap` `{depth, width, heap}` | ok | +| `kll-percall` `{k}` | `DatasketchesKLL` `{K}` (per group via `labels.grouping`) | ok | +| `hll` `{precision}` | `HLL` | ok | +| `countsketch-fastpath-vector2d`, `dd`, `univmon-cardinality` | none | **Exclude from candidates.** ASAPQuery can't deploy them | -See D1. +The optimizer's costs are measured on sketch-bench's implementations, not +ASAPQuery's. The plan's *estimated* cost is reported next to the *measured* one, +and the gap between them is a result in its own right. ### 2.4 Gaps @@ -111,7 +136,8 @@ See D1. | No Zipf/Pareto **JSONL** generator. The fake exporter's Zipf is Prometheus-only, with a fixed α=1.01 | Can't sweep skew or cardinality | | Data loads once per run, then the modes loop | MV maintenance would be charged to *every* ClickHouse mode, and `baseline` inserts would slow down. Run each arm separately (§4) | | `sketchdb` requires the ClickHouse raw load for fallback | Report ASAP's cost without it, and require 0 fallbacks | -| No way to feed an externally chosen plan to `sketchdb` | Needed if D1 = `rqe-optimizer` | +| No way to feed an externally chosen plan to `sketchdb`: it always runs `asap-planner` | Needed to deploy the `rqe-optimizer` plan | +| `rqe-optimizer` has no workload-file input or plan-file output | Needed to drive it from the sweep | | `post_experiment/single_experiment/compare_costs.py` keys on the Prometheus process | Needs ClickHouse and precompute paths | | `cloudlab_setup/single_node/constants.sh` assumes `/scratch` | These nodes use `/mydata` (§9) | @@ -141,22 +167,23 @@ These are the AutoSketch plan's templates that ASAPQuery's SQL path can serve (§2.2), plus count-distinct, which SQL adds. `W` is the window, and queries repeat every `T` over the span. -| # | SQL (per window) | Planned summary | Mirrors AutoSketch template | +| # | SQL (per window) | `rqe-optimizer` RQE (capability, labels) | Mirrors AutoSketch template | |---|---|---|---| -| 1 | `SUM(value) … GROUP BY label_0` | Sum / CountMin | 1, 8 | -| 2 | `label_0, SUM(value) … GROUP BY label_0 ORDER BY 2 DESC LIMIT 3` | CountMin with heap | 2 | -| 3 | `quantile(q)(value) … GROUP BY label_0`, q ∈ {0.5, 0.75, 0.9, 0.95, 0.99} | KLL per group | 3 | -| 4 | `SUM(value) … GROUP BY label_0, instance` | Sum / CountMin per series | 4 | -| 5 | `quantile(q)(value) … GROUP BY label_0, instance`, same q | KLL per series | 5 | -| 6 | `COUNT(DISTINCT instance) … GROUP BY label_0` | HLL | none (SQL-only) | -| 7 | `MAX(value) … GROUP BY label_0` | MinMax | control | +| 1 | `SUM(value) … GROUP BY label_0` | Freq, `{label_0}` → CMS | 1, 8 | +| 2 | `label_0, SUM(value) … GROUP BY label_0 ORDER BY 2 DESC LIMIT 3` | TopK, `{label_0}` → CMS with heap | 2 | +| 3 | `quantile(q)(value) … GROUP BY label_0`, q ∈ {0.5, 0.75, 0.9, 0.95, 0.99} | Quantile, `{label_0}` → KLL per group | 3 | +| 4 | `SUM(value) … GROUP BY label_0, instance` | Freq, `{label_0, instance}` → CMS | 4 | +| 5 | `quantile(q)(value) … GROUP BY label_0, instance`, same q | Quantile, `{label_0, instance}` → KLL per series | 5 | +| 6 | `COUNT(DISTINCT instance) … GROUP BY label_0` | Cardinality, `{label_0}` → HLL | none (SQL-only) | -Dropped, because ASAPQuery SQL cannot express them: AutoSketch templates 6, 7 -and 9 (`rate`), and 10 (a ratio of two quantiles). They can come back if the -SQL path gains them. +Each template is one RQE per window in `W`, repeating every `T`. Quantiles of +one template share a stream, so the optimizer can serve all five from one +deployment. -Templates 1 and 7 are exact on both systems. They are **parity controls**, -not wins. +Dropped, because ASAPQuery SQL cannot express them: AutoSketch templates 6, 7 +and 9 (`rate`), and 10 (a ratio of two quantiles). The earlier `MAX` control is +also dropped: `rqe-optimizer` has no MinMax capability, so it would not come +from the plan. ### 3.3 Grid @@ -164,7 +191,7 @@ One dimension at a time around the bold defaults: | Dimension | Values | |---|---| -| Query mix | **all 7**, frequency only {1, 4}, quantile only {3, 5}, top-k only {2} | +| Query mix | **all 6**, frequency only {1, 4}, quantile only {3, 5}, top-k only {2} | | `C` | 1e2, **1e3**, 1e4, 1e5 | | `s` | 1, 10, **100** | | θ / a | θ ∈ {0, 0.5, **1.0**, 1.5}; a ∈ {1.1, **2**, 3} | @@ -193,9 +220,7 @@ Plan → ClickHouse mapping for `baseline_mv_sketch`: | Planned summary | ClickHouse state | Gap reported | |---|---|---| -| Sum / count | `sumState` / `countState` | none: parity control | -| MinMax | `maxState` | none: parity control | -| KLL (`DatasketchesKLL` / `HydraKLL`) | `quantilesState(levels)(v)` (reservoir 8192) or `quantilesTDigestState` | No KLL. Size/accuracy cannot be set | +| KLL (`DatasketchesKLL`) | `quantilesState(levels)(v)` (reservoir 8192) or `quantilesTDigestState` | No KLL. Size/accuracy cannot be set | | CountMin (`CountMinSketch`) | none: exact `sumState` per key | Not implemented. State is one row per key, not `d×w` | | CountMin with heap | `topKWeightedState(k)` (Filtered Space-Saving) | No CMS. The load factor cannot be set | | HLL | `uniqHLL12State` or `uniqCombinedState(p)` | `uniqHLL12` is fixed at 12 bits. `uniqCombined` lets you set p | @@ -205,22 +230,44 @@ state, and it is counted and listed under the figure. ## 5. Plan source and deployment -The plan source depends on D1: - -- **D1 = `asap-planner` (works today).** `sketchdb` already runs it. - 1. Run `sketchdb` first. - 2. Take `controller_output/streaming_config.yaml`: the summary type, - parameters, window and group-by of each deployment. - 3. Generate the ClickHouse MV arms from it. -- **D1 = `rqe-optimizer`.** - 1. Translate its plan into ASAPQuery's `streaming_config.yaml` + - `inference_config.yaml`. - 2. Add a runner option that skips `asap-planner` and rsyncs the given - configs. - 3. Generate the ClickHouse arms from the same plan. - -Either way, the translator reads **one** plan file and writes both the ASAPQuery -configs and the ClickHouse `init.sql`, so the two systems cannot drift apart. +The pipeline for one (dataset, workload) point: + +1. **Workload → RQEs.** From the template set, `W`, `T`, `C`, `s` and the + accuracy target, emit `workload.json`: + - one RQE per (template, window); + - one label set per grouping, with `cardinality` (`C`, or `C·s`) and + `arrival_rate` (`100·C·s`/s at the replay rate). + + Accuracy metrics follow `small_problem.rs`: + - `are_top100` for Freq; + - `mean_rank_err` for Quantile; + - `relative_error` for Cardinality; + - `precision_at_k` (a floor) for TopK. +2. **Solve.** A new `rqe-optimizer` binary (in sketch-bench) reads + `workload.json` and the `AtomicCostTable`, with the candidate families + restricted to the ASAPQuery-deployable variants (§2.3). It runs + the objective chosen in D1a and writes `plan.json`: + - per deployment: capability, labels, sketch variant + `sketch_config`, + `x`, `y`, and the RQEs it serves; + - the MILP's estimated CPU, memory and $/h. +3. **Translate** (new, in this repo). `plan.json` is the single source for + every arm: + - ASAPQuery `streaming_config.yaml`: one aggregation per deployment, with + `aggregationType`/`parameters` from §2.3, `labels.grouping`, + `windowSizeMs = x` and `windowType` tumbling if `x = y`, else sliding with + slide `y`; + - `inference_config.yaml`: each RQE's SQL → its aggregation ID; + - the ClickHouse `init.sql` for `baseline_mv` and `baseline_mv_sketch` + (§4), plus the per-mode SQL files. + + The two systems therefore cannot drift apart. +4. **Deploy.** Add a runner option (e.g. `controller.plan_dir=…`) that skips + `asap-planner` in `sketchdb` mode and rsyncs the translated configs into + `controller_output/`. Everything after that stays as it is. + +**Check before building:** whether `query_engine_rust` accepts sliding windows +(`x ≠ y`) for every summary type from SQL. If not, constrain the optimizer to +`y = x` and note it. `sketchdb` ingests by itself from the JSONL. Its cost excludes the ClickHouse raw load, which exists only for fallback. Every RQE counted as served must show @@ -257,11 +304,13 @@ CPU-seconds and bytes as well. - **D0: data plane.** Decided: ASAPQuery `query_engine_rust` (`experiment_run_clickhouse.py` `sketchdb`). -- **D1: plan source.** `asap-planner`'s SQL generator works today, but it is - not cost-optimal, so the paper should not call it "optimal". `rqe-optimizer` - matches the paper's claim but needs the plan translator and a runner option. - Proposal: run the pilot with `asap-planner`, and switch the sweep to - `rqe-optimizer` once the translator lands. +- **D1: plan source.** Decided: sketch-bench `rqe-optimizer` (§2.3, §5). +- **D1a: the optimizer objective.** At sketch-bench `964ccb5`, `rqe-optimizer` + offers `minimize_tco` (minimum steady-state CPU) and `minimize_cost(family)` + (fractional EC2 instances of one family, from #137). The model A and model B + objectives in the AutoSketch plan are not in the code yet. Proposal: use + `minimize_cost` with `general_purpose` now, and switch to model A when it + lands. Every arm is still measured and priced under model A. - **D2:** keep `baseline_mv` as an extra bar, or show only `baseline_mv_sketch`. - **D3:** the ClickHouse quantile state, `quantilesState` or `quantilesTDigestState`. Report the one closest to ASAP in accuracy, and put @@ -273,18 +322,23 @@ CPU-seconds and bytes as well. 1. Data generator: seeded JSONL with Zipf θ weights and Pareto a values over `(C, s)`, plus a test on the empirical rank-frequency slope. -2. SQL workload generator for templates 1–7 over the grid (replaces the - deleted `generate_queries.py`), with a differential test against ClickHouse - on a small `C, s`. -3. Plan translator: one plan file in; the ASAPQuery configs and the ClickHouse +2. Workload generator for templates 1–6 over the grid. It emits both the SQL + files (replacing the deleted `generate_queries.py`) and `workload.json` for + `rqe-optimizer`, and includes a differential test against ClickHouse on a + small `C, s`. +3. **sketch-bench PR:** an `rqe-optimizer` binary with + `workload.json` + cost table → `plan.json`, plus a flag restricting the + candidate families. +4. Plan translator: `plan.json` in; the ASAPQuery configs and the ClickHouse `init.sql` + per-mode SQL out. It logs every unsupported mapping. -4. Runner: +5. Runner: - commit `config/experiment_type/clickhouse.yaml`; - - add an option to use a given plan instead of running `asap-planner` (D1); + - add `controller.plan_dir` to skip `asap-planner` and deploy the + translated configs; - collect MV bytes from `system.parts` after the load. -5. Post-processing: ClickHouse and precompute paths in `compare_costs.py`, +6. Post-processing: ClickHouse and precompute paths in `compare_costs.py`, model-A pricing, and the figure script. -6. Sweep driver that fans out (workload, arm, trial) across node1–3. +7. Sweep driver that fans out (workload, arm, trial) across node1–3. ## 9. Cluster and run procedure @@ -298,7 +352,8 @@ Setup: configurable. 2. `deploy_from_scratch.sh` without its storage step: Docker, the Rust toolchain, and `cargo build --release` for `asap-query-engine` and - `asap-planner-rs`. + `asap-planner-rs`, plus sketch-bench (`rqe-optimizer`, and the cost table + from `scripts/export_rqe_optimizer_costs.sh`). 3. The orchestrator needs SSH to `node{1,2,3}.`, including itself. 4. Generate datasets once into `/mydata/datasets/`. The runner rsyncs them. From 0ef4d8e27962059e1bb027153cb35615ed1c6808 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 5 Oct 2026 18:49:04 +0000 Subject: [PATCH 4/6] docs(asap-tools): record settled decisions for data-plane eval plan Quantile-only first pass with DDSketch on both sides, plans from asap-planner --planner milp, paced ingest at 1 sample/s per series, 8 workloads x 4 arms x 3 trials, 16-core caps. Co-Authored-By: Claude Opus 5.5 --- .../DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md | 517 +++++++----------- 1 file changed, 186 insertions(+), 331 deletions(-) diff --git a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md index 71730d12..29f08822 100644 --- a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md +++ b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md @@ -1,373 +1,228 @@ # Evaluation plan: ASAPQuery data plane vs ClickHouse with MVs and summaries -Status: **draft for discussion**, no results yet. Backs the paper subsection -"ASAP's data plane". Companion to the planner-level plan in +Status: **design settled, implementation not started**. Backs the paper +subsection "ASAP's data plane". The data model follows the synthetic workload in `docs/evaluation/autosketch-vs-planner.md` (branch -`docs/autosketch-vs-planner-eval`), whose synthetic data model this plan reuses. - -The data plane under test is **ASAPQuery's** precompute engine and query -engine (`asap-query-engine`, run as `query_engine_rust`), driven by -`asap-tools/experiments/experiment_run_clickhouse.py`. +`docs/autosketch-vs-planner-eval`), narrowed as described in §3. ## 1. Question -Fix a dataset and a query workload. Run the planner once and get its set of -summary-based plans. Deploy that **same** plan set two ways: - -1. **ASAPQuery**: the precompute engine builds the planned summaries, and the - query engine answers SQL from them. -2. **ClickHouse**: the closest equivalent ClickHouse can express, which is - materialized views (MVs) over the raw table that store ClickHouse's built-in - aggregate states. - -How much cheaper and faster is (1) than (2), at what accuracy? - -The planner is held fixed, so this isolates the data plane. ClickHouse cannot -deploy MVs or summaries automatically, so the harness translates each plan. -Where ClickHouse cannot express a planned summary, the translation records the -reason. That list becomes the paper's "ClickHouse may not support it" point: - -- (a) the sketch is not implemented: Count-Min, KLL; -- (b) the sketch's parameters are not configurable: `quantilesState`'s - reservoir is fixed at 8192, `uniqHLL12` is fixed at 12 bits, and `topK`'s - load factor is fixed. - -## 2. What exists today - -Checked on `main` @ `52821e0`. - -### 2.1 `experiment_run_clickhouse.py` - -This is a single-node runner (`num_nodes=1`; `node_offset` picks the node). - -1. rsync the JSONL dataset to the node and start ClickHouse in Docker - (`ClickHouseService`). -2. **Load the data once** (`ClickHouseDataLoaderService`), under an ingest - monitor that samples the `clickhouse` process. With - `dataset.init_sql_file` set, that SQL owns all DDL, *including MVs*. It runs - before the load, so the MVs fill on insert. -3. Loop over the modes in `experiment_params.experiment` (`[{mode, server}]`). - - **`sketchdb`:** - - `generate_sql_planner_input` builds the planner input. - - `asap-planner` runs in SQL mode and writes `streaming_config.yaml` + - `inference_config.yaml`. - - `query_engine_rust` starts. It ingests the **same JSONL file** - (`ingest_json_config`), builds summaries, and forwards unsupported - queries to ClickHouse. - - After `flow.steady_state_wait`, the query client runs against `:8088` - under one continuous monitor that covers ingest + query, with per-thread - attribution (`pc-worker` threads = precompute). - - **Any other mode** (`baseline`, `baseline_mv`, `baseline_sketch` or - `baseline_mv_sketch`, defined in `constants.py`): the query client runs - against ClickHouse `:8123` with that mode's SQL file. -4. rsync the results back. - -`query_groups[i].sql_file` is a dict keyed by mode. `sketchdb` always reuses -`baseline`'s SQL text so the engine can pattern-match it. - -The mode names, per-mode SQL and MV-capable init SQL already exist. Nothing -generates the MV DDL or the MV-targeted SQL. - -### 2.2 SQL shapes ASAPQuery serves - -From `sql_utilities` (`sqlpattern_matcher.rs`, `sqlhelper.rs`): -- `SUM`, `COUNT`, `AVG`, `MIN`, `MAX` and `QUANTILE`, with `GROUP BY` labels - over a `WHERE ts BETWEEN …` window; -- `COUNT(DISTINCT col)`, normalised to `CARDINALITY` and served by HLL; -- top-k as `ORDER BY SUM|COUNT(..) DESC LIMIT k`, served by - `CountMinSketchWithHeap`; -- nested subqueries (spatial-of-temporal). - -There is no `rate`/`increase` and no binary operator between aggregates. +Fix a dataset and a query workload, and let the planner choose its +summary-based plan. Deploy that **same** plan two ways: -Example of the shape the engine matches (from the deleted -`clickhouse_quantile_queries.sql`): +1. **ASAPQuery**: the precompute engine and query engine (`query_engine_rust`). +2. **ClickHouse**: materialized views (MVs) over the raw table, with exact + states or ClickHouse's own sketch states. -```sql -SELECT quantile(0.95)(v1) FROM h2o_groupby -WHERE timestamp BETWEEN '2024-01-01 00:00:00' AND '2024-01-01 00:00:10' -GROUP BY id1, id2; -``` +How much cheaper and faster is (1) than (2), at the same accuracy target? -### 2.3 Planner: sketch-bench `rqe-optimizer` - -The plan comes from **sketch-bench `rqe-optimizer`**, the MILP used in the -AutoSketch plan (checked at sketch-bench `964ccb5`). The `sketchdb` mode -currently runs `asap-planner`'s hardcoded SQL generator instead, so the runner -needs to bypass it (§5). - -What `rqe-optimizer` provides today: -- **A library, not a CLI.** RQEs and label sets are built in Rust - (`examples/small_problem.rs`), and `milp::minimize_cost` solves them. -- **Input:** - - `Rqe {capability, lookback S, interval T, labels, accuracy metric/tolerance}`; - - `LabelSetInfo {cardinality, arrival_rate}`; - - the measured `AtomicCostTable` from `scripts/export_rqe_optimizer_costs.sh`. -- **Output:** a `Mapping` from each RQE to a - `Deployment {capability, labels, config: AtomicCostEntry, window x, slide y}`. - `config.sketch` is a sketch-bench variant, and `config.sketch_config` holds - its parameters. It is not written to a file. -- **Capabilities:** `Freq`, `Quantile`, `Cardinality` and `TopK`. Exact - Sum/MinMax are not capabilities. - -Mapping its variants to ASAPQuery `aggregationType`s: - -| `rqe-optimizer` variant | ASAPQuery `aggregationType` | Status | -|---|---|---| -| `cms-fastpath-vector2d` `{rows, cols}` | `CountMinSketch` `{depth, width}` | ok | -| `cms-heap-topk-fastpath-vector2d` `{rows, cols}` | `CountMinSketchWithHeap` `{depth, width, heap}` | ok | -| `kll-percall` `{k}` | `DatasketchesKLL` `{K}` (per group via `labels.grouping`) | ok | -| `hll` `{precision}` | `HLL` | ok | -| `countsketch-fastpath-vector2d`, `dd`, `univmon-cardinality` | none | **Exclude from candidates.** ASAPQuery can't deploy them | +The first pass covers **quantile queries only**, with **DDSketch on both +sides**: the same algorithm and the same relative-accuracy parameter α. Only +the data plane differs between them. The frequency and top-k templates are +deferred (§10); for those, ClickHouse has no equivalent sketch. -The optimizer's costs are measured on sketch-bench's implementations, not -ASAPQuery's. The plan's *estimated* cost is reported next to the *measured* one, -and the gap between them is a result in its own right. +## 2. Decisions -### 2.4 Gaps - -| Gap | Impact | +| # | Decision | |---|---| -| `config/experiment_type/clickhouse.yaml` is used in the runner docstring but was never committed | Can't run from a clean checkout | -| `execution-utilities/benchmark/generate_queries.py` was deleted in #404 | No SQL workload generator | -| The MV init SQL examples (`netflow_init.sql`, `quantile_demo/init.sql`) are referenced in comments but absent | No template for the MV arms | -| No Zipf/Pareto **JSONL** generator. The fake exporter's Zipf is Prometheus-only, with a fixed α=1.01 | Can't sweep skew or cardinality | -| Data loads once per run, then the modes loop | MV maintenance would be charged to *every* ClickHouse mode, and `baseline` inserts would slow down. Run each arm separately (§4) | -| `sketchdb` requires the ClickHouse raw load for fallback | Report ASAP's cost without it, and require 0 fallbacks | -| No way to feed an externally chosen plan to `sketchdb`: it always runs `asap-planner` | Needed to deploy the `rqe-optimizer` plan | -| `rqe-optimizer` has no workload-file input or plan-file output | Needed to drive it from the sweep | -| `post_experiment/single_experiment/compare_costs.py` keys on the Prometheus process | Needs ClickHouse and precompute paths | -| `cloudlab_setup/single_node/constants.sh` assumes `/scratch` | These nodes use `/mydata` (§9) | +| Data plane | ASAPQuery `query_engine_rust`, via `experiment_run_clickhouse.py`. Not ASAPQuery-backend | +| Plan source | The sketch-bench MILP via `asap-planner --planner milp` ([#753](https://github.com/ProjectASAP/ASAPQuery/issues/753), Milind). The planner's `streaming_config.yaml` **is** the plan. No separate plan file | +| Optimizer objective | `minimize_cost` for the `general_purpose` EC2 family, until model A is in code. Every arm is measured and priced under model A | +| Quantile sketch | DDSketch on both sides: ClickHouse `quantilesDD(α, …)`; ASAPQuery `DDSketch` (to be added, [#787](https://github.com/ProjectASAP/ASAPQuery/issues/787), Zeying). The optimizer's quantile candidates are limited to `dd` | +| KLL | Appendix only: the optimizer may choose KLL, and ClickHouse uses its closest substitute | +| ClickHouse bars | Exact MV, and MV + sketch state. Plain ClickHouse is the accuracy reference (in a table, not a bar) | +| Key tracker (`DeltaSetAggregator`) | Not needed for quantile plans. For the later frequency/top-k pass: if it is still required, it is deployed, measured and reported as ASAP's cost. Milind is checking whether it can be removed, and a cost-table row is being added separately | +| Ingest | Paced at the workload's arrival rate, with queries running during ingest | +| Rate | **1 sample/s per series**, so the total rate is `C·s` samples/s | +| Fixed data parameters | θ = 1.1, Pareto a = 1.5, accuracy target 95%. Not swept | +| Templates | Quantile only (§3.2). Per group in the main figure; per series in the appendix | +| Run length | 15 min of event time per run (largest W = 10 min), plus one 1 h run of the default workload | +| Resources | 16 cores / 64 GiB cap per system under test. The feeder and fallback ClickHouse get separate cores | +| Pilot | Run now with ASAPQuery on KLL from today's `asap-planner`, to debug the plumbing; not reported. Rerun once DD and `--planner milp` land | ## 3. Workload ### 3.1 Data -The data model is the one in `autosketch-vs-planner.md` §6, written as JSONL -for both systems. ASAPQuery's JSON ingest and ClickHouse's loader read the -**same file**. +- One metric with labels `label_0` (`C` groups) and `instance` (`s` series per + group), so there are `C·s` time series. +- Every series emits **one sample per second** of event time. +- Values are drawn from **Pareto a = 1.5**. +- θ = 1.1 (Zipf key weights) is fixed for the later frequency/top-k pass. It + has no effect on the quantile-only pass, because every series emits at the + same rate. +- Generation is seeded. The ClickHouse and ASAPQuery feeders read the same + JSONL file `{ts, label_0, instance, value}`. -- Rows are `{ts, label_0, instance, value}`. -- `label_0` takes `C` values (the groups). `instance` takes `s` values per - group. -- Each series emits 100 samples/s of event time. -- Frequency and top-k: the weight of each key follows Zipf θ. -- Quantiles: values follow Pareto a. -- Generation is seeded, so every arm and trial reads identical bytes. +### 3.2 Queries -Event time is replayed, not real time: generate a fixed span (default 2 h of -event time) and let each system ingest as fast as it can. Ingest throughput is -reported separately. +Each query repeats every `T = 1 min` (a refreshing dashboard panel) over the +last `W`, with q ∈ {0.5, 0.75, 0.9, 0.95, 0.99}: -### 3.2 Query templates (SQL) +```sql +-- per group (main figure) +SELECT label_0, quantiles(0.5, 0.75, 0.9, 0.95, 0.99)(value) FROM data +WHERE ts BETWEEN now() - W AND now() GROUP BY label_0; -These are the AutoSketch plan's templates that ASAPQuery's SQL path can serve -(§2.2), plus count-distinct, which SQL adds. `W` is the window, and queries -repeat every `T` over the span. +-- per series (appendix) +... GROUP BY label_0, instance; +``` -| # | SQL (per window) | `rqe-optimizer` RQE (capability, labels) | Mirrors AutoSketch template | -|---|---|---|---| -| 1 | `SUM(value) … GROUP BY label_0` | Freq, `{label_0}` → CMS | 1, 8 | -| 2 | `label_0, SUM(value) … GROUP BY label_0 ORDER BY 2 DESC LIMIT 3` | TopK, `{label_0}` → CMS with heap | 2 | -| 3 | `quantile(q)(value) … GROUP BY label_0`, q ∈ {0.5, 0.75, 0.9, 0.95, 0.99} | Quantile, `{label_0}` → KLL per group | 3 | -| 4 | `SUM(value) … GROUP BY label_0, instance` | Freq, `{label_0, instance}` → CMS | 4 | -| 5 | `quantile(q)(value) … GROUP BY label_0, instance`, same q | Quantile, `{label_0, instance}` → KLL per series | 5 | -| 6 | `COUNT(DISTINCT instance) … GROUP BY label_0` | Cardinality, `{label_0}` → HLL | none (SQL-only) | +The ASAPQuery SQL path matches these shapes (`QUANTILE` with `GROUP BY` labels +over a `WHERE ts BETWEEN …` window). The five quantiles read one stream, so a +single deployment serves all of them. -Each template is one RQE per window in `W`, repeating every `T`. Quantiles of -one template share a stream, so the optimizer can serve all five from one -deployment. +Values per sketch per window are `s·W` per group and `W` per series. With the +defaults (s = 100, W = 1 min), that is 6,000 per group and 60 per series. This +is why per series is in the appendix: sketches that small mostly measure +per-sketch overhead, not summarization. -Dropped, because ASAPQuery SQL cannot express them: AutoSketch templates 6, 7 -and 9 (`rate`), and 10 (a ratio of two quantiles). The earlier `MAX` control is -also dropped: `rqe-optimizer` has no MinMax capability, so it would not come -from the plan. +### 3.3 Workloads -### 3.3 Grid +One dimension is varied at a time, around the default: -One dimension at a time around the bold defaults: +| # | Varied | Value | C | s | W | Series | Rate (samples/s) | +|---|---|---|---|---|---|---|---| +| 1 | default | — | 1e3 | 100 | 1 m | 1e5 | 1e5 | +| 2 | C | 1e2 | 1e2 | 100 | 1 m | 1e4 | 1e4 | +| 3 | C | 1e4 | 1e4 | 100 | 1 m | 1e6 | 1e6 | +| 4 | C | 1e5 | 1e5 | **10** | 1 m | 1e6 | 1e6 | +| 5 | s | 1 | 1e3 | 1 | 1 m | 1e3 | 1e3 | +| 6 | s | 10 | 1e3 | 10 | 1 m | 1e4 | 1e4 | +| 7 | W | 10 s | 1e3 | 100 | 10 s | 1e5 | 1e5 | +| 8 | W | 10 m | 1e3 | 100 | 10 m | 1e5 | 1e5 | -| Dimension | Values | -|---|---| -| Query mix | **all 6**, frequency only {1, 4}, quantile only {3, 5}, top-k only {2} | -| `C` | 1e2, **1e3**, 1e4, 1e5 | -| `s` | 1, 10, **100** | -| θ / a | θ ∈ {0, 0.5, **1.0**, 1.5}; a ∈ {1.1, **2**, 3} | -| Window `W` / repeat `T` | {10 s}, **{1 m}**, {10 m}; `T = W` (tumbling) | -| Accuracy target | 90%, **95%**, 99% | +Workload 4 uses s = 10, which caps every workload at 1e6 series and 1e6 +samples/s. ## 4. Arms -All arms see the same rows and the same plan set. **Each arm is a separate -runner invocation** (`experiment=[]`), with its own `init_sql_file`, -so ingest cost isn't cross-charged. +Each arm is a **separate run**, with its own data feed and DDL, so no arm pays +for another's MV maintenance. -| Arm (runner mode) | Init SQL | Role | +| Arm | Deployed | Role | |---|---|---| -| `baseline` | raw `MergeTree` only | Exact reference and accuracy ground truth | -| `baseline_mv` | raw + MVs with exact states (`sumState`, `countState`, `maxState`, `quantilesExactState`, `uniqExactState`, per-key `sumState` for top-k) | "MVs without summaries" | -| `baseline_mv_sketch` | raw + MVs with the closest summary state (table below) | **Main ClickHouse point** | -| `sketchdb` | raw only (for fallback) | **ASAPQuery**. Must report 0 fallbacks | - -Each MV is keyed by `(toStartOfInterval(ts, W), group-by labels)`. The -`baseline_mv*` SQL files query the MV target tables with the `-Merge` -combinators. Every mode's `sql_file` entry in `query_groups[i].sql_file` -covers the same logical query. +| Reference | ClickHouse raw `MergeTree`, no MVs | Accuracy ground truth (table) | +| Exact MV | MV keyed by `(toStartOfInterval(ts, x), label_0)`, storing `quantilesExactState(...)(value)` | Bar 1: what MVs alone buy | +| Sketch MV | Same MV layout, storing `quantilesDDState(α, ...)(value)` with the plan's α | Bar 2: ClickHouse's best summary option | +| ASAPQuery | `streaming_config.yaml` from `asap-planner --planner milp`, with `DDSketch(α)` | Bar 3 | + +- **MV layout.** The MV's interval `x` and slide come from the plan's window + and slide. ClickHouse queries merge the panes inside `W` with + `quantilesExactMerge` / `quantilesDDMerge`. +- **ASAPQuery's fallback ClickHouse is empty.** Any query that falls back fails + loudly instead of being answered by ClickHouse, so "0 fallbacks" is checked + by construction. That ClickHouse runs on its own 2 cores / 4 GiB and is not + counted in ASAP's cost. +- **ClickHouse version.** The image is pinned to ≥ 24.1, for `quantileDD`. The + pilot checks that `quantilesDD` works with `-State`/`-Merge` in an + `AggregatingMergeTree` MV. + +## 5. Pipeline per workload -Plan → ClickHouse mapping for `baseline_mv_sketch`: +``` +generator ──► data.jsonl + SQL files + planner input + │ +asap-planner --planner milp ──► streaming_config.yaml + inference_config.yaml (the plan) + │ +translator ──► ClickHouse init.sql (raw + MVs) and per-arm SQL + │ +runner: one run per arm × trial, paced feeders, resource caps, monitors +``` -| Planned summary | ClickHouse state | Gap reported | -|---|---|---| -| KLL (`DatasketchesKLL`) | `quantilesState(levels)(v)` (reservoir 8192) or `quantilesTDigestState` | No KLL. Size/accuracy cannot be set | -| CountMin (`CountMinSketch`) | none: exact `sumState` per key | Not implemented. State is one row per key, not `d×w` | -| CountMin with heap | `topKWeightedState(k)` (Filtered Space-Saving) | No CMS. The load factor cannot be set | -| HLL | `uniqHLL12State` or `uniqCombinedState(p)` | `uniqHLL12` is fixed at 12 bits. `uniqCombined` lets you set p | - -The translator emits every row of this table. A "none" row becomes an exact -state, and it is counted and listed under the figure. - -## 5. Plan source and deployment - -The pipeline for one (dataset, workload) point: - -1. **Workload → RQEs.** From the template set, `W`, `T`, `C`, `s` and the - accuracy target, emit `workload.json`: - - one RQE per (template, window); - - one label set per grouping, with `cardinality` (`C`, or `C·s`) and - `arrival_rate` (`100·C·s`/s at the replay rate). - - Accuracy metrics follow `small_problem.rs`: - - `are_top100` for Freq; - - `mean_rank_err` for Quantile; - - `relative_error` for Cardinality; - - `precision_at_k` (a floor) for TopK. -2. **Solve.** A new `rqe-optimizer` binary (in sketch-bench) reads - `workload.json` and the `AtomicCostTable`, with the candidate families - restricted to the ASAPQuery-deployable variants (§2.3). It runs - the objective chosen in D1a and writes `plan.json`: - - per deployment: capability, labels, sketch variant + `sketch_config`, - `x`, `y`, and the RQEs it serves; - - the MILP's estimated CPU, memory and $/h. -3. **Translate** (new, in this repo). `plan.json` is the single source for - every arm: - - ASAPQuery `streaming_config.yaml`: one aggregation per deployment, with - `aggregationType`/`parameters` from §2.3, `labels.grouping`, - `windowSizeMs = x` and `windowType` tumbling if `x = y`, else sliding with - slide `y`; - - `inference_config.yaml`: each RQE's SQL → its aggregation ID; - - the ClickHouse `init.sql` for `baseline_mv` and `baseline_mv_sketch` - (§4), plus the per-mode SQL files. - - The two systems therefore cannot drift apart. -4. **Deploy.** Add a runner option (e.g. `controller.plan_dir=…`) that skips - `asap-planner` in `sketchdb` mode and rsyncs the translated configs into - `controller_output/`. Everything after that stays as it is. - -**Check before building:** whether `query_engine_rust` accepts sliding windows -(`x ≠ y`) for every summary type from SQL. If not, constrain the optimizer to -`y = x` and note it. - -`sketchdb` ingests by itself from the JSONL. Its cost excludes the ClickHouse -raw load, which exists only for fallback. Every RQE counted as served must show -0 forwards to ClickHouse, taken from the query-engine logs. +- **Translator.** It reads ASAPQuery's `streaming_config.yaml`, which gives + each aggregation's type, `parameters`, `labels.grouping`, `windowSizeMs` and + `windowType`/slide. From it, the translator writes the MV DDL and the queries + that read the MVs. Both systems therefore deploy one plan. +- **Paced feeders.** There are two identical feeder processes, one per system + under test, started on the same clock. Each one sends every second's rows: + ASAPQuery receives them over Prometheus Remote Write, and ClickHouse via + `INSERT … FORMAT JSONEachRow`. Each feeder runs on its own 2 cores, outside + both caps. The existing bulk paths (JSON ingest, `_load_json_batched`) do not + pace. `batch_delay_ms` in `json_ingest.rs` is a fixed sleep and is not + exposed in `engine_config.rs`. +- **Runner.** It extends `experiment_run_clickhouse.py`: + - one mode per arm; + - a feeder-driven ingest phase in place of the bulk load; + - CPU/memory caps (`docker --cpuset-cpus/--memory` for ClickHouse, + `taskset` + a memory cgroup for `query_engine_rust`); + - collection of MV bytes from `system.parts`. ## 6. Metrics -| Metric | ClickHouse arms | `sketchdb` | +| Metric | ClickHouse arms | ASAPQuery | |---|---|---| -| Ingest CPU (core-s), including MV/summary maintenance | `monitor_output_ingest.json` (`clickhouse` process) | `pc-worker` threads in `monitor_output.json` | -| Retained state bytes | `system.parts.bytes_on_disk` + `primary_key_bytes_in_memory` for the MV target tables (raw table reported separately) | query-engine store size (to add if not already exported) | -| RSS (steady and peak) | process | process | -| Query latency p50/p95/p99 | prometheus-client per query (`analyze_latencies.py`) | same | -| Query CPU | monitor during the query phase | same | -| Accuracy | vs `baseline` (`calculate_fidelity.py --exact_experiment_mode baseline`): rank error for quantiles (as in #784), relative error for sums, recall@k for top-k, relative error for distinct counts | same | - -**Cost.** Use the AutoSketch plan's **model A** -(`$/h = a·avg vCPU + b·GiB`, with `a ≈ 0.0368` and `b ≈ 0.00364` from the -2026-10-04 EC2 fit) over measured ingest + query CPU and retained state memory. -This keeps the planner and data-plane sections in the same units. Report raw -CPU-seconds and bytes as well. - -**Fairness** -- Same node type: `c6320` (2× E5-2683 v3, 56 threads, 251 GB). -- ClickHouse and `query_engine_rust` get the same cores and memory cap - (`docker --cpuset-cpus/--memory` and `taskset`/cgroup). -- ClickHouse image pinned by tag or digest (`dataset.clickhouse_image_tag`), - not `latest`. -- Query cache off (`use_query_cache=0`). The first query pass is a discarded - warm-up. +| Ingest CPU (core-s), including MV/summary maintenance | `clickhouse` process | `pc-worker` threads | +| Retained state | `system.parts.bytes_on_disk` + `primary_key_bytes_in_memory` for MV tables (raw table reported separately) | Store size | +| RSS (steady state and peak) | process | process | +| Query latency p50/p95/p99 | per query, client-side | same | +| Query CPU | process CPU over the query phase | same | +| Accuracy | Rank error vs the reference, per quantile (as in #784). Target: 95% | same | + +**Cost** uses model A from the AutoSketch plan: `$/h = a·avg vCPU + b·GiB`, +with `a ≈ 0.0368` and `b ≈ 0.00364` (2026-10-04 EC2 fit), over measured ingest + +query CPU and retained-state memory. Raw CPU-seconds and bytes are reported +too, as is the planner's estimated cost next to the measured one. + +**Fairness:** +- Same node type: `c6320`, 2× E5-2683 v3, 56 threads, 251 GB. +- Same cap for each system under test. +- Query cache off. The first query pass is discarded as warm-up. - 3 trials per point. Report the median, with min/max error bars. -## 7. Open decisions - -- **D0: data plane.** Decided: ASAPQuery `query_engine_rust` - (`experiment_run_clickhouse.py` `sketchdb`). -- **D1: plan source.** Decided: sketch-bench `rqe-optimizer` (§2.3, §5). -- **D1a: the optimizer objective.** At sketch-bench `964ccb5`, `rqe-optimizer` - offers `minimize_tco` (minimum steady-state CPU) and `minimize_cost(family)` - (fractional EC2 instances of one family, from #137). The model A and model B - objectives in the AutoSketch plan are not in the code yet. Proposal: use - `minimize_cost` with `general_purpose` now, and switch to model A when it - lands. Every arm is still measured and priced under model A. -- **D2:** keep `baseline_mv` as an extra bar, or show only `baseline_mv_sketch`. -- **D3:** the ClickHouse quantile state, `quantilesState` or - `quantilesTDigestState`. Report the one closest to ASAP in accuracy, and put - the other in the appendix. -- **D4:** whether to add `rate` or a quantile ratio to the ASAPQuery SQL path, - to cover all 10 AutoSketch templates. - -## 8. Work items (each a small PR) - -1. Data generator: seeded JSONL with Zipf θ weights and Pareto a values over - `(C, s)`, plus a test on the empirical rank-frequency slope. -2. Workload generator for templates 1–6 over the grid. It emits both the SQL - files (replacing the deleted `generate_queries.py`) and `workload.json` for - `rqe-optimizer`, and includes a differential test against ClickHouse on a - small `C, s`. -3. **sketch-bench PR:** an `rqe-optimizer` binary with - `workload.json` + cost table → `plan.json`, plus a flag restricting the - candidate families. -4. Plan translator: `plan.json` in; the ASAPQuery configs and the ClickHouse - `init.sql` + per-mode SQL out. It logs every unsupported mapping. -5. Runner: - - commit `config/experiment_type/clickhouse.yaml`; - - add `controller.plan_dir` to skip `asap-planner` and deploy the - translated configs; - - collect MV bytes from `system.parts` after the load. -6. Post-processing: ClickHouse and precompute paths in `compare_costs.py`, - model-A pricing, and the figure script. -7. Sweep driver that fans out (workload, arm, trial) across node1–3. - -## 9. Cluster and run procedure +## 7. Run count and time + +| Set | Runs | Time each | Wall time on 3 nodes | +|---|---|---|---| +| Main: 8 workloads × 4 arms × 3 trials, 15 min span | 96 | ~25 min | ~13 h | +| 1 h span, default workload, 4 arms × 3 trials | 12 | ~70 min | ~5 h | +| Appendix: per-series queries, default workload, 4 arms × 3 trials | 12 | ~25 min | ~2 h | +| Appendix: KLL plan, default workload, 4 arms × 3 trials | 12 | ~25 min | ~2 h | +| **Total** | **132** | | **~22 h** | + +The pilot (the default workload, 4 arms, 1 trial) comes before all of these. +It checks: +- 0 fallbacks; +- exact-MV results equal the reference exactly; +- DD accuracy is within target on both sides; +- the feeders hold 1e6 samples/s under the cap (workloads 3 and 4). + +## 8. Work items + +| # | Item | Owner | Depends on | +|---|---|---|---| +| 1 | Data + workload generator: JSONL (Pareto a, `C`, `s`, 1 sample/s), SQL files, planner input | Zeying | — | +| 2 | Paced feeder (Remote Write and ClickHouse INSERT, one per system) | Zeying | — | +| 3 | Translator: `streaming_config.yaml` → ClickHouse `init.sql` + per-arm SQL | Zeying | — | +| 4 | Runner: per-arm modes, feeder ingest, caps, `system.parts` collection, `config/experiment_type/clickhouse.yaml` | Zeying | 2 | +| 5 | Post-processing: ClickHouse and precompute paths in `compare_costs.py`, model-A pricing, figure | Zeying | 4 | +| 6 | DDSketch in ASAPQuery (the DD slice of #762 + #787) | Zeying | #762 | +| 7 | `asap-planner --planner milp` with quantiles limited to `dd` | Milind | #753 | +| 8 | Key-tracker removal check, and its cost-table row (for the later frequency/top-k pass) | Milind | — | + +Items 1–5 can start now, and the pilot runs on today's `asap-planner` (KLL). +The main runs need items 6 and 7. + +## 9. Cluster CloudLab experiment `scratch2.cloudmigration-pg0.clemson.cloudlab.us`. -`node1`–`node3` run 3 configs in parallel (`node_offset` 1/2/3, -`num_nodes=1`). `node1` is also the orchestrator. - -Setup: -1. `/mydata` (845 GB ext4) is already mounted, world-writable, on node1–3. The - setup scripts expect `/scratch`, so symlink it or make the root volume - configurable. -2. `deploy_from_scratch.sh` without its storage step: Docker, the Rust - toolchain, and `cargo build --release` for `asap-query-engine` and - `asap-planner-rs`, plus sketch-bench (`rqe-optimizer`, and the cost table - from `scripts/export_rqe_optimizer_costs.sh`). -3. The orchestrator needs SSH to `node{1,2,3}.`, including itself. -4. Generate datasets once into `/mydata/datasets/`. The runner rsyncs them. - -Order: -1. **Pilot**: the default workload with `C=1e3`, `s=10`, all 4 arms, 1 trial. - Check: - - 0 fallbacks in `sketchdb`; - - exact MV results equal `baseline`; - - accuracy is within target. -2. **Sweep**: the §3.3 dimensions one at a time - (3+3+2+5+2+2 = 17 non-default points + default = 18 workloads), - × 4 arms × 3 trials = **216 runs**. Allowing about 15 min per run, that is - about 18 h on 3 nodes. -3. **Figure**: grouped bars of model-A $/h and p95 latency per workload, with - `sketchdb` ÷ `baseline_mv_sketch` annotated (the "up to X×" numbers). Each - bar carries its accuracy, and the plans ClickHouse can't express are listed - under the figure. +`node1`–`node3` run in parallel (`node_offset` 1/2/3, `num_nodes=1`), and +`node1` is also the orchestrator. + +- `/mydata` (845 GB ext4) is mounted and world-writable on all three nodes. + The setup scripts expect `/scratch`, so add a symlink or make the root volume + configurable. +- Install Docker and the Rust toolchain, and build `asap-query-engine` and + `asap-planner-rs` in release mode (`deploy_from_scratch.sh`, minus its + storage step). +- The orchestrator needs SSH to `node{1,2,3}.`, including itself. + +## 10. Later passes (out of scope here) + +- **Frequency and top-k templates.** Matching sketches where ClickHouse has + them (HLL ↔ `uniqCombined(p)`). Where it has none (Count-Min, CMS-heap vs + `topKWeighted`), the bars are labeled "no ClickHouse equivalent", and a top-k + sketch MV that misses precision@k is shown with that marked. +- **`rate` and quantile-ratio templates.** These need ASAPQuery SQL support. +- **The connection between ASAPQuery's planner and the ILP** belongs to #753, + not to this evaluation. From 38f6f6838e99a8238dc77953843a161c447a847b Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 5 Oct 2026 19:04:29 +0000 Subject: [PATCH 5/6] docs(asap-tools): price each eval run under both cost models Co-Authored-By: Claude Opus 5.5 --- .../DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md | 30 ++++++++++++++----- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md index 29f08822..c0c12e41 100644 --- a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md +++ b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md @@ -27,8 +27,9 @@ deferred (§10); for those, ClickHouse has no equivalent sketch. |---|---| | Data plane | ASAPQuery `query_engine_rust`, via `experiment_run_clickhouse.py`. Not ASAPQuery-backend | | Plan source | The sketch-bench MILP via `asap-planner --planner milp` ([#753](https://github.com/ProjectASAP/ASAPQuery/issues/753), Milind). The planner's `streaming_config.yaml` **is** the plan. No separate plan file | -| Optimizer objective | `minimize_cost` for the `general_purpose` EC2 family, until model A is in code. Every arm is measured and priced under model A | -| Quantile sketch | DDSketch on both sides: ClickHouse `quantilesDD(α, …)`; ASAPQuery `DDSketch` (to be added, [#787](https://github.com/ProjectASAP/ASAPQuery/issues/787), Zeying). The optimizer's quantile candidates are limited to `dd` | +| Optimizer objective | `minimize_cost` for the `general_purpose` EC2 family, until model A is in code | +| Cost reporting | One run per arm, priced two ways from the same CPU/memory timeline: usage-based (model A) and peak-provisioned EC2 (model B) (§6) | +| Quantile sketch | DDSketch on both sides: ClickHouse `quantilesDD(α, …)`; ASAPQuery `DDSketch` ([#792](https://github.com/ProjectASAP/ASAPQuery/pull/792), part of [#787](https://github.com/ProjectASAP/ASAPQuery/issues/787)). The optimizer's quantile candidates are limited to `dd` | | KLL | Appendix only: the optimizer may choose KLL, and ClickHouse uses its closest substitute | | ClickHouse bars | Exact MV, and MV + sketch state. Plain ClickHouse is the accuracy reference (in a table, not a bar) | | Key tracker (`DeltaSetAggregator`) | Not needed for quantile plans. For the later frequency/top-k pass: if it is still required, it is deployed, measured and reported as ASAP's cost. Milind is checking whether it can be removed, and a cost-table row is being added separately | @@ -159,10 +160,23 @@ runner: one run per arm × trial, paced feeders, resource caps, monitors | Query CPU | process CPU over the query phase | same | | Accuracy | Rank error vs the reference, per quantile (as in #784). Target: 95% | same | -**Cost** uses model A from the AutoSketch plan: `$/h = a·avg vCPU + b·GiB`, -with `a ≈ 0.0368` and `b ≈ 0.00364` (2026-10-04 EC2 fit), over measured ingest + -query CPU and retained-state memory. Raw CPU-seconds and bytes are reported -too, as is the planner's estimated cost next to the measured one. +**Cost.** Each arm runs once. The monitor samples the system's CPU and +memory every second, and that one timeline is priced two ways, using the cost +models of the AutoSketch plan: + +- **Usage-based (model A).** Total CPU = the area under the CPU curve + (CPU-seconds), divided by the run length to give average vCPU. Then + `$/h = a·avg vCPU + b·GiB`, with `a ≈ 0.0368` and `b ≈ 0.00364` (2026-10-04 + EC2 fit) and the average retained-state memory. +- **Peak-provisioned (model B).** Take the peak of the CPU curve and peak + memory, and buy enough instances of one EC2 family to cover both: + `n_f = max(peak vCPU / vCPU_f, peak GiB / GiB_f)`, `$/h = n_f · price_f`. + This is reported for c7i, m7i and r7i. + +Model A shows the work each system does. Model B shows what it costs to +provision for the bursts when queries fire on top of ingest. Raw CPU-seconds, +peaks and bytes are reported too, as is the planner's estimated cost next to +the measured one. **Fairness:** - Same node type: `c6320`, 2× E5-2683 v3, 56 threads, 251 GB. @@ -195,8 +209,8 @@ It checks: | 2 | Paced feeder (Remote Write and ClickHouse INSERT, one per system) | Zeying | — | | 3 | Translator: `streaming_config.yaml` → ClickHouse `init.sql` + per-arm SQL | Zeying | — | | 4 | Runner: per-arm modes, feeder ingest, caps, `system.parts` collection, `config/experiment_type/clickhouse.yaml` | Zeying | 2 | -| 5 | Post-processing: ClickHouse and precompute paths in `compare_costs.py`, model-A pricing, figure | Zeying | 4 | -| 6 | DDSketch in ASAPQuery (the DD slice of #762 + #787) | Zeying | #762 | +| 5 | Post-processing: ClickHouse and precompute paths in `compare_costs.py`, model A and model B pricing from the monitor timeline, figure | Zeying | 4 | +| 6 | DDSketch in ASAPQuery (the DD slice of #762 + #787) | Zeying | Draft PR #792 | | 7 | `asap-planner --planner milp` with quantiles limited to `dd` | Milind | #753 | | 8 | Key-tracker removal check, and its cost-table row (for the later frequency/top-k pass) | Milind | — | From bcbe3687270d2bd43c57cfd2fda7bc9a4a9319c6 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 5 Oct 2026 19:15:21 +0000 Subject: [PATCH 6/6] docs(asap-tools): generate eval data on the fly in the paced feeder Co-Authored-By: Claude Opus 5.5 --- .../DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md | 37 ++++++++++++------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md index c0c12e41..d3ef4ef2 100644 --- a/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md +++ b/asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md @@ -52,8 +52,11 @@ deferred (§10); for those, ClickHouse has no equivalent sketch. - θ = 1.1 (Zipf key weights) is fixed for the later frequency/top-k pass. It has no effect on the quantile-only pass, because every series emits at the same rate. -- Generation is seeded. The ClickHouse and ASAPQuery feeders read the same - JSONL file `{ts, label_0, instance, value}`. +- The data is generated on the fly by the paced feeder + ([#793](https://github.com/ProjectASAP/ASAPQuery/pull/793)), with no dataset + file. Each value is a hash of `(seed, series, sample index)`, so the ClickHouse + and ASAPQuery feeders send identical data. A file would be about 9e8 rows + (~50 GB) per workload at 1e6 series × 900 s. ### 3.2 Queries @@ -122,7 +125,7 @@ for another's MV maintenance. ## 5. Pipeline per workload ``` -generator ──► data.jsonl + SQL files + planner input +workload generator ──► SQL files + planner input + feeder flags (C, s, seed) │ asap-planner --planner milp ──► streaming_config.yaml + inference_config.yaml (the plan) │ @@ -135,13 +138,20 @@ runner: one run per arm × trial, paced feeders, resource caps, monitors each aggregation's type, `parameters`, `labels.grouping`, `windowSizeMs` and `windowType`/slide. From it, the translator writes the MV DDL and the queries that read the MVs. Both systems therefore deploy one plan. -- **Paced feeders.** There are two identical feeder processes, one per system - under test, started on the same clock. Each one sends every second's rows: - ASAPQuery receives them over Prometheus Remote Write, and ClickHouse via - `INSERT … FORMAT JSONEachRow`. Each feeder runs on its own 2 cores, outside - both caps. The existing bulk paths (JSON ingest, `_load_json_batched`) do not - pace. `batch_delay_ms` in `json_ingest.rs` is a fixed sleep and is not - exposed in `engine_config.rs`. +- **Paced feeders** (`asap-tools/data-sources/paced-feeder`, + [#793](https://github.com/ProjectASAP/ASAPQuery/pull/793)). There is one + feeder process per system under test, both run with the same workload flags, + and each runs on its own 2 cores outside the caps. Every second it sends that + second's rows: + - ASAPQuery gets them over Prometheus Remote Write (50k rows per request, + under axum's 2 MB body limit); + - ClickHouse gets them via `INSERT … FORMAT RowBinary` (100k rows per + request), its efficient native path, just as Remote Write is ASAPQuery's. + JSONEachRow would have charged ClickHouse for JSON parsing. + + Measured pinned to 2 cores, both sinks hold 1e6 rows/s with no tick over + 1 s; the feeder uses 0.15 cores (ClickHouse) and 0.34 cores (Remote Write). + The existing bulk paths (JSON ingest, `_load_json_batched`) do not pace. - **Runner.** It extends `experiment_run_clickhouse.py`: - one mode per arm; - a feeder-driven ingest phase in place of the bulk load; @@ -199,14 +209,15 @@ It checks: - 0 fallbacks; - exact-MV results equal the reference exactly; - DD accuracy is within target on both sides; -- the feeders hold 1e6 samples/s under the cap (workloads 3 and 4). +- the systems absorb 1e6 samples/s under their caps (workloads 3 and 4). The + feeders themselves are already shown to keep up at that rate. ## 8. Work items | # | Item | Owner | Depends on | |---|---|---|---| -| 1 | Data + workload generator: JSONL (Pareto a, `C`, `s`, 1 sample/s), SQL files, planner input | Zeying | — | -| 2 | Paced feeder (Remote Write and ClickHouse INSERT, one per system) | Zeying | — | +| 1 | Workload generator: SQL files, planner input and feeder flags per workload | Zeying | — | +| 2 | Paced feeder with on-the-fly data generation (Remote Write and ClickHouse RowBinary) | Zeying | Draft PR #793 | | 3 | Translator: `streaming_config.yaml` → ClickHouse `init.sql` + per-arm SQL | Zeying | — | | 4 | Runner: per-arm modes, feeder ingest, caps, `system.parts` collection, `config/experiment_type/clickhouse.yaml` | Zeying | 2 | | 5 | Post-processing: ClickHouse and precompute paths in `compare_costs.py`, model A and model B pricing from the monitor timeline, figure | Zeying | 4 |