From 867b5a9696d0e4781df3eae10146a08c7d500ac2 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 13 Oct 2026 17:52:04 +0000 Subject: [PATCH 01/12] =?UTF-8?q?docs(eval):=20plan=20the=20end-to-end=20e?= =?UTF-8?q?valuation=20(paper=20=C2=A76.2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae --- docs/evaluation/end-to-end-eval.md | 288 +++++++++++++++++++++++++++++ 1 file changed, 288 insertions(+) create mode 100644 docs/evaluation/end-to-end-eval.md diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md new file mode 100644 index 00000000..c4066411 --- /dev/null +++ b/docs/evaluation/end-to-end-eval.md @@ -0,0 +1,288 @@ +# Evaluation plan: end-to-end cost and latency (paper §6.2) + +Status: draft 2026-10-13. Reuses the §6.3 workloads (#777) at 1/20 of their +data rate, on the plumbing from #840–#845, #841 and #828. The plans re-solved +at 10 samples/s are in §6. Open decisions are in §12. + +## 1. Question + +On the same data and the same repeating queries, how much CPU and memory does +ASAPQuery use, and how fast does it answer, compared with Prometheus, +VictoriaMetrics, ClickHouse, and ClickHouse with materialized views? The result +is a cost–latency scatter, one point per (system, workload), with cost defined +the way the planner defines it (§9). + +## 2. Approach + +- **The set of RQEs is the workload.** The workloads are the §6.3 ones: their + sketch-bench tables already hold the RQEs, the PromQL (classic) or the + structured query fields (multi-grouping), and the data description + (`workloads[0].e2e`, sketch-bench #208, #209, #210). +- **ASAPQuery runs the §6.3 planner's plan**, converted with + `asap-planner --planner saved` (#828). The planner is not re-run inside the + experiment. +- **Every system gets identical data**: one paced feeder per system, same seed + and start time (#840–#843). +- **One system per CloudLab node**, so each system's CPU and memory are measured + alone. +- **At 1/20 of the §6.3 rate** (§5), so that every system can ingest it on one + node and the long-window baseline queries are not hopeless. The plan is + re-solved at that rate (§6). + +## 3. Systems + +| System | Version | Queries | Data path | Runner | +|---|---|---|---|---| +| ASAPQuery | `main` + this plan's PRs | PromQL (classic); SQL frontend (multi-grouping) | remote write from the feeder | `experiment_run_e2e.py` with `controller.prebuilt_output_dir` (#828, #841) | +| Prometheus | 2.53.2 (bare metal) | PromQL | remote-write receiver | `experiment_run_e2e.py` (#841) | +| VictoriaMetrics | vmsingle | PromQL (MetricsQL-compatible subset) | remote write into vmsingle | `experiment_run_e2e.py` (feeder sink: to do, §11) | +| ClickHouse | 26.10 | SQL with Prometheus 2.53 semantics (#844) | `INSERT … RowBinary` from the feeder | `experiment_run_clickhouse.py` (live feed: to do, §11) | +| ClickHouse + MV | 26.10 | SQL over exact materialized views, bucket = T (#845) | same, views created before data | same, `baseline_mv` arm | + +PilotDB (the AQP comparison in the paper) is out of scope here; it needs its +own plan (PromQL→SQL for PilotDB, its sampling modes). + +## 4. Workloads + +| Workload | sketch-bench table | RQEs | Queries | Windows `W` | Repeat `T` | +|---|---|---|---|---|---| +| **mixed** | `synthetic-templatesclassic-shared1` | 42 (50 incl. template 10's shared operands) | 46 PromQL | 15m, 1h, 6h, 24h | 60 s; 1 s for templates 1 and 3 | +| **all** | `synthetic-templatesall-shared1` | 104 (102 served at 10/s, §6) | 46 PromQL + 52 structured (SQL); 50 run | classic as above; multi-grouping 5m, 15m | 60 s; 1 s for templates 1 and 3 | + +- **mixed** is the paper's \WorkloadAggregations: one gauge `data` and one + counter `data_total`, labels `job` (10) → `label_0` (1,000 per job), 1e4 + series. Templates 1–6, 8, 10 (7 and 9 dropped in sketch-bench #210; template + 2's PromQL is `topk(32, sum_over_time(data[w]))`, same RQE). +- **all** adds the multi-grouping templates (11–17) on two records tables, + `http` (region × service × endpoint × status; `user_id`, `latency`) and + `flows` (dst_subnet × dst_port × proto; `src_ip`): distinct counts and a + pooled p99 `GROUP BY CUBE(region, service, endpoint)`. They are SQL only: + PromQL can express neither a distinct count of a value field nor a p99 + pooled over a group's series and a window. +- **Who runs what.** Prometheus and VictoriaMetrics run the classic queries + only, in both workloads; in **all** they have no answer for the + multi-grouping queries, which the results state. ASAPQuery, ClickHouse and + ClickHouse + MV run everything. +- Random workloads (8 RAQEs + 2 binary queries per workload, to answer + cherry-picking) come after this plan works on mixed and all; they would be a + new template set in the sketch-bench export, with the same e2e block. + +## 5. Data and rate + +| | §6.3 | This plan | +|---|---|---| +| Samples/s per series (classic) | 200 (5 ms apart) | **10** (100 ms apart) | +| Classic rows/s (`data` + `data_total`) | 4e6 | 2e5 | +| `http` records/s | 2e6 | 1e5 | +| `flows` records/s | 4e6 | 2e5 | + +- **Why 1/20.** At the §6.3 rate one node cannot ingest it: on clnode109 a + single ClickHouse fell behind at 6e6 rows/s and a single ASAPQuery engine at + 4e6 samples/s. Prometheus at 10 samples/s already took up to 235 s per query + on 25 minutes of data (A6, `/mydata/a6-results/`). +- **Generator.** The paced feeder reads `e2e.metrics`: `child_of` fan-out, + Zipf weights folded into Pareto(2) values for the gauge, a closed-form + counter, Zipf label shares with the parent's condition for records, the + `scaled_by`/anomaly/burst features. `--verify-facts` checks every label + subset's cardinality before a run. +- **Order.** Ticks start on whole seconds and hold samples in `(t, t+1s]`; + each series' samples of a tick go in one request, in timestamp order, and + requests run concurrently (#840). ASAPQuery's allowed lateness stays at its + 5 s default. Prometheus needs no out-of-order window. + +## 6. Planner at 10 samples/s + +The §6.3 plans were solved at 200 samples/s. Ingest CPU scales with the rate, +but each sketch's accuracy and memory depend on its items per window `n`, so +the plan can change at 1/20 of the rate, and the tables and plans are +regenerated at 10 samples/s (`SYNTHETIC_SAMPLES_PER_SEC`). Constraints: + +- **Accuracy curves start at N = 1e3.** Below a curve's first measured N the + optimizer has no accuracy for the pair and the deployment is ineligible + (`rqe-optimizer/src/saturation.rs`, `first_measured_n`). Classic is above it + everywhere (a series' 15m window holds 9,000 items). Multi-grouping + templates that must cover every group (16's finest grouping, 17) can fall + below it in 5m windows. +- **Settings.** ASAP method, cheapest plan (no latency SLA), p95 accuracy as in + §6.3, cpu weights and Fargate weights. + +**Feasibility run** (sketch-bench runner on the #210 tables with +`SYNTHETIC_SAMPLES_PER_SEC = 10`; the 200/s tables reproduce the committed +plans exactly with the same binary): + +- **Served RQEs.** Every classic RQE is served. In **all**, 102 of 104: template + 16's 5m p99 by (region, endpoint) and by (region, service, endpoint) have + 2,500 groups whose smallest covered group holds 403 items, below the curves' + first N (1e3), so no method can serve them (the runner drops them; no theory + fallback). Their 15m windows pass (1,210). The next smallest RQE is at 2,245. + These two queries are left out of the e2e run (§12, Q2). +- **Cost lookups.** The cost table has one row per config, measured at one size + (1e6 items for sketches, 2e5 for exact accumulators). Per-insert, merge and + query CPU and per-instance memory are constants in the cost model + (`analytical_cost_model.rs`), except DDSketch memory, which is capped by the + values a window holds. So nothing is extrapolated at the lower rate; only + accuracy depends on N. +- **Plans.** Classic keeps its 6 deployments, families and parameters, with + longer windows, since ingest is cheaper (CPU weights): quantile_over_time + DDSketch 300/60 s → 900/60 s, exact-increase 60/60 s → 180/60 s, top-k CMS + 3×1024 60/60 s → 300/60 s (heap 46,080 → 9,216). Multi-grouping uses 9 + deployments instead of 5 and rolls up 41 RQEs instead of 52; the p99 cube + moves from DDSketch α = 0.02 to α = 0.05. Every chosen accuracy is measured; + the worst is 0.025 against 0.05. +- **Predicted cost** (cheapest plan, p95): + + | Workload | Weights | ASAP | ASAP, no roll-ups | PerQuery | AutoSketch | + |---|---|---|---|---|---| + | mixed | CPU (vCPU) | 2.38 → **0.309** | same | 7.77 → 0.725 | 2,739 → 137 | + | mixed | Fargate ($/h) | 0.170 → **0.0787** | same | 0.623 → 0.324 | 136 → 7.02 | + | all | CPU (vCPU) | 2.60 → **0.404** | 2.93 → 0.424 | 8.84 → 0.865 | 2,750 → 137.5 | + | all | Fargate ($/h) | 0.184 → **0.0874** | 0.204 → 0.0945 | 0.683 → 0.345 | 136 → 7.06 | + + ASAP's ingest CPU falls about 10× (mixed 1.84 → 0.18 vCPU), not 20×, + because of the longer windows. Its merge and query CPU barely moves (0.54 → + 0.13) and is now half or more of its cost. Memory hardly moves (ASAP 21.4 → + 15.3 GiB), so Fargate costs fall only about 2×. AutoSketch, almost all + ingest, falls the full 20×. +- **Files.** The 10/s tables, plan files and runner output are in + `/mydata/rate10-plans/` (tables, plans, results, scripts), to be committed + to sketch-bench as their own results directory before the runs. + +## 7. Semantics and ground truth + +| | Window | Instant query (`sum by (job) (data)`, T = 1 s) | +|---|---|---| +| Prometheus 2.53 | closed `[E−W, E]` | latest sample per series within the lookback | +| ClickHouse (#844) | closed (classic), `toDateTime64` bounds | same as Prometheus | +| ClickHouse + MV (#845) | same as raw, exactly | same | +| ASAPQuery | left-open `(E−W, E]` | every sample of the last second (≈10× Prometheus's sum at 10 samples/s) | +| Multi-grouping (all systems) | left-open `(E−W, E]` | — | + +- **Exact baselines** are checked against each other (Prometheus = ClickHouse = + ClickHouse + MV up to float rounding, as in A6). +- **ASAPQuery's accuracy** is measured against ClickHouse SQL in ASAPQuery's own + semantics (left-open windows; all samples of the last second for instant + queries). The instant-query difference is a semantic difference between the + systems, not an error; the paper should say ASAPQuery answers instant + queries over the whole last interval. +- **Evaluation times** are aligned to multiples of T on every system + (`align_to_interval`, #841; floored in the ClickHouse SQL, #844), with a 10 s + offset for ingestion to settle. + +## 8. Procedure + +1. **Plan.** Regenerate the tables and plans at 10 samples/s (§6), convert the + ASAPQuery plan with `--planner saved`, run `--verify-facts` on the table. +2. **Pilot (≈2 h).** All systems, real windows, check the 15m and 1h answers + and every monitor, before the long runs. +3. **Run.** Per workload, one node per system, all started together: feed for + the largest window (24 h; sliding-window queries answer only once every + sub-window is present) plus a measurement hour, querying throughout. Only + the measurement hour is reported. +4. **No timeouts.** Baseline queries run to completion: #861 lifts + Prometheus' 2-minute `--query.timeout`, VictoriaMetrics' 30 s query limit + and the ClickHouse client's 30 s timeout. A query still running when its + next firing is due is reported as late, with its full latency (§12, Q3). + +## 9. Metrics and figures + +Per (system, workload), over the measurement hour: + +- **CPU**: core-seconds of the system's own processes (ASAPQuery split into + ingest/precompute and query threads), mean vCPU. +- **Memory**: RSS, mean and peak, in GiB. +- **Latency**: per query, median and p99; misses counted per §8. +- **Results**: every answer saved; accuracy for ASAPQuery per §7 + (quantile rank error / relative error, top-k precision, relative error for + exact accumulators). +- **Cost**, as the planner defines it (cost by use, sketch-bench #188): + `w_cpu · mean vCPU + w_mem · mean GiB`, reported with cpu weights (vCPU) and + with Fargate prices (0.0405 $/vCPU-h, 0.00445 $/GB-h). Absolute costs per + system, not ratios. + +Figures and tables: + +1. Cost vs. median (and p99) latency scatter, one point per (system, + workload). +2. The paper's table (CPU reduction, median and p99 latency speedup vs. each + baseline), at this scale. +3. ASAPQuery's measured cost against the plan's predicted cost (a check of the + cost model). + +## 10. Time and resources (estimates) + +| Step | Wall time | Nodes | +|---|---|---| +| Re-plan at 10/s, build, verify | < 1 h | 1 | +| Pilot | ≈ 2 h | 5 | +| mixed: 24 h fill + 1 h measure + ≈1 h setup | ≈ 26 h | 5 (ASAPQuery, Prometheus, VictoriaMetrics, ClickHouse, ClickHouse + MV) | +| all: same | ≈ 26 h | 3 (ASAPQuery, ClickHouse, ClickHouse + MV); Prometheus and VictoriaMetrics reuse mixed's run, since their data and queries are identical | +| **Total** | **≈ 2.3 days** | the five c6320 nodes (clnode109, 155, 178, 138, 167) | + +On one node, one system at a time, the same runs take ≈ 8.5 days. + +Storage per node for a 25 h run (estimates): ClickHouse ≈ 150–250 GB for mixed +(1.8e10 rows) and ≈ 300–400 GB more for all's records; Prometheus ≈ 25–30 GB; +VictoriaMetrics ≈ 10–20 GB. Each node has ≈ 800 GB on `/mydata`; clean up +between rounds. + +## 11. What exists and what is left + +| Piece | Where | Status | +|---|---|---| +| Workload tables with e2e blocks; templates 7/9 dropped; §6.3 rerun | sketch-bench #209, #210 | open | +| Paced feeder: facts-driven classic data, records, per-series order | #840 → #842 → #843 | open | +| ClickHouse SQL and exact MV arms from the tables | #844 → #845 | open (base was #837, merged: retarget to `main`) | +| Feeder + aligned query client in `experiment_run_e2e.py` | #841 | open (base was #828, merged: retarget to `main`) | +| `--planner saved`; prebuilt configs in both runners | #828, #860 | merged | +| Tables and plans at 10 samples/s | `/mydata/rate10-plans/`; sketch-bench, new PR | generated (§6); to commit | +| Feeder → VictoriaMetrics (remote write into vmsingle) | `experiment_run_e2e.py` | to do | +| Feeder → ClickHouse live ingest, tables/views before data, server CPU/memory monitoring | `experiment_run_clickhouse.py` | to do | +| No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | #861 | open | +| Analysis: per-system CPU/memory/latency/accuracy, planner cost, scatter, table | `asap-tools/experiments/post_experiment/` | to do | +| ASAPQuery multi-grouping (all): `--planner saved` for SQL | #856, sketch-bench #209 comment | Milind | +| Sending SQL to ASAPQuery from a runner | — | run the multi-grouping part through `experiment_run_clickhouse.py`'s sketchdb mode (one language per engine process) | + +## 12. Open questions (proposed answers) + +1. **Q1, rate.** 10 samples/s per series, all other §6.3 parameters unchanged. + *Proposed: yes.* +2. **Q2, the two template-16 5m RQEs below the accuracy curves at 10/s** + (§6). *Proposed: report them as unservable in the plan and leave their two + queries out of the e2e run (all: 96 queries), rather than change the + template.* +3. **Q3, baseline queries slower than T.** No timeouts (decided). *Proposed: + keep firing on schedule and report each query's full latency and how many + firings finished after their next firing was due.* The alternative (fire + once per hour) understates the baselines' CPU. +4. **Q4, all on Prometheus/VictoriaMetrics.** *Proposed: classic part only, + reused from mixed; multi-grouping shown as "not expressible in PromQL".* +5. **Q5, ASAPQuery's instant queries.** *Proposed: keep ASAPQuery's semantics + (all samples of the last second) and state it; accuracy is measured in its + semantics.* +6. **Q6, top-k accuracy.** The plan's CMS reaches precision 0.9375 on two of + three data seeds (sketch-bench #211). *Proposed: run with the plan as + solved, report precision per query; revisit if #211 changes the accuracy + check.* + +## 13. Blocked on others + +| Item | Owner | Blocks | +|---|---|---| +| #800: MILP output uses `NoCleanup` (`milp_output.rs:230`), so planner YAML carries no retention and every window is kept | Milind (proposal: CircularBuffer with `num_aggregates_to_retain = retained_instance_count`, exactly what the MILP charges memory for) | ASAPQuery's memory grows for the whole 25 h run and no longer matches the plan. Needed before the long runs. | +| #856: `--planner saved` for SQL (with sketch-bench #209's proposed SQL strings) | Milind | ASAPQuery on all's multi-grouping queries | +| Engine roll-ups for keyed aggregations (#846) | Milind | the all plan's 52 rolled-up multi-grouping queries, if #851/#852 don't cover them | + +Not blocking, reported on #828: `IncreaseAccumulator` treats an out-of-order +sample as a counter reset (the feeder now keeps per-series order), and sliding +windows don't answer on partial data (the 24 h fill avoids it). + +## 14. Known limitations + +- One data seed per run; the top-k result depends on the seed (#211). +- 1/20 of the §6.3 rate; costs are not the §6.3 numbers scaled. +- The feeder shares the node with the system it feeds; only the system's own + processes are measured. +- Prometheus and VictoriaMetrics answer only the classic queries. +- VictoriaMetrics evaluates PromQL with MetricsQL semantics (e.g. `rate` and + range windows use the sample before the window), so its answers are compared + with Prometheus's, not required to match them. From b684e771d3ce82a958fac1c648e68c1e4c98cde1 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 13 Oct 2026 17:57:01 +0000 Subject: [PATCH 02/12] docs(eval): ASAPQuery's retention in memory, then on disk Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae --- docs/evaluation/end-to-end-eval.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index c4066411..3bbfd44d 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -168,6 +168,26 @@ plans exactly with the same binary): (`align_to_interval`, #841; floored in the ClickHouse SQL, #844), with a 10 s offset for ingestion to settle. +### ASAPQuery's retention: memory, then disk + +The baselines keep their whole history on disk and can answer any range. +ASAPQuery keeps sketch windows instead, in two tiers (#800): + +- **Memory: CircularBuffer.** Each aggregation keeps its last + `retained_instance_count` windows (`num_aggregates_to_retain`), which is + exactly what the MILP charges memory for, sized for its longest-lookback + reader. Today the MILP output uses `NoCleanup` and keeps every window, so + memory grows for the whole run. +- **Disk: spill, not drop.** A window that leaves the buffer is written to disk + (its serialized accumulator, indexed by window start) and kept for a + configured retention. A query reaching past the in-memory windows reads the + older ones from disk. The scheduled RQEs never need it, since the buffer + holds their whole lookback; ad-hoc and longer-range queries do. +- **Accounting.** The cost model is unchanged: memory is the retained windows, + and disk is not charged, as in §6.3. In the runs, spill writes count in + ASAPQuery's measured CPU, and its disk bytes are reported next to the + baselines' (§9). + ## 8. Procedure 1. **Plan.** Regenerate the tables and plans at 10 samples/s (§6), convert the @@ -190,6 +210,8 @@ Per (system, workload), over the measurement hour: - **CPU**: core-seconds of the system's own processes (ASAPQuery split into ingest/precompute and query threads), mean vCPU. - **Memory**: RSS, mean and peak, in GiB. +- **Disk**: bytes on disk at the end of the run (TSDB, ClickHouse parts, + ASAPQuery's spilled windows), reported, not charged. - **Latency**: per query, median and p99; misses counted per §8. - **Results**: every answer saved; accuracy for ASAPQuery per §7 (quantile rank error / relative error, top-k precision, relative error for @@ -263,12 +285,16 @@ between rounds. three data seeds (sketch-bench #211). *Proposed: run with the plan as solved, report precision per query; revisit if #211 changes the accuracy check.* +7. **Q7, the disk tier is not ready when the runs start.** *Proposed: run + with CircularBuffer alone and say in the paper that ASAPQuery keeps only the + windows its RQEs read; add the disk tier's write cost from a short separate + measurement.* ## 13. Blocked on others | Item | Owner | Blocks | |---|---|---| -| #800: MILP output uses `NoCleanup` (`milp_output.rs:230`), so planner YAML carries no retention and every window is kept | Milind (proposal: CircularBuffer with `num_aggregates_to_retain = retained_instance_count`, exactly what the MILP charges memory for) | ASAPQuery's memory grows for the whole 25 h run and no longer matches the plan. Needed before the long runs. | +| #800: MILP output uses `NoCleanup` (`milp_output.rs:230`), so planner YAML carries no retention and every window is kept | Milind. Agreed design (§7, "ASAPQuery's retention"): CircularBuffer with `num_aggregates_to_retain = retained_instance_count`, then evicted windows spilled to disk instead of dropped | CircularBuffer: needed before the long runs, since ASAPQuery's memory otherwise grows for 25 h and stops matching the plan. Disk tier: separate PR, ideally before the runs too. | | #856: `--planner saved` for SQL (with sketch-bench #209's proposed SQL strings) | Milind | ASAPQuery on all's multi-grouping queries | | Engine roll-ups for keyed aggregations (#846) | Milind | the all plan's 52 rolled-up multi-grouping queries, if #851/#852 don't cover them | From e2cfc3195f3f0a99c5f7a96685246515723b68ae Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 13 Oct 2026 17:58:58 +0000 Subject: [PATCH 03/12] docs(eval): link the PRs and issues in the end-to-end plan Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae --- docs/evaluation/end-to-end-eval.md | 64 +++++++++++++++--------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index 3bbfd44d..8caffca3 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -1,7 +1,7 @@ # Evaluation plan: end-to-end cost and latency (paper §6.2) -Status: draft 2026-10-13. Reuses the §6.3 workloads (#777) at 1/20 of their -data rate, on the plumbing from #840–#845, #841 and #828. The plans re-solved +Status: draft 2026-10-13. Reuses the §6.3 workloads ([#777](https://github.com/ProjectASAP/ASAPQuery/issues/777)) at 1/20 of their +data rate, on the plumbing from [#840](https://github.com/ProjectASAP/ASAPQuery/issues/840)–[#845](https://github.com/ProjectASAP/ASAPQuery/issues/845), [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841) and [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828). The plans re-solved at 10 samples/s are in §6. Open decisions are in §12. ## 1. Question @@ -17,12 +17,12 @@ the way the planner defines it (§9). - **The set of RQEs is the workload.** The workloads are the §6.3 ones: their sketch-bench tables already hold the RQEs, the PromQL (classic) or the structured query fields (multi-grouping), and the data description - (`workloads[0].e2e`, sketch-bench #208, #209, #210). + (`workloads[0].e2e`, sketch-bench [#208](https://github.com/ProjectASAP/sketch-bench/issues/208), [#209](https://github.com/ProjectASAP/sketch-bench/issues/209), [#210](https://github.com/ProjectASAP/sketch-bench/issues/210)). - **ASAPQuery runs the §6.3 planner's plan**, converted with - `asap-planner --planner saved` (#828). The planner is not re-run inside the + `asap-planner --planner saved` ([#828](https://github.com/ProjectASAP/ASAPQuery/issues/828)). The planner is not re-run inside the experiment. - **Every system gets identical data**: one paced feeder per system, same seed - and start time (#840–#843). + and start time ([#840](https://github.com/ProjectASAP/ASAPQuery/issues/840)–[#843](https://github.com/ProjectASAP/ASAPQuery/issues/843)). - **One system per CloudLab node**, so each system's CPU and memory are measured alone. - **At 1/20 of the §6.3 rate** (§5), so that every system can ingest it on one @@ -33,11 +33,11 @@ the way the planner defines it (§9). | System | Version | Queries | Data path | Runner | |---|---|---|---|---| -| ASAPQuery | `main` + this plan's PRs | PromQL (classic); SQL frontend (multi-grouping) | remote write from the feeder | `experiment_run_e2e.py` with `controller.prebuilt_output_dir` (#828, #841) | -| Prometheus | 2.53.2 (bare metal) | PromQL | remote-write receiver | `experiment_run_e2e.py` (#841) | +| ASAPQuery | `main` + this plan's PRs | PromQL (classic); SQL frontend (multi-grouping) | remote write from the feeder | `experiment_run_e2e.py` with `controller.prebuilt_output_dir` ([#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841)) | +| Prometheus | 2.53.2 (bare metal) | PromQL | remote-write receiver | `experiment_run_e2e.py` ([#841](https://github.com/ProjectASAP/ASAPQuery/issues/841)) | | VictoriaMetrics | vmsingle | PromQL (MetricsQL-compatible subset) | remote write into vmsingle | `experiment_run_e2e.py` (feeder sink: to do, §11) | -| ClickHouse | 26.10 | SQL with Prometheus 2.53 semantics (#844) | `INSERT … RowBinary` from the feeder | `experiment_run_clickhouse.py` (live feed: to do, §11) | -| ClickHouse + MV | 26.10 | SQL over exact materialized views, bucket = T (#845) | same, views created before data | same, `baseline_mv` arm | +| ClickHouse | 26.10 | SQL with Prometheus 2.53 semantics ([#844](https://github.com/ProjectASAP/ASAPQuery/issues/844)) | `INSERT … RowBinary` from the feeder | `experiment_run_clickhouse.py` (live feed: to do, §11) | +| ClickHouse + MV | 26.10 | SQL over exact materialized views, bucket = T ([#845](https://github.com/ProjectASAP/ASAPQuery/issues/845)) | same, views created before data | same, `baseline_mv` arm | PilotDB (the AQP comparison in the paper) is out of scope here; it needs its own plan (PromQL→SQL for PilotDB, its sampling modes). @@ -51,7 +51,7 @@ own plan (PromQL→SQL for PilotDB, its sampling modes). - **mixed** is the paper's \WorkloadAggregations: one gauge `data` and one counter `data_total`, labels `job` (10) → `label_0` (1,000 per job), 1e4 - series. Templates 1–6, 8, 10 (7 and 9 dropped in sketch-bench #210; template + series. Templates 1–6, 8, 10 (7 and 9 dropped in sketch-bench [#210](https://github.com/ProjectASAP/sketch-bench/issues/210); template 2's PromQL is `topk(32, sum_over_time(data[w]))`, same RQE). - **all** adds the multi-grouping templates (11–17) on two records tables, `http` (region × service × endpoint × status; `user_id`, `latency`) and @@ -87,7 +87,7 @@ own plan (PromQL→SQL for PilotDB, its sampling modes). subset's cardinality before a run. - **Order.** Ticks start on whole seconds and hold samples in `(t, t+1s]`; each series' samples of a tick go in one request, in timestamp order, and - requests run concurrently (#840). ASAPQuery's allowed lateness stays at its + requests run concurrently ([#840](https://github.com/ProjectASAP/ASAPQuery/issues/840)). ASAPQuery's allowed lateness stays at its 5 s default. Prometheus needs no out-of-order window. ## 6. Planner at 10 samples/s @@ -106,7 +106,7 @@ regenerated at 10 samples/s (`SYNTHETIC_SAMPLES_PER_SEC`). Constraints: - **Settings.** ASAP method, cheapest plan (no latency SLA), p95 accuracy as in §6.3, cpu weights and Fargate weights. -**Feasibility run** (sketch-bench runner on the #210 tables with +**Feasibility run** (sketch-bench runner on the [#210](https://github.com/ProjectASAP/sketch-bench/issues/210) tables with `SYNTHETIC_SAMPLES_PER_SEC = 10`; the 200/s tables reproduce the committed plans exactly with the same binary): @@ -152,8 +152,8 @@ plans exactly with the same binary): | | Window | Instant query (`sum by (job) (data)`, T = 1 s) | |---|---|---| | Prometheus 2.53 | closed `[E−W, E]` | latest sample per series within the lookback | -| ClickHouse (#844) | closed (classic), `toDateTime64` bounds | same as Prometheus | -| ClickHouse + MV (#845) | same as raw, exactly | same | +| ClickHouse ([#844](https://github.com/ProjectASAP/ASAPQuery/issues/844)) | closed (classic), `toDateTime64` bounds | same as Prometheus | +| ClickHouse + MV ([#845](https://github.com/ProjectASAP/ASAPQuery/issues/845)) | same as raw, exactly | same | | ASAPQuery | left-open `(E−W, E]` | every sample of the last second (≈10× Prometheus's sum at 10 samples/s) | | Multi-grouping (all systems) | left-open `(E−W, E]` | — | @@ -165,13 +165,13 @@ plans exactly with the same binary): systems, not an error; the paper should say ASAPQuery answers instant queries over the whole last interval. - **Evaluation times** are aligned to multiples of T on every system - (`align_to_interval`, #841; floored in the ClickHouse SQL, #844), with a 10 s + (`align_to_interval`, [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841); floored in the ClickHouse SQL, [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844)), with a 10 s offset for ingestion to settle. ### ASAPQuery's retention: memory, then disk The baselines keep their whole history on disk and can answer any range. -ASAPQuery keeps sketch windows instead, in two tiers (#800): +ASAPQuery keeps sketch windows instead, in two tiers ([#800](https://github.com/ProjectASAP/ASAPQuery/issues/800)): - **Memory: CircularBuffer.** Each aggregation keeps its last `retained_instance_count` windows (`num_aggregates_to_retain`), which is @@ -198,7 +198,7 @@ ASAPQuery keeps sketch windows instead, in two tiers (#800): the largest window (24 h; sliding-window queries answer only once every sub-window is present) plus a measurement hour, querying throughout. Only the measurement hour is reported. -4. **No timeouts.** Baseline queries run to completion: #861 lifts +4. **No timeouts.** Baseline queries run to completion: [#861](https://github.com/ProjectASAP/ASAPQuery/issues/861) lifts Prometheus' 2-minute `--query.timeout`, VictoriaMetrics' 30 s query limit and the ClickHouse client's 30 s timeout. A query still running when its next firing is due is reported as late, with its full latency (§12, Q3). @@ -216,7 +216,7 @@ Per (system, workload), over the measurement hour: - **Results**: every answer saved; accuracy for ASAPQuery per §7 (quantile rank error / relative error, top-k precision, relative error for exact accumulators). -- **Cost**, as the planner defines it (cost by use, sketch-bench #188): +- **Cost**, as the planner defines it (cost by use, sketch-bench [#188](https://github.com/ProjectASAP/sketch-bench/issues/188)): `w_cpu · mean vCPU + w_mem · mean GiB`, reported with cpu weights (vCPU) and with Fargate prices (0.0405 $/vCPU-h, 0.00445 $/GB-h). Absolute costs per system, not ratios. @@ -251,17 +251,17 @@ between rounds. | Piece | Where | Status | |---|---|---| -| Workload tables with e2e blocks; templates 7/9 dropped; §6.3 rerun | sketch-bench #209, #210 | open | -| Paced feeder: facts-driven classic data, records, per-series order | #840 → #842 → #843 | open | -| ClickHouse SQL and exact MV arms from the tables | #844 → #845 | open (base was #837, merged: retarget to `main`) | -| Feeder + aligned query client in `experiment_run_e2e.py` | #841 | open (base was #828, merged: retarget to `main`) | -| `--planner saved`; prebuilt configs in both runners | #828, #860 | merged | +| Workload tables with e2e blocks; templates 7/9 dropped; §6.3 rerun | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209), [#210](https://github.com/ProjectASAP/sketch-bench/issues/210) | open | +| Paced feeder: facts-driven classic data, records, per-series order | [#840](https://github.com/ProjectASAP/ASAPQuery/issues/840) → [#842](https://github.com/ProjectASAP/ASAPQuery/issues/842) → [#843](https://github.com/ProjectASAP/ASAPQuery/issues/843) | open | +| ClickHouse SQL and exact MV arms from the tables | [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844) → [#845](https://github.com/ProjectASAP/ASAPQuery/issues/845) | open (base was [#837](https://github.com/ProjectASAP/ASAPQuery/issues/837), merged: retarget to `main`) | +| Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841) | open (base was [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), merged: retarget to `main`) | +| `--planner saved`; prebuilt configs in both runners | [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), [#860](https://github.com/ProjectASAP/ASAPQuery/issues/860) | merged | | Tables and plans at 10 samples/s | `/mydata/rate10-plans/`; sketch-bench, new PR | generated (§6); to commit | | Feeder → VictoriaMetrics (remote write into vmsingle) | `experiment_run_e2e.py` | to do | | Feeder → ClickHouse live ingest, tables/views before data, server CPU/memory monitoring | `experiment_run_clickhouse.py` | to do | -| No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | #861 | open | +| No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | [#861](https://github.com/ProjectASAP/ASAPQuery/issues/861) | open | | Analysis: per-system CPU/memory/latency/accuracy, planner cost, scatter, table | `asap-tools/experiments/post_experiment/` | to do | -| ASAPQuery multi-grouping (all): `--planner saved` for SQL | #856, sketch-bench #209 comment | Milind | +| ASAPQuery multi-grouping (all): `--planner saved` for SQL | [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856), sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209) comment | Milind | | Sending SQL to ASAPQuery from a runner | — | run the multi-grouping part through `experiment_run_clickhouse.py`'s sketchdb mode (one language per engine process) | ## 12. Open questions (proposed answers) @@ -282,8 +282,8 @@ between rounds. (all samples of the last second) and state it; accuracy is measured in its semantics.* 6. **Q6, top-k accuracy.** The plan's CMS reaches precision 0.9375 on two of - three data seeds (sketch-bench #211). *Proposed: run with the plan as - solved, report precision per query; revisit if #211 changes the accuracy + three data seeds (sketch-bench [#211](https://github.com/ProjectASAP/sketch-bench/issues/211)). *Proposed: run with the plan as + solved, report precision per query; revisit if [#211](https://github.com/ProjectASAP/sketch-bench/issues/211) changes the accuracy check.* 7. **Q7, the disk tier is not ready when the runs start.** *Proposed: run with CircularBuffer alone and say in the paper that ASAPQuery keeps only the @@ -294,17 +294,17 @@ between rounds. | Item | Owner | Blocks | |---|---|---| -| #800: MILP output uses `NoCleanup` (`milp_output.rs:230`), so planner YAML carries no retention and every window is kept | Milind. Agreed design (§7, "ASAPQuery's retention"): CircularBuffer with `num_aggregates_to_retain = retained_instance_count`, then evicted windows spilled to disk instead of dropped | CircularBuffer: needed before the long runs, since ASAPQuery's memory otherwise grows for 25 h and stops matching the plan. Disk tier: separate PR, ideally before the runs too. | -| #856: `--planner saved` for SQL (with sketch-bench #209's proposed SQL strings) | Milind | ASAPQuery on all's multi-grouping queries | -| Engine roll-ups for keyed aggregations (#846) | Milind | the all plan's 52 rolled-up multi-grouping queries, if #851/#852 don't cover them | +| [#800](https://github.com/ProjectASAP/ASAPQuery/issues/800): MILP output uses `NoCleanup` (`milp_output.rs:230`), so planner YAML carries no retention and every window is kept | Milind. Agreed design (§7, "ASAPQuery's retention"): CircularBuffer with `num_aggregates_to_retain = retained_instance_count`, then evicted windows spilled to disk instead of dropped | CircularBuffer: needed before the long runs, since ASAPQuery's memory otherwise grows for 25 h and stops matching the plan. Disk tier: separate PR, ideally before the runs too. | +| [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856): `--planner saved` for SQL (with sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209)'s proposed SQL strings) | Milind | ASAPQuery on all's multi-grouping queries | +| Engine roll-ups for keyed aggregations ([#846](https://github.com/ProjectASAP/ASAPQuery/issues/846)) | Milind | the all plan's 52 rolled-up multi-grouping queries, if [#851](https://github.com/ProjectASAP/ASAPQuery/issues/851)/[#852](https://github.com/ProjectASAP/ASAPQuery/issues/852) don't cover them | -Not blocking, reported on #828: `IncreaseAccumulator` treats an out-of-order +Not blocking, reported on [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828): `IncreaseAccumulator` treats an out-of-order sample as a counter reset (the feeder now keeps per-series order), and sliding windows don't answer on partial data (the 24 h fill avoids it). ## 14. Known limitations -- One data seed per run; the top-k result depends on the seed (#211). +- One data seed per run; the top-k result depends on the seed ([#211](https://github.com/ProjectASAP/sketch-bench/issues/211)). - 1/20 of the §6.3 rate; costs are not the §6.3 numbers scaled. - The feeder shares the node with the system it feeds; only the system's own processes are measured. From 45156d94d9409df43bb9e318caed547339b6a1eb Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 13 Oct 2026 18:32:43 +0000 Subject: [PATCH 04/12] docs(eval): sketch-bench tables now carry the ASAPQuery SQL Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae --- docs/evaluation/end-to-end-eval.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index 8caffca3..7ea5d025 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -261,7 +261,7 @@ between rounds. | Feeder → ClickHouse live ingest, tables/views before data, server CPU/memory monitoring | `experiment_run_clickhouse.py` | to do | | No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | [#861](https://github.com/ProjectASAP/ASAPQuery/issues/861) | open | | Analysis: per-system CPU/memory/latency/accuracy, planner cost, scatter, table | `asap-tools/experiments/post_experiment/` | to do | -| ASAPQuery multi-grouping (all): `--planner saved` for SQL | [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856), sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209) comment | Milind | +| ASAPQuery multi-grouping (all): `--planner saved` for SQL | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209) (each structured entry carries its ASAPQuery `sql`, cd51be9); [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856) | SQL strings in the tables: done (Milind). Planner reads them: Milind, no PR yet. [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844) switches to the table's `sql` (only `NOW()` substituted). | | Sending SQL to ASAPQuery from a runner | — | run the multi-grouping part through `experiment_run_clickhouse.py`'s sketchdb mode (one language per engine process) | ## 12. Open questions (proposed answers) @@ -295,7 +295,7 @@ between rounds. | Item | Owner | Blocks | |---|---|---| | [#800](https://github.com/ProjectASAP/ASAPQuery/issues/800): MILP output uses `NoCleanup` (`milp_output.rs:230`), so planner YAML carries no retention and every window is kept | Milind. Agreed design (§7, "ASAPQuery's retention"): CircularBuffer with `num_aggregates_to_retain = retained_instance_count`, then evicted windows spilled to disk instead of dropped | CircularBuffer: needed before the long runs, since ASAPQuery's memory otherwise grows for 25 h and stops matching the plan. Disk tier: separate PR, ideally before the runs too. | -| [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856): `--planner saved` for SQL (with sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209)'s proposed SQL strings) | Milind | ASAPQuery on all's multi-grouping queries | +| [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856): `--planner saved` for SQL, reading the tables' `sql` strings (sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209), done) | Milind | ASAPQuery on all's multi-grouping queries | | Engine roll-ups for keyed aggregations ([#846](https://github.com/ProjectASAP/ASAPQuery/issues/846)) | Milind | the all plan's 52 rolled-up multi-grouping queries, if [#851](https://github.com/ProjectASAP/ASAPQuery/issues/851)/[#852](https://github.com/ProjectASAP/ASAPQuery/issues/852) don't cover them | Not blocking, reported on [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828): `IncreaseAccumulator` treats an out-of-order From f505680b680ef983e2cb1b1aecde0bb951d41b2a Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 13 Oct 2026 18:38:34 +0000 Subject: [PATCH 05/12] docs(eval): windows are not scaled with the rate Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae --- docs/evaluation/end-to-end-eval.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index 7ea5d025..cb2fd721 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -90,12 +90,21 @@ own plan (PromQL→SQL for PilotDB, its sampling modes). requests run concurrently ([#840](https://github.com/ProjectASAP/ASAPQuery/issues/840)). ASAPQuery's allowed lateness stays at its 5 s default. Prometheus needs no out-of-order window. +**Windows are not scaled.** The windows (15m–24h) define the RQEs; the rate +only sets how much data a window holds. Keeping them makes the e2e workload the +§6.3 one. Scaling them up 20× to keep the items per window would mean 5 h to 20 +day windows; scaling them down 20× would put most per-series windows below the +accuracy curves (a series' 15m window would hold 450 items) and change the +workload. At 1/20 of the rate each baseline query scans 1/20 of the data, so +the measured speedups understate ASAPQuery's advantage at the §6.3 rate. + ## 6. Planner at 10 samples/s The §6.3 plans were solved at 200 samples/s. Ingest CPU scales with the rate, but each sketch's accuracy and memory depend on its items per window `n`, so the plan can change at 1/20 of the rate, and the tables and plans are -regenerated at 10 samples/s (`SYNTHETIC_SAMPLES_PER_SEC`). Constraints: +regenerated at 10 samples/s (`SYNTHETIC_SAMPLES_PER_SEC`). These plans are for +the e2e run only; §6.3's results stay at 200 samples/s. Constraints: - **Accuracy curves start at N = 1e3.** Below a curve's first measured N the optimizer has no accuracy for the pair and the deployment is ineligible From 1fd94b79c7f8642ca930d09f01ab1437e532fb75 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 13 Oct 2026 18:53:24 +0000 Subject: [PATCH 06/12] docs(eval): the 10/s plans are sketch-bench #212 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae --- docs/evaluation/end-to-end-eval.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index cb2fd721..5e214a34 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -152,9 +152,9 @@ plans exactly with the same binary): 0.13) and is now half or more of its cost. Memory hardly moves (ASAP 21.4 → 15.3 GiB), so Fargate costs fall only about 2×. AutoSketch, almost all ingest, falls the full 20×. -- **Files.** The 10/s tables, plan files and runner output are in - `/mydata/rate10-plans/` (tables, plans, results, scripts), to be committed - to sketch-bench as their own results directory before the runs. +- **Files.** The 10/s tables, plan files and runner output are in sketch-bench + [#212](https://github.com/ProjectASAP/sketch-bench/issues/212), `rqe-optimizer/results/e2e-rate10/` (export with `--samples-per-sec 10`; + the two unservable RQEs are left out of the tables and their e2e queries). ## 7. Semantics and ground truth @@ -265,7 +265,7 @@ between rounds. | ClickHouse SQL and exact MV arms from the tables | [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844) → [#845](https://github.com/ProjectASAP/ASAPQuery/issues/845) | open (base was [#837](https://github.com/ProjectASAP/ASAPQuery/issues/837), merged: retarget to `main`) | | Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841) | open (base was [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), merged: retarget to `main`) | | `--planner saved`; prebuilt configs in both runners | [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), [#860](https://github.com/ProjectASAP/ASAPQuery/issues/860) | merged | -| Tables and plans at 10 samples/s | `/mydata/rate10-plans/`; sketch-bench, new PR | generated (§6); to commit | +| Tables and plans at 10 samples/s | sketch-bench [#212](https://github.com/ProjectASAP/sketch-bench/issues/212) | open | | Feeder → VictoriaMetrics (remote write into vmsingle) | `experiment_run_e2e.py` | to do | | Feeder → ClickHouse live ingest, tables/views before data, server CPU/memory monitoring | `experiment_run_clickhouse.py` | to do | | No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | [#861](https://github.com/ProjectASAP/ASAPQuery/issues/861) | open | From 41ba4ebe8ffebf771a4ac5cca9b6477e23117bdf Mon Sep 17 00:00:00 2001 From: Milind Srivastava Date: Tue, 13 Oct 2026 15:51:34 -0400 Subject: [PATCH 07/12] docs(eval): add accelerated history backfill --- docs/evaluation/end-to-end-eval.md | 83 +++++++++++++++++++++++------- 1 file changed, 65 insertions(+), 18 deletions(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index 5e214a34..1cc147ce 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -21,8 +21,10 @@ the way the planner defines it (§9). - **ASAPQuery runs the §6.3 planner's plan**, converted with `asap-planner --planner saved` ([#828](https://github.com/ProjectASAP/ASAPQuery/issues/828)). The planner is not re-run inside the experiment. -- **Every system gets identical data**: one paced feeder per system, same seed - and start time ([#840](https://github.com/ProjectASAP/ASAPQuery/issues/840)–[#843](https://github.com/ProjectASAP/ASAPQuery/issues/843)). +- **Every system gets identical data**: the same feeder, seed, event-time + interval, and per-series timestamp order. It bulk-loads the 24 h history, + then sends the measurement tail at the paced rate + ([#840](https://github.com/ProjectASAP/ASAPQuery/issues/840)–[#843](https://github.com/ProjectASAP/ASAPQuery/issues/843)). - **One system per CloudLab node**, so each system's CPU and memory are measured alone. - **At 1/20 of the §6.3 rate** (§5), so that every system can ingest it on one @@ -89,6 +91,20 @@ own plan (PromQL→SQL for PilotDB, its sampling modes). each series' samples of a tick go in one request, in timestamp order, and requests run concurrently ([#840](https://github.com/ProjectASAP/ASAPQuery/issues/840)). ASAPQuery's allowed lateness stays at its 5 s default. Prometheus needs no out-of-order window. +- **History and logical time.** Before each measurement, the feeder sends the + largest lookback as historical event-time data at the system's maximum + sustainable write rate. The history's last timestamp, rather than wall-clock + `now`, anchors the query clients' evaluation time. The following measurement + hour advances that logical time at the stated paced rate. Thus every query + still sees the intended 15 m–24 h of samples without waiting 24 wall-clock + hours to fill it. +- **Bulk-sender policy.** The accelerated feeder keeps one documented request + shape, timestamp ordering, batch size, and bounded in-flight request/byte + limit; it removes pacing only. The accelerated-backfill pilot selects one + conservative configuration that drains without receiver errors or runaway + backlog, then freezes it for every workload and repetition. Each system's + backpressure determines its achieved bulk rate. There is no separate + bulk-rate benchmark or per-system concurrency tuning. **Windows are not scaled.** The windows (15m–24h) define the RQEs; the rate only sets how much data a window holds. Keeping them makes the e2e workload the @@ -175,7 +191,8 @@ plans exactly with the same binary): queries over the whole last interval. - **Evaluation times** are aligned to multiples of T on every system (`align_to_interval`, [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841); floored in the ClickHouse SQL, [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844)), with a 10 s - offset for ingestion to settle. + offset for ingestion to settle. They are explicit logical timestamps during + the historical backfill and paced tail, not implicit wall-clock `now`. ### ASAPQuery's retention: memory, then disk @@ -201,12 +218,26 @@ ASAPQuery keeps sketch windows instead, in two tiers ([#800](https://github.com/ 1. **Plan.** Regenerate the tables and plans at 10 samples/s (§6), convert the ASAPQuery plan with `--planner saved`, run `--verify-facts` on the table. -2. **Pilot (≈2 h).** All systems, real windows, check the 15m and 1h answers - and every monitor, before the long runs. -3. **Run.** Per workload, one node per system, all started together: feed for - the largest window (24 h; sliding-window queries answer only once every - sub-window is present) plus a measurement hour, querying throughout. Only - the measurement hour is reported. +2. **Accelerated-backfill pilot (≈2 h).** On all systems, bulk-load a + representative 1 h event-time history, then run a short paced tail. Compare + 15 m and 1 h answers, ASAPQuery accuracy, and retained data with an + equivalent paced pilot at identical explicit evaluation timestamps. Confirm + that every receiver accepts the historical timestamps, no write or late-data + rejections occur, monitors split the phases, and the systems reach a stable + post-load flush/compaction state. Use this pilot to choose and freeze one + conservative bulk-sender configuration (§5); do not create a separate + bulk-rate benchmark. ASAPQuery uses + `wall_clock_grace_period_ms: 0` for the pilot and runs: its normal + wall-clock fallback would treat historical panes as idle; strict event-time + closure is required instead. +3. **Run.** Per workload, one node per system, all started together: bulk-load + the largest lookback (24 h) with the frozen bulk-sender configuration (§5), + allowing each system's backpressure to determine its achieved rate; wait for + each system's feeder backlog to drain and every scheduled baseline query to + return an answer, then feed and query for one paced measurement hour. A + final event-time tail advances the watermark so ASAPQuery closes the last + panes. Phase boundaries and logical evaluation times are recorded with the + results. 4. **No timeouts.** Baseline queries run to completion: [#861](https://github.com/ProjectASAP/ASAPQuery/issues/861) lifts Prometheus' 2-minute `--query.timeout`, VictoriaMetrics' 30 s query limit and the ClickHouse client's 30 s timeout. A query still running when its @@ -216,8 +247,13 @@ ASAPQuery keeps sketch windows instead, in two tiers ([#800](https://github.com/ Per (system, workload), over the measurement hour: -- **CPU**: core-seconds of the system's own processes (ASAPQuery split into - ingest/precompute and query threads), mean vCPU. +- **CPU**: core-seconds of the system's own processes for both the bulk-history + phase and paced measurement phase (ASAPQuery split into ingest/precompute and + query threads), plus mean vCPU during the paced measurement hour. Bulk CPU + seconds capture the total work to construct the 24 h history; they are not + assumed equal to paced-ingestion CPU seconds, since batching, concurrency, + and storage compaction can change work. Preserve phase boundaries and raw + samples so the analysis can later choose whether and how to amortize them. - **Memory**: RSS, mean and peak, in GiB. - **Disk**: bytes on disk at the end of the run (TSDB, ClickHouse parts, ASAPQuery's spilled windows), reported, not charged. @@ -227,8 +263,10 @@ Per (system, workload), over the measurement hour: exact accumulators). - **Cost**, as the planner defines it (cost by use, sketch-bench [#188](https://github.com/ProjectASAP/sketch-bench/issues/188)): `w_cpu · mean vCPU + w_mem · mean GiB`, reported with cpu weights (vCPU) and - with Fargate prices (0.0405 $/vCPU-h, 0.00445 $/GB-h). Absolute costs per - system, not ratios. + with Fargate prices (0.0405 $/vCPU-h, 0.00445 $/GB-h). Retain both phase + measurements before choosing whether the final figures use paced steady-state + cost, total CPU work, or an amortized combination. Absolute costs per system, + not ratios. Figures and tables: @@ -244,10 +282,10 @@ Figures and tables: | Step | Wall time | Nodes | |---|---|---| | Re-plan at 10/s, build, verify | < 1 h | 1 | -| Pilot | ≈ 2 h | 5 | -| mixed: 24 h fill + 1 h measure + ≈1 h setup | ≈ 26 h | 5 (ASAPQuery, Prometheus, VictoriaMetrics, ClickHouse, ClickHouse + MV) | -| all: same | ≈ 26 h | 3 (ASAPQuery, ClickHouse, ClickHouse + MV); Prometheus and VictoriaMetrics reuse mixed's run, since their data and queries are identical | -| **Total** | **≈ 2.3 days** | the five c6320 nodes (clnode109, 155, 178, 138, 167) | +| Accelerated-backfill pilot | ≈ 2 h | 5 | +| mixed: bulk 24 h history + settle + 1 h measure + ≈1 h setup | bulk duration + ≈2 h | 5 (ASAPQuery, Prometheus, VictoriaMetrics, ClickHouse, ClickHouse + MV) | +| all: same | bulk duration + ≈2 h | 3 (ASAPQuery, ClickHouse, ClickHouse + MV); Prometheus and VictoriaMetrics reuse mixed's run, since their data and queries are identical | +| **Total** | determined by measured bulk throughput; expected below 2.3 days | the five c6320 nodes (clnode109, 155, 178, 138, 167) | On one node, one system at a time, the same runs take ≈ 8.5 days. @@ -263,7 +301,7 @@ between rounds. | Workload tables with e2e blocks; templates 7/9 dropped; §6.3 rerun | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209), [#210](https://github.com/ProjectASAP/sketch-bench/issues/210) | open | | Paced feeder: facts-driven classic data, records, per-series order | [#840](https://github.com/ProjectASAP/ASAPQuery/issues/840) → [#842](https://github.com/ProjectASAP/ASAPQuery/issues/842) → [#843](https://github.com/ProjectASAP/ASAPQuery/issues/843) | open | | ClickHouse SQL and exact MV arms from the tables | [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844) → [#845](https://github.com/ProjectASAP/ASAPQuery/issues/845) | open (base was [#837](https://github.com/ProjectASAP/ASAPQuery/issues/837), merged: retarget to `main`) | -| Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841) | open (base was [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), merged: retarget to `main`) | +| Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841) | open (base was [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), merged: retarget to `main`); add bulk-history mode with a bounded, frozen sender configuration, explicit logical evaluation times, and phase-boundary metadata | | `--planner saved`; prebuilt configs in both runners | [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), [#860](https://github.com/ProjectASAP/ASAPQuery/issues/860) | merged | | Tables and plans at 10 samples/s | sketch-bench [#212](https://github.com/ProjectASAP/sketch-bench/issues/212) | open | | Feeder → VictoriaMetrics (remote write into vmsingle) | `experiment_run_e2e.py` | to do | @@ -298,6 +336,12 @@ between rounds. with CircularBuffer alone and say in the paper that ASAPQuery keeps only the windows its RQEs read; add the disk tier's write cost from a short separate measurement.* +8. **Q8, accelerated history.** *Proposed: bulk-load the real 24 h event-time + history through each system's normal write path and retain phase-tagged CPU + measurements for both history construction and the paced tail.* The analysis + chooses their final cost presentation after inspecting the measurements. The + accelerated-backfill pilot in §8 must pass before this replaces the original + paced 24 h fill. ## 13. Blocked on others @@ -315,6 +359,9 @@ windows don't answer on partial data (the 24 h fill avoids it). - One data seed per run; the top-k result depends on the seed ([#211](https://github.com/ProjectASAP/sketch-bench/issues/211)). - 1/20 of the §6.3 rate; costs are not the §6.3 numbers scaled. +- History is bulk-loaded, not paced for 24 wall-clock hours. Bulk CPU seconds + are retained with a separate phase label; batching and compaction can produce + a different physical layout and CPU cost from paced ingestion. - The feeder shares the node with the system it feeds; only the system's own processes are measured. - Prometheus and VictoriaMetrics answer only the classic queries. From 4cfa6aece1de88c82e6f016c080225d5b3babbd5 Mon Sep 17 00:00:00 2001 From: Milind Srivastava Date: Tue, 13 Oct 2026 15:52:25 -0400 Subject: [PATCH 08/12] docs(eval): map randomized evaluation decisions --- .../rapid-randomized-e2e-decision-map.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/evaluation/rapid-randomized-e2e-decision-map.md diff --git a/docs/evaluation/rapid-randomized-e2e-decision-map.md b/docs/evaluation/rapid-randomized-e2e-decision-map.md new file mode 100644 index 00000000..29140f85 --- /dev/null +++ b/docs/evaluation/rapid-randomized-e2e-decision-map.md @@ -0,0 +1,51 @@ +# Rapid randomized E2E evaluation: decision map + +## #1: Paper claim + +Type: Discuss + +### Question + +What claim must the fast experiment support? + +### Answer + +Across randomly generated supported query mixes, ASAPQuery has lower resource +cost and/or latency at bounded fidelity error than exact baselines, after the +same logical history is loaded. This is a rapid randomized E2E result, not a +claim to reproduce the full 24-hour online workload. + +## #2: Workload sampling + +Blocked by: #1 + +Type: Discuss + +### Question + +Which supported templates may contribute to a workload, and how will ten +queries be sampled so each workload is representative rather than accidentally +dominated by cheap or similar queries? + +## #3: Measurement protocol + +Blocked by: #1 + +Type: Discuss + +### Question + +What common logical history, backfill, query repetitions, and monitored phase +give each random workload a credible cost-latency-fidelity comparison within the +paper deadline? + +## #4: Aggregation and reporting + +Blocked by: #2, #3 + +Type: Discuss + +### Question + +How will results over ten sampled workloads be summarized and shown, including +unsupported baseline queries and fidelity metrics for each query family? From 5056a3b5ecd3ac5fa9bf5ccbe9c335af629369cd Mon Sep 17 00:00:00 2001 From: Milind Srivastava Date: Tue, 13 Oct 2026 15:53:16 -0400 Subject: [PATCH 09/12] docs(eval): scope randomized workloads to classic queries --- docs/evaluation/rapid-randomized-e2e-decision-map.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/evaluation/rapid-randomized-e2e-decision-map.md b/docs/evaluation/rapid-randomized-e2e-decision-map.md index 29140f85..07febfc7 100644 --- a/docs/evaluation/rapid-randomized-e2e-decision-map.md +++ b/docs/evaluation/rapid-randomized-e2e-decision-map.md @@ -27,6 +27,13 @@ Which supported templates may contribute to a workload, and how will ten queries be sampled so each workload is representative rather than accidentally dominated by cheap or similar queries? +### Answer + +The rapid result uses classic PromQL templates only, so ASAPQuery, Prometheus, +VictoriaMetrics, ClickHouse, and ClickHouse + exact MVs can all participate. +Multi-grouping SQL templates are a separate ASAPQuery-versus-ClickHouse result +if time permits. + ## #3: Measurement protocol Blocked by: #1 From 102fed5fbcff0574edfea5909c5b5b33c963881e Mon Sep 17 00:00:00 2001 From: Milind Srivastava Date: Tue, 13 Oct 2026 20:54:05 -0400 Subject: [PATCH 10/12] docs(eval): add randomized e2e fallback --- docs/evaluation/end-to-end-eval.md | 37 ++++++++++++++ .../rapid-randomized-e2e-decision-map.md | 50 ++++++++++++++++++- 2 files changed, 85 insertions(+), 2 deletions(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index 1cc147ce..939eef9d 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -243,6 +243,43 @@ ASAPQuery keeps sketch windows instead, in two tiers ([#800](https://github.com/ and the ClickHouse client's 30 s timeout. A query still running when its next firing is due is reported as late, with its full latency (§12, Q3). +### Rapid randomized E2E fallback (draft) + +This is a smaller deadline-oriented result, not a replacement for the full +mixed/all runs. Its claim is: across randomly generated supported query mixes, +ASAPQuery has lower resource cost and/or latency at bounded fidelity error than +exact baselines, after the same logical history is loaded. + +- **Systems.** ASAPQuery, Prometheus, and raw ClickHouse are required. + VictoriaMetrics and ClickHouse + exact MVs are optional and do not gate this + result. +- **Workloads.** Generate ten workloads from the classic PromQL query-string + pool, using seeds 0–9. Each workload samples ten distinct query strings + uniformly without replacement; different parameterizations, such as two + quantiles, are distinct strings and may co-occur. Windowed queries use only + 15 m and 1 h windows. Quantile-over-time may occur more often because it has + more eligible parameterizations. +- **Instant queries.** Templates 1 and 3 are included only if a planner-only + sensitivity run with 100 ms instant windows, using the existing saturation + curves at N = 1,000 values/job, meets their fidelity targets. The saved 10/s + plan currently models them as 1 s windows, so it cannot establish + Prometheus-equivalent latest-sample semantics by itself. +- **Protocol.** For each workload, bulk-load one hour of logical history and + run a 15-minute paced tail. Query instances keep their source cadence: one + second for instant templates and one minute for windowed templates. ASAPQuery + receives a workload-specific saved plan and a fresh run. Each baseline may + bulk-load the common history once, then run the ten workloads as + phase-separated measurements in independently randomized, recorded order. +- **Measures.** During each 15-minute phase, collect mean vCPU, RSS, raw + latencies and maxima, and fidelity. Report median latency rather than a + per-query p99 for minute-cadence queries. The common fidelity headline is + the share of ASAPQuery evaluations meeting their template target; retain + rank/relative error for quantiles, precision for top-k, and relative error + or exact match for sums, rates, and instant aggregates. +- **Pilot.** Run one sampled workload through all three systems before the + ten-workload sweep, validating data acceptance, results, monitor boundaries, + and ClickHouse ground truth in ASAPQuery's configured window semantics. + ## 9. Metrics and figures Per (system, workload), over the measurement hour: diff --git a/docs/evaluation/rapid-randomized-e2e-decision-map.md b/docs/evaluation/rapid-randomized-e2e-decision-map.md index 07febfc7..ec341a19 100644 --- a/docs/evaluation/rapid-randomized-e2e-decision-map.md +++ b/docs/evaluation/rapid-randomized-e2e-decision-map.md @@ -32,7 +32,14 @@ dominated by cheap or similar queries? The rapid result uses classic PromQL templates only, so ASAPQuery, Prometheus, VictoriaMetrics, ClickHouse, and ClickHouse + exact MVs can all participate. Multi-grouping SQL templates are a separate ASAPQuery-versus-ClickHouse result -if time permits. +if time permits. Sample queries uniformly from the eligible query-string pool; +the larger quantile-over-time family may therefore appear more often. Restrict +windowed queries to 15 m and 1 h, so each workload needs at most 1 h of logical +history rather than a 24 h fill. For each workload, sample ten query instances +without replacement from the eligible query-string pool; generate ten +independent workloads with recorded seeds 0–9. Parameterizations such as +different quantiles are distinct eligible query strings and may co-occur, but +an exact query string appears at most once in a workload. ## #3: Measurement protocol @@ -46,9 +53,40 @@ What common logical history, backfill, query repetitions, and monitored phase give each random workload a credible cost-latency-fidelity comparison within the paper deadline? +### Answer + +Each of the ten query sets receives its own ASAPQuery plan and fresh run. A +union plan for all sampled queries would charge a ten-query workload for +precomputes it does not use. Instant templates repeat every second; windowed +templates repeat every minute, matching their source workload cadence. Measure +each workload for 15 minutes: report median latency and CPU/memory, retain raw +latencies and maxima, and do not claim a robust per-query p99 for minute-cadence +queries. The required systems are ASAPQuery, Prometheus, and raw ClickHouse; +VictoriaMetrics and ClickHouse + exact MVs are optional additions and do not +gate the rapid result. + +## #5: Baseline history reuse + +Blocked by: #3 + +Type: Discuss + +### Question + +May a baseline bulk-load the common 1 h history once and run all ten +phase-separated query workloads against it, or must every workload use a fresh +baseline process and reload? The former is faster; the latter removes cache and +physical-layout carryover. + +### Answer + +Each baseline bulk-loads the common 1 h history once, then runs the ten query +sets as separately monitored phases with query statistics reset. Randomize and +record workload order independently per baseline to mitigate cache carryover. + ## #4: Aggregation and reporting -Blocked by: #2, #3 +Blocked by: #2, #3, #5 Type: Discuss @@ -56,3 +94,11 @@ Type: Discuss How will results over ten sampled workloads be summarized and shown, including unsupported baseline queries and fidelity metrics for each query family? + +### Answer + +Use the percentage of ASAPQuery evaluations meeting their template's +predeclared accuracy target as the common fidelity headline. Retain +family-specific errors alongside it: rank/relative error for quantiles, +precision for top-k, and relative error or exact match for sums, rates, and +instant aggregates. From 755823618ea89d965d11409fe1334b9a919b6c14 Mon Sep 17 00:00:00 2001 From: Milind Srivastava Date: Wed, 14 Oct 2026 15:55:57 -0400 Subject: [PATCH 11/12] docs(eval): refresh e2e implementation status --- docs/evaluation/end-to-end-eval.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index 939eef9d..2074d9c7 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -335,18 +335,18 @@ between rounds. | Piece | Where | Status | |---|---|---| -| Workload tables with e2e blocks; templates 7/9 dropped; §6.3 rerun | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209), [#210](https://github.com/ProjectASAP/sketch-bench/issues/210) | open | -| Paced feeder: facts-driven classic data, records, per-series order | [#840](https://github.com/ProjectASAP/ASAPQuery/issues/840) → [#842](https://github.com/ProjectASAP/ASAPQuery/issues/842) → [#843](https://github.com/ProjectASAP/ASAPQuery/issues/843) | open | -| ClickHouse SQL and exact MV arms from the tables | [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844) → [#845](https://github.com/ProjectASAP/ASAPQuery/issues/845) | open (base was [#837](https://github.com/ProjectASAP/ASAPQuery/issues/837), merged: retarget to `main`) | -| Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/issues/841) | open (base was [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), merged: retarget to `main`); add bulk-history mode with a bounded, frozen sender configuration, explicit logical evaluation times, and phase-boundary metadata | -| `--planner saved`; prebuilt configs in both runners | [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828), [#860](https://github.com/ProjectASAP/ASAPQuery/issues/860) | merged | -| Tables and plans at 10 samples/s | sketch-bench [#212](https://github.com/ProjectASAP/sketch-bench/issues/212) | open | +| Workload tables with e2e blocks; templates 7/9 dropped; §6.3 rerun | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/pull/209), [#210](https://github.com/ProjectASAP/sketch-bench/pull/210), [#212](https://github.com/ProjectASAP/sketch-bench/pull/212) | merged | +| Paced feeder: facts-driven classic data, records, per-series order | [#840](https://github.com/ProjectASAP/ASAPQuery/pull/840) → [#842](https://github.com/ProjectASAP/ASAPQuery/pull/842) → [#843](https://github.com/ProjectASAP/ASAPQuery/pull/843) | merged | +| ClickHouse SQL and exact MV arms from the tables | [#844](https://github.com/ProjectASAP/ASAPQuery/pull/844) → [#845](https://github.com/ProjectASAP/ASAPQuery/pull/845) | merged | +| Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/pull/841) | open: add bulk-history mode with a bounded, frozen sender configuration, explicit logical evaluation times, and phase-boundary metadata | +| `--planner saved`; prebuilt configs in both runners | [#828](https://github.com/ProjectASAP/ASAPQuery/pull/828), [#860](https://github.com/ProjectASAP/ASAPQuery/pull/860) | merged | +| Tables and plans at 10 samples/s | sketch-bench [#212](https://github.com/ProjectASAP/sketch-bench/pull/212) | merged | | Feeder → VictoriaMetrics (remote write into vmsingle) | `experiment_run_e2e.py` | to do | | Feeder → ClickHouse live ingest, tables/views before data, server CPU/memory monitoring | `experiment_run_clickhouse.py` | to do | -| No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | [#861](https://github.com/ProjectASAP/ASAPQuery/issues/861) | open | +| No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | [#861](https://github.com/ProjectASAP/ASAPQuery/pull/861) | merged | | Analysis: per-system CPU/memory/latency/accuracy, planner cost, scatter, table | `asap-tools/experiments/post_experiment/` | to do | -| ASAPQuery multi-grouping (all): `--planner saved` for SQL | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209) (each structured entry carries its ASAPQuery `sql`, cd51be9); [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856) | SQL strings in the tables: done (Milind). Planner reads them: Milind, no PR yet. [#844](https://github.com/ProjectASAP/ASAPQuery/issues/844) switches to the table's `sql` (only `NOW()` substituted). | -| Sending SQL to ASAPQuery from a runner | — | run the multi-grouping part through `experiment_run_clickhouse.py`'s sketchdb mode (one language per engine process) | +| ASAPQuery multi-grouping (all): `--planner saved` for SQL | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/pull/209), [#856](https://github.com/ProjectASAP/ASAPQuery/pull/856), [#844](https://github.com/ProjectASAP/ASAPQuery/pull/844) | merged: structured table entries carry the ASAPQuery SQL; the planner reads it; the ClickHouse arm substitutes only `NOW()`. | +| Sending SQL to ASAPQuery from a runner | `experiment_run_clickhouse.py` | ready: run the multi-grouping part through sketchdb mode (one language per engine process) | ## 12. Open questions (proposed answers) @@ -384,8 +384,8 @@ between rounds. | Item | Owner | Blocks | |---|---|---| -| [#800](https://github.com/ProjectASAP/ASAPQuery/issues/800): MILP output uses `NoCleanup` (`milp_output.rs:230`), so planner YAML carries no retention and every window is kept | Milind. Agreed design (§7, "ASAPQuery's retention"): CircularBuffer with `num_aggregates_to_retain = retained_instance_count`, then evicted windows spilled to disk instead of dropped | CircularBuffer: needed before the long runs, since ASAPQuery's memory otherwise grows for 25 h and stops matching the plan. Disk tier: separate PR, ideally before the runs too. | -| [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856): `--planner saved` for SQL, reading the tables' `sql` strings (sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/issues/209), done) | Milind | ASAPQuery on all's multi-grouping queries | +| [#800](https://github.com/ProjectASAP/ASAPQuery/issues/800): planner retention bound | merged: MILP output now bounds retention with CircularBuffer and spills evicted windows to disk | resolved | +| [#856](https://github.com/ProjectASAP/ASAPQuery/issues/856): `--planner saved` for SQL, reading the tables' `sql` strings | merged | resolved | | Engine roll-ups for keyed aggregations ([#846](https://github.com/ProjectASAP/ASAPQuery/issues/846)) | Milind | the all plan's 52 rolled-up multi-grouping queries, if [#851](https://github.com/ProjectASAP/ASAPQuery/issues/851)/[#852](https://github.com/ProjectASAP/ASAPQuery/issues/852) don't cover them | Not blocking, reported on [#828](https://github.com/ProjectASAP/ASAPQuery/issues/828): `IncreaseAccumulator` treats an out-of-order From dcb7eb4ec5b8160b584f347dcda8a7dd04bcc8c9 Mon Sep 17 00:00:00 2001 From: Milind Srivastava Date: Wed, 14 Oct 2026 19:02:05 -0400 Subject: [PATCH 12/12] docs(eval): refresh remaining e2e work --- docs/evaluation/end-to-end-eval.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/evaluation/end-to-end-eval.md b/docs/evaluation/end-to-end-eval.md index 2074d9c7..138d5d58 100644 --- a/docs/evaluation/end-to-end-eval.md +++ b/docs/evaluation/end-to-end-eval.md @@ -338,15 +338,17 @@ between rounds. | Workload tables with e2e blocks; templates 7/9 dropped; §6.3 rerun | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/pull/209), [#210](https://github.com/ProjectASAP/sketch-bench/pull/210), [#212](https://github.com/ProjectASAP/sketch-bench/pull/212) | merged | | Paced feeder: facts-driven classic data, records, per-series order | [#840](https://github.com/ProjectASAP/ASAPQuery/pull/840) → [#842](https://github.com/ProjectASAP/ASAPQuery/pull/842) → [#843](https://github.com/ProjectASAP/ASAPQuery/pull/843) | merged | | ClickHouse SQL and exact MV arms from the tables | [#844](https://github.com/ProjectASAP/ASAPQuery/pull/844) → [#845](https://github.com/ProjectASAP/ASAPQuery/pull/845) | merged | -| Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/pull/841) | open: add bulk-history mode with a bounded, frozen sender configuration, explicit logical evaluation times, and phase-boundary metadata | +| Feeder + aligned query client in `experiment_run_e2e.py` | [#841](https://github.com/ProjectASAP/ASAPQuery/pull/841) | merged for paced classic workloads; bulk-history mode with a bounded, frozen sender configuration and phase-boundary metadata remains to do | | `--planner saved`; prebuilt configs in both runners | [#828](https://github.com/ProjectASAP/ASAPQuery/pull/828), [#860](https://github.com/ProjectASAP/ASAPQuery/pull/860) | merged | | Tables and plans at 10 samples/s | sketch-bench [#212](https://github.com/ProjectASAP/sketch-bench/pull/212) | merged | | Feeder → VictoriaMetrics (remote write into vmsingle) | `experiment_run_e2e.py` | to do | -| Feeder → ClickHouse live ingest, tables/views before data, server CPU/memory monitoring | `experiment_run_clickhouse.py` | to do | +| Feeder → ClickHouse live ingest, tables/views before data, server CPU/memory monitoring | [#875](https://github.com/ProjectASAP/ASAPQuery/pull/875), `experiment_run_clickhouse.py` | live feed and pre-feed tables/views merged; server CPU/memory monitoring remains to do | | No query timeouts on the baselines (Prometheus `--query.timeout`, VictoriaMetrics search limits, ClickHouse client) | [#861](https://github.com/ProjectASAP/ASAPQuery/pull/861) | merged | | Analysis: per-system CPU/memory/latency/accuracy, planner cost, scatter, table | `asap-tools/experiments/post_experiment/` | to do | | ASAPQuery multi-grouping (all): `--planner saved` for SQL | sketch-bench [#209](https://github.com/ProjectASAP/sketch-bench/pull/209), [#856](https://github.com/ProjectASAP/ASAPQuery/pull/856), [#844](https://github.com/ProjectASAP/ASAPQuery/pull/844) | merged: structured table entries carry the ASAPQuery SQL; the planner reads it; the ClickHouse arm substitutes only `NOW()`. | -| Sending SQL to ASAPQuery from a runner | `experiment_run_clickhouse.py` | ready: run the multi-grouping part through sketchdb mode (one language per engine process) | +| Sending SQL to ASAPQuery from a runner | [#874](https://github.com/ProjectASAP/ASAPQuery/issues/874) | open for a live-fed sketchdb arm: wire remote-write ingest and shared ASAPQuery/ClickHouse feed into `experiment_run_clickhouse.py` | +| 100 ms exact ClickHouse MV arm | [#876](https://github.com/ProjectASAP/ASAPQuery/issues/876) | open; not required by the current rapid plan (instant queries repeat every second), but needed if it later adds a 100 ms query cadence | +| Rapid randomized workload manifests and offline planner inputs | [#877](https://github.com/ProjectASAP/ASAPQuery/pull/877) | merged; executing them depends on the remaining runner work above | ## 12. Open questions (proposed answers)