Repository navigation
feat(asap-tools): render sketch-bench e2e queries as ClickHouse SQL - #844
Conversation
…nerated2 workloads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e SQL configs Adds a client_options.simulated_start_time so templated SQL can be evaluated against bulk-loaded data with fixed timestamps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… configs
sketch_bench_workloads.py reads a sketch-bench table's workloads[0].e2e block
and writes, per table, the metric tables' init SQL, one SQL file per query
interval and a baseline experiment_run_clickhouse.py config whose query
groups repeat every interval.
Queries keep generate_workloads.py's Prometheus 2.53 semantics (range
vectors read (T - d, T], extrapolatedRate for rate/increase,
quantileExactInclusive) and add the classic template shapes: topk over
sum by, sum/count/quantile by over instant vectors (latest sample per series
in (T - 1s, T]) and rates, and the ratio of two quantile_over_time. Eval
time is {{ end_time }} floored to the interval. Unsupported shapes and
structured (non-PromQL) entries raise.
generate_workloads.extrapolated_rate now builds on rate_from_samples, which
the new module reuses; its SQL output is unchanged.
The test compares every generated statement in clickhouse-local against a
Python implementation of the Prometheus semantics at several eval times,
covering empty windows and every extrapolation branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
… SQL Experiments run Prometheus 2.53.2 (installation/prometheus/install.sh), whose range selectors and instant lookback are closed, [T - d, T]; left-open windows arrived in 3.0. Aligned eval times put a sample on the left boundary, so the SQL now reads ts >= T - d. The Python reference selects [T - d, T] as well. generate_workloads.py's generated2 SQL is unchanged; only the shared rate_from_samples docstring no longer names a window shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Structured e2e entries {table, value, capability, grouping, quantile,
window_secs, interval_secs} become pooled queries over a records table.
Cardinality is uniqExact of the value field per group, and quantile is
quantileExactInclusive over all of a group's samples. Both read the same
closed [T - W, T] window, with T floored to the interval as the PromQL
queries are. init.sql now creates records tables, with one Float64 column
per value field. Each table's structured queries are also written in
ASAPQuery's SQL frontend dialect (COUNT(DISTINCT col), QUANTILE(q, col),
BETWEEN DATEADD(s, -W, NOW()) AND NOW()) to asapquery.sql.
The clickhouse-local test adds a skewed records dataset in which endpoint is
a child of service. It checks template 16's CUBE over (region, service,
endpoint) and several cardinality groupings against a Python reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Multigroup (structured) queries involve no Prometheus, so they now read
ASAPQuery's left-open window (E - W, E] instead of the closed [E - W, E]
that classic PromQL keeps to match Prometheus 2.53. ASAPQuery's SQL
frontend and precompute windows are (start, end], so ClickHouse and
ASAPQuery compute over the same samples. Query gets a `left_open` flag that
structured entries set; the raw SQL uses `ts > E - W`.
The ASAPQuery-dialect file ends its window at the floored eval time instead
of NOW(). The frontend cannot floor a literal, so the client's
{{ end_datetime }} is moved back by {{ end_time % T }} inside DATEADD:
ts BETWEEN DATEADD(s, -{{ end_time % T + W }}, '{{ end_datetime }}')
AND DATEADD(s, -{{ end_time % T }}, '{{ end_datetime }}')
which the frontend parses to (start, duration) = (E - W, W).
The Python reference for multigroup now pools records in (t - w, t].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…se configs
The SQL floors {{ end_time }} to T, but the current query client sleeps T
after each repetition's queries finish, so its eval times drift by the
query latency and the arms answer different windows. Each query group now
sets `align_to_interval: true` and `query_time_offset: 10` (PR #841's
default). With #841's client, repetition k is evaluated at
floor((start - O) / T) * T + k * T and sent O seconds later.
The current client ignores `align_to_interval` (unknown client options
become unused attributes) and shifts its eval time back by the offset. The
module docstring records the drift until #841's client lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…w SQL ClickHouse 26.10 turns `ts > <integer>` on a DateTime64 key column into `ts in [<integer> + 1, +Inf)` for index pruning, so the left-open multigroup SQL dropped the samples in the window's first second from the result (A6: raw vs MV relative differences of 1e-6 to 2.6e-3 on 49 of 52 queries). The raw SQL now compares ts with toDateTime64(<bound>, 3) on both ends, for classic and multigroup queries. A regression test runs a left-open and a closed query in clickhouse-local over a table with 8-row granules and samples every 7 ms around the window start. Without the cast it reproduces the loss (8435 of 8571 rows). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Brings in #837 (this branch's base, squash-merged) and #828. The one conflict, generate_workloads.py (add/add), keeps this branch's version: main's copy is #837's original commit, and this branch later moved extrapolatedRate into rate_from_samples and closed its windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
sketch-bench's exporter now writes each structured e2e entry's ASAPQuery
SQL (`sql`, e.g. `... WHERE ts BETWEEN DATEADD(s, -300, NOW()) AND NOW()
...`), so it is the single ASAPQuery SQL renderer. asapquery.sql now takes
that SQL as written and only replaces NOW() with the floored eval time; this
module's own dialect renderer is gone. The ClickHouse raw SQL still comes
from the structured fields.
The frontend's DATEADD takes only NOW(), a literal or a CAST as its base, so
NOW() becomes a datetime literal floored in the client's Jinja template:
{{ end_datetime }} (UTC) with its seconds floored for T dividing 60, or its
minutes for 60 | T | 3600. Other intervals raise.
All 936 distinct structured queries in sketch-bench cd51be9's tables
(T = 60 s and 300 s), rendered through the client's QueryTemplate at three
end times, parse in ASAPQuery's SQL frontend to (start, duration) =
(floor(E) - W, W) and pass its matcher.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Code reviewNote: eval-time alignment depends on #841; ClickHouse vs PromQL comparisons aren't valid until it lands. The
Tests: strong (clickhouse-local reference, mutation checks, DateTime64 regression). Gaps: empty grouping, multi-line PromQL, PromQL on a records metric. |
- Render an empty structured grouping as the grand total, no GROUP BY. - Comment PromQL on one line so multi-line PromQL stays a comment. - Reject PromQL over a records metric, and ASAPQuery sql without NOW(). - Document the 1 s instant lookback and the UTC server requirement. - Drop PR references from comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed in 6418827:
🤖 Generated with Claude Code |
milindsrivastava1997
left a comment
There was a problem hiding this comment.
Automated code review (Claude Code). Findings are unverified candidates; ordered by severity — the first two affect correctness of baseline numbers on current main.
| f"-- {describe(entry)}\n{raw_sql(q, labels)}" | ||
| ) | ||
| if "promql" not in entry: | ||
| asapquery.append(f"-- {describe(entry)}\n{asapquery_sql(entry)}") |
There was a problem hiding this comment.
align_to_interval is a no-op on main. QueryGroupConfig just copies it into __dict__; main_prometheus_client.py uses now() - query_time_offset each repetition and sleeps repetition_delay_ms after queries finish, so eval times drift by query latency. With intDiv(end_time, T)*T in the SQL, consecutive repetitions eventually floor to the same T (duplicate window) or skip one — breaks the docstring's "every run and arm answers the same windows" until #841 lands.
| "dataset": { | ||
| "name": "custom", | ||
| "local_data_file": data_file, | ||
| # The runner's loader fills one table; the paced feeder writes all. |
There was a problem hiding this comment.
All data lands in one table. dataset.table = next(iter(labels)) and the runner's custom loader streams the whole data_file into that one table; nothing on main ("the paced feeder") fills the others, and unlike generate_workloads.py no simulated_start_time is set. For multi-metric configs (e.g. all-metrics8) the first table gets polluted with other metrics' rows and the rest stay empty; with fixed-timestamp data files, wall-clock windows return empty results.
There was a problem hiding this comment.
Real. The comment was wrong: on main the feeder's ClickHouse sink has a fixed schema, and nothing starts one feeder per metric for the ClickHouse runner (#841 feeds only Prometheus and ASAPQuery). In 18b9091 the comment now says the loader fills only dataset.table. I didn't restrict the generator to single-metric tables, since multigroup and all-metrics8 are the point of this PR. Wiring the feeder into the ClickHouse runner is tracked in #868.
| @@ -71,14 +71,23 @@ def extrapolated_rate(cols: str, table: str, window: int, is_rate: bool) -> str: | |||
| f"FROM {table} WHERE ts > {END} - {window} AND ts <= {END} " | |||
There was a problem hiding this comment.
Two contradictory "Prometheus 2.53" window conventions now coexist. This file uses left-open ts > END - window; sketch_bench_workloads.py uses closed [T-d, T] for the same Prometheus version. With 1s whole-second scrapes a [1m] query reads 60 samples here vs 61 there, so sum/count/quantile and rate extrapolation (first_t/to_start) diverge. Suggest one shared window helper. (Anchored here; the select() at line ~60 has the same predicate.)
There was a problem hiding this comment.
Prometheus 2.x range selectors are closed, [T−d, T]; left-open arrived in 3.0. So this PR is correct and generated2 is off by one sample. Changing generated2 (merged in #837) changes its results, so it gets its own PR rather than going into this one: #867. generated2's ts is DateTime, so the fix there is just > → >=.
| by_interval.setdefault(q.interval_secs, []).append( | ||
| f"-- {describe(entry)}\n{raw_sql(q, labels)}" | ||
| ) | ||
| if "promql" not in entry: |
There was a problem hiding this comment.
Repetitions / starting delay unvalidated. repetitions = duration_secs // interval silently yields 0 when interval_secs > --duration-secs. And starting_delay < QUERY_TIME_OFFSET_SECS + max window (e.g. 1h window, 60s delay) makes early windows reach before ingestion started, biasing rate extrapolation and pooled quantiles. Raise on both.
There was a problem hiding this comment.
Fixed in 18b9091. convert raises when an interval exceeds --duration-secs (0 repetitions). A short starting delay now warns instead of raising: every arm sees the same partial early windows, so the comparison holds, and raising would force a delay over 1 h for 1 h-window tables. Both are tested.
| text = " ".join(promql.split()) | ||
| m = TOPK.match(text) | ||
| if m: | ||
| inner = parse_promql(m[2], interval, labels) |
There was a problem hiding this comment.
parse_promql only collapses whitespace, so valid formatting variants fail: multi-line sum by (job) (\n data\n) normalizes to sum by (job) ( data ) and SELECTOR ^(\w+)$ rejects ' data ' — the test's own describe output hits this. Same for sum by(job) (...), sum(...) by (job), topk(5,sum ...).
There was a problem hiding this comment.
Deferring. These forms fail loudly at generation time (ValueError), and all 1908 statements of the real classic, multigroup and all-metrics8 tables parse. One correction: the describe test feeds multi-line PromQL to describe only, never to the parser, so it isn't evidence that real tables hit this. Will widen the regexes if sketch-bench starts emitting these forms.
| if t % 60 == 0 and 3600 % t == 0: | ||
| floored = f"end_time % 3600 // {t} * {t} // 60" | ||
| return f'{{{{ end_datetime[:14] }}}}{{{{ "%02d" | format({floored}) }}}}:00' | ||
| raise ValueError(f"cannot floor a datetime to {t}s: T must divide 60 or 3600") |
There was a problem hiding this comment.
No-label metrics produce invalid SQL: labels: [] → ORDER BY (, ts) in init.sql, and SELECT , argMax(value, ts) ... GROUP BY / SELECT , delta * ... in instant/rate paths. Fails at ClickHouse run time rather than at generation time. Either support it or raise here.
There was a problem hiding this comment.
Fixed in 18b9091: init_sql raises on a metric with no labels. convert calls it first, so generation fails before any query is rendered. Tested.
| raise ValueError(f"unknown labels or value for {table!r}: {entry}") | ||
| return Query( | ||
| table, | ||
| entry["interval_secs"], |
There was a problem hiding this comment.
Unknown table in a structured entry raises bare KeyError from metric_labels(metrics)[table] (no entry context) instead of the module's ValueError. Also metric_labels(metrics) is rebuilt per entry (1696× for all-metrics8) — pass in the labels convert() already has.
There was a problem hiding this comment.
Fixed in 18b9091: an unknown table raises ValueError with the entry, and query_from_entry takes the labels map convert already built. Tested.
| if 60 % t == 0: | ||
| floored = f"end_time % 60 // {t} * {t}" | ||
| return f'{{{{ end_datetime[:17] }}}}{{{{ "%02d" | format({floored}) }}}}' | ||
| if t % 60 == 0 and 3600 % t == 0: |
There was a problem hiding this comment.
Label / value-field names go into DDL unquoted and aren't checked against ts/value or each other. A gauge label named value, or a records value field sharing a label name, gives a duplicate-column CREATE TABLE; a keyword-named label breaks SELECT/GROUP BY.
There was a problem hiding this comment.
Deferring: the current tables don't have these names, and a clash fails loudly at CREATE TABLE.
| q, | ||
| tuple(labels[q.table]), | ||
| q.table, | ||
| # DateTime64 bounds: ClickHouse 26.10 prunes `ts > <integer>` on the |
There was a problem hiding this comment.
asapquery_sql copies the table's sql verbatim then joins with ';\n\n'; a trailing ; in sketch-bench's SQL yields ;;. Splitting on ; (as _load_sql_queries does) then produces empty statements and misaligns ASAPQuery-arm query indices vs the ClickHouse arm. Strip trailing ;.
There was a problem hiding this comment.
Fixed in 18b9091: trailing ; is stripped. Tested. For the record, it couldn't misalign indices: _load_sql_queries and the runner both drop empty statements (if s.strip()).
| } | ||
|
|
||
|
|
||
| def write_statements(path: str, statements: List[str]) -> None: |
There was a problem hiding this comment.
(workload,) = json.load(f)['workloads'] requires exactly one workload, but the docstring says it reads workloads[0].e2e. More than one gives an opaque unpack error — either use [0] or raise a clear error.
There was a problem hiding this comment.
Fixed in 18b9091: anything other than exactly one workload raises a ValueError with the count. Tested.
- Raise on an unknown structured table, a label-less metric, more than one workload, and an interval longer than --duration-secs. - Warn when --starting-delay-secs is shorter than offset + longest window. - Strip a trailing ';' from ASAPQuery sql; reuse convert()'s label map. - Correct the comment on which tables the runner fills. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds on #837 (PromQL → ClickHouse SQL for generated2, merged); now based on
main.Summary
Renders a sketch-bench #777 workload table's
e2e.queriesas ClickHouse SQL, for the ClickHouse baseline of the end-to-end runs (#824). The newdatasets/fake_metric/sketch_bench_workloads.pysits next to #837'sgenerate_workloads.pyand shares itsextrapolatedRate(split out asrate_from_samples, whose generated text is unchanged).Queryrecord, adding the shapes feat(asap-tools): run generated2 PromQL workloads on ClickHouse #837 lacks:topk(k, sum by (x) (...)),sum by (x) (rate(...)),quantile by (x) (q, instant), the ratio of twoquantile_over_time, and the per-series forms.[T−d, T], and instant vectors take each series' latest sample in[T−1s, T]. Measured on CloudLab against a real Prometheus 2.53: its[5s]sums match ClickHouse'sts >= T−5s AND ts <= Texactly, while(T−5s, T]misses the boundary samples. The paced feeder's timestamps and aligned eval times always put samples on that boundary.(T−d, T](left-open is Prometheus 3.0). With 1 s scrapes on whole seconds, a[1m]window reads 60 samples where Prometheus 2.53 reads 61. This PR leaves it unchanged; it's your call there.uniqExact(value)for cardinality andquantileExactInclusive(q)(value)pooled over every sample of a group. Windows are ASAPQuery's left-open(E−W, E], since no Prometheus is involved and ASAPQuery's SQL frontend and precompute windows are(start, end].<table>/asapquery.sqlfor the ASAPQuery arm: each entry'ssqlfrom the table as written (sketch-bench#209 renders the ASAPQuery dialect once,QUANTILE(q, col),COUNT(DISTINCT col),BETWEEN DATEADD(s, -W, NOW()) AND NOW()), with onlyNOW()replaced by the eval time floored to T. All 936 distinct queries of the 12 tables parse in ASAPQuery's SQL frontend to (⌊E⌋−W, W).{{ end_time }}is floored to a multiple of T in the SQL, matching the aligned query client (feat(asap-tools): run sketch-bench workload tables end to end with the paced feeder #841).experiment_run_clickhouse.pyconfig per table, one query group perinterval_secs(repetition = T), plusinit.sqlwith the tables (gauge/counter:ts, labels,value; records:ts, labels, value fields). Unknown shapes, capabilities or labels raise.Testing
httprecords table withendpointa child ofservice.extrapolatedRatebranch.SQLPatternParser/SQLPatternMatcherto (start, duration) = (⌊E⌋−W, W).CLICKHOUSE_BINARY=<path> python3 -m pytest -q tests/test_sketch_bench_workloads.pypasses (the ClickHouse tests are skipped without a binary). The experiments suite passes. black/flake8 clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae