feat(asap-tools): run sketch-bench workload tables end to end with the paced feeder

[zzylol]

Uses #828's --planner saved and controller.prebuilt_output_dir (merged); now based on main. This is #824 B4, revised: the data comes from the paced feeder instead of fake_exporter.

Summary

Runs a sketch-bench #777 workload table end to end with experiment_run_e2e.py. The paced feeder pushes the table's data in real time, and queries repeat at each RQE's interval T at evaluation times aligned to T.

• Review fixes: rate flags per metric kind (--samples-per-sec for gauges/counters, --records-per-sec for records), time_window_seconds on SQL groups, the shared start time taken after setup, and a minimum feed duration that includes the feeder's startup check.
• Feeder service (experiment_utils/services/paced_feeder.py), configured by exporters.exporter_list.paced_feeder: {workload_table, metrics, seed, duration_secs, samples_per_sec?, extra_flags?}:
  • One feeder process per (sink, metric), on the worker nodes (the coordinator in local mode). All processes get the same seed, table and --start-ms, so every system receives identical data.
  • sketchdb mode pushes to ASAPQuery's ingest, and to Prometheus too with query_prometheus_too; other modes push to Prometheus. In sketchdb mode Prometheus no longer forwards the fed metrics to ASAPQuery, so they don't arrive twice.
  • It starts once its sinks are ready, and fails the run if a feeder dies at startup (wrong URL, etc.).
  • Startup checks: the metric is in the table and is a gauge/counter, streaming.parallelism == 1, and duration_secs outlasts the queries.
  • setup_dependencies.sh builds the feeder.
• Prometheus: when the feeder is configured, it starts with --web.enable-remote-write-receiver (bare metal and container). No out-of-order window is needed: the feeder keeps each series' samples of a tick in one ordered request (fix(asap-tools): send paced feeder ticks concurrently, ordered per series, in (t, t+1s] #840), and a CloudLab run (48M samples, 200,000 rows/s) had 0 rejections without one.
• Query-time alignment: client_options.align_to_interval evaluates repetition k at floor((start − O)/T)·T + k·T and sends it O = query_time_offset seconds later, so all systems answer the same windows.
• Config generator (sketch_bench_e2e/sketch_bench_e2e_configs.py): from a workload table (+ optional prebuilt controller output dir), it writes an experiment_type YAML. Query groups come from e2e.queries grouped by interval_secs (the same grouping as --planner saved), with metrics from e2e.metrics and the feeder block.
• ASAPQuery's allowed lateness in these runs is the engine default, 5 s (the framework never overrides it), so the feeder's within-tick reordering is safe.

Not covered yet

• Multigroup (records) metrics (feeder support in feat(asap-tools): generate records metrics in the paced feeder #843) are fed to ASAPQuery only, as <table>.<field> for its SQL frontend. The startup check refuses them when any mode also sends to Prometheus (baseline, or query_prometheus_too). The generator feeds them only with --sql-queries, and then writes a sketchdb-only config; the Prometheus baseline for the classic queries of an all table is a separate run.
• SQL query groups: the query client uses one protocol per server, so multigroup's SQL queries can't be sent to ASAPQuery's SQL endpoint from this runner yet. The generator includes them only with --sql-queries FILE.
• No cluster run yet. The end-to-end check (ASAPQuery vs Prometheus vs ClickHouse on the same data) comes with the rest of the stack.

Testing

• Tests: tests/test_paced_feeder.py (feeder command lines, sinks per mode, startup checks, the Prometheus receiver flag), tests/test_sketch_bench_e2e_configs.py (config generation from a real classic table snippet) and queriers/prometheus-client/tests/test_query_time.py (alignment math). The experiments suite passes (89 tests) under pytest and unittest. A CI step now runs the prometheus-client tests. black/flake8 are clean.
• Dry run: the generated YAML for sketch-bench's classic shared1 table loads through Hydra and passes the runner's startup checks; controller, client and Prometheus configs are generated. In dry-run mode the client, started at 12:00:35 with T = 60 s and a 10 s offset, evaluated at 12:00:00, 12:01:00 and 12:02:00.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

[milindsrivastava1997]

Code review

Note: relies on #840's feeder behavior (per-series ordered requests within a tick).

1. High — sketch_bench_e2e_configs.py:59 + core.py:204-207: uncovered queries silently return empty in sketchdb mode. The generated SKETCHDB_MODE has no query_prometheus_too, so fed metrics never reach Prometheus, but streaming.forward_unsupported_queries defaults to true (config/config.yaml:54). Any query the planner didn't cover is forwarded to an empty Prometheus and recorded as ASAPQuery's (empty) result. Suggest setting forward_unsupported_queries=false in the generator or rejecting the combination at startup.
2. High — paced_feeder.py:208 vs query_time.py (first_aligned_eval_time): sketchdb and baseline runs see different data windows. start_ms = ceil(now) + 5s is not aligned to T, while eval times are wall-clock multiples of T. Each mode runs at a different wall time, so the window ending at aligned t_k covers a different slice of the generated data per mode. Suggest aligning start_ms to the LCM of group intervals (or largest T).
3. Medium — core.py:204: fed_metrics comes from experiment_params.metrics entries with exporter: paced_feeder, but the feeder sends exporter_list.paced_feeder.metrics. If a hand-written config lists them only in the feeder block, fed_metrics is empty and with query_prometheus_too ASAPQuery receives every sample twice (inflated sums/counts). Derive from the feeder block or assert they agree.
4. Medium — paced_feeder.py:208-239: feeders launch via sequential SSH after start_ms is fixed, with a fixed 5s lead; with ~20 jobs the later feeders start after start_ms. Launch in parallel or scale the lead.
5. Medium — main_prometheus_client.py:334: first eval time floor((now - O)/T)*T can precede the feed start. With starting_delay=120 and T=3600s, repetition 0 evaluates an empty/partial window. Use the first multiple of T ≥ feeder start.
6. Medium — experiment_run_e2e.py:520/:606: no try/finally between feeder start and stop; an exception leaves feeders pushing for duration_secs into the next run's sinks.
7. Low-medium — experiment_run_e2e.py:327: with controller_auto_discover_labels, label inference runs against Prometheus before the feeder starts (and in sketchdb mode Prometheus never gets the metrics). Generated configs avoid it; hand-written ones don't.
8. Low — paced_feeder.py:202: in local mode a relative workload_table passes check_paced_feeder_config (resolved from the Python cwd) but the feeder runs in data-sources/paced-feeder, so it can't open the table.
9. Low — paced_feeder.py:106: the streaming.parallelism == 1 check fires even for baseline-only configs.

Design: the workload table is parsed in two places (paced_feeder.py:70 table_metric_types and load_e2e) and re-read on every PacedFeederService.start; a shared loader would help.

Tests: no coverage for findings 1, 2, 3, 5.

[milindsrivastava1997]

Plan for review findings #2 and #5

Real numbers from the sketch-bench tables: queries repeat every T = 60s (most), 300s or 1s, with ranges of 15m / 1h / 6h / 24h. The feeder sends one sample per second per series. The first query goes out ~3 min after the feeder starts.

Problem

Take sum(rate(x[15m])) repeating every minute. The baseline and sketchdb modes run at different times, each with its own feeder:

• Baseline: feeder starts at 10:02:23. The first query evaluates at 10:05:00 and sees samples #0–[Discussion] Requirements and design choices for a sketch-based streaming pre-compute #156. The 20th evaluates at 10:24:00 (window 10:09–10:24) and sees samples refactor(query-engine): make sketch config params required in accumulator factory #397–#1296.
• Sketchdb: feeder starts at 14:02:42. The first query evaluates at 14:05:00 and sees samples #0–Change ArroyoSketch UDF sketches to use sketchlib-rust #137. The 20th evaluates at 14:24:00 and sees samples chore: update versions in quickstart and dropin to v0.5.1 #378–#1277.

Every query compares answers over windows shifted by 19 samples, because the feeders started at different seconds within the minute. The difference comes from the input, not the sketches. (#2)

With T = 300s, the first query rounds down to the previous 5-minute mark, which falls before the feed started roughly 40% of the time, so it sees no data. (#5)

Why not just query at "feeder start + k minutes" (10:03:23, 10:04:23, ...)? ASAPQuery precomputes windows on whole minutes, so those times wouldn't line up with them.

Proposed fix: backdate the data to the last 5-minute mark

The feeder stamps its first sample with the most recent multiple of the largest T (5 min) instead of the current time. Every timestamp then runs a fixed few minutes behind real time.

• Baseline, launched at 10:02:23: the first sample is stamped 10:00:00 (2m23s behind). The 20th query evaluates at 10:20:00 and sees samples precompute_engine has no source abstraction — ingest is hardcoded to HTTP #300–#1199.
• Sketchdb, launched at 14:02:42: the first sample is stamped 14:00:00 (2m42s behind). The 20th query evaluates at 14:20:00 and sees samples precompute_engine has no source abstraction — ingest is hardcoded to HTTP #300–#1199.

Each query is sent once its data has arrived (at real time eval + delay + query_time_offset). Both modes now see the same samples in every query, queries stay on whole minutes / 5-minute marks, query k always starts k·T after the data begins (no empty first window), and nothing waits.

Risks

• Will Prometheus accept backdated samples? We only backdate by < 5 min. Tested locally on v2.53.2 and v3.9.1 with samples ~1h old: all were accepted and queryable.
• Work needed. The feeder needs a "backdate by X" option, so feeders launched a few seconds apart over SSH use the same timestamps. The client needs to be told the backdated start, so it knows which times to evaluate and when to send.
• SQL queries are fine, because they use the time the client passes in. The exception is a hand-written query that calls NOW() itself.
• Not addressed here: 1h / 6h / 24h ranges are mostly empty during a ~1h run (the data is only minutes to an hour old). After this fix both modes see the same partial data, so the comparison is fair, but those queries don't exercise a full window.

[milindsrivastava1997]

Automated code review: 10 findings (inline).

[milindsrivastava1997]

SQL query groups get a group-level time_window_seconds: 0, but generate_controller_client_configs only strips client_options, and the planner's QueryGroup is #[serde(deny_unknown_fields)]. A run with --sql-queries and no --prebuilt-output-dir fails because asap-planner rejects the config with 'unknown field time_window_seconds'.

[milindsrivastava1997]

PACED_FEEDER_START_LEAD_SECS (5 s) has to cover SSH launches that run one node at a time, and run_cmd_with_retry sleeps 5 s on each rc=255 retry. With roughly 8 or more worker nodes, or a single retried SSH, the launches pass send_at_ms and the run aborts. Launch each node with Popen and wait on all of them, or scale the lead with the node count.

[milindsrivastava1997]

start_ms is backdated to a multiple of the LCM of the aligned intervals, so the lag can approach the full LCM. Prometheus rejects samples older than its newest sample minus 1 h, and its newest samples come from wall-clock scrapes. With intervals of 420 s and 3600 s, the LCM is 25200 s and every fed sample is rejected, so the baseline comes back empty. Consider capping or rejecting an LCM above Prometheus's out-of-order window.

[milindsrivastava1997]

repetition_delay_ms // 1000 turns a sub-second interval into 0, so the LCM is 0 and backdated_start_ms divides by zero (repetition_delay_ms: 500 raises ZeroDivisionError). The client's 'whole-second repetition_delay_ms' error is never reached. A value like 1500 ms is silently truncated to 1 s. Validate that the value is a whole number of seconds here.

[milindsrivastava1997]

When a paced feeder is configured and the filtered remote-write list is empty, all forwarding to ASAPQuery is now disabled. Before this change, an empty list meant 'forward everything'. With a paced feeder plus another scraped exporter whose metrics aren't listed in experiment_params.metrics, ASAPQuery never receives that exporter's data.

[milindsrivastava1997]

atexit.register(paced_feeder_service.stop) runs on every exit, even when flow.no_teardown is set. Prometheus and ASAPQuery are left running for inspection, but the feeders are killed, so data stops arriving.

[milindsrivastava1997]

Aligned groups default query_time_offset to 0. Each repetition then fires at t_k + L, before ASAPQuery's 5 s allowed lateness closes the window, so sketchdb returns partial results. Consider requiring an offset for aligned groups, or defaulting it to at least the allowed lateness.

[milindsrivastava1997]

pgrep -fc counts every paced_feeder process on the node and requires the count to equal exactly this run's job count. A stray or concurrent feeder on a shared node (count 3, expected 2) fails a healthy run. Checking the PIDs of the launched processes (e.g. echo $! per launch) would be more robust.

[milindsrivastava1997]

extra_flags and other config values are inserted unescaped into a command that get_ssh_cmd wraps in double quotes. A value like --label "a b", or one containing $, breaks the quoting, and the only symptom is the vague 'expected N processes' error later. Consider shlex.quote.

[milindsrivastava1997]

Nit: get_exporter_target_nodes now has a generic name but still lives in fake_exporters.py, so the paced feeder depends on the fake-exporter module. Consider moving it to a shared module.

[milindsrivastava1997]

Review follow-up (06289b2)

Backdating is gone. It needed too many guards (Prometheus' sample-age limit, client lag plumbing). Instead, each mode's feeders now wait for the next multiple of the aligned intervals' LCM and start there. For today's tables (T = 1, 60, 300 s) that's a wait of at most 5 min per mode. Every mode's data still starts on the same boundary, so each query covers the same samples in every mode. The client sends each query at its eval time + query_time_offset. --send-at-ms is removed from the feeder.

Addressed:

• LCM too large: refused above 600 s (that's how long the feeder would wait).
• Sub-second intervals: aligned groups must use whole-second repetition_delay_ms.
• query_time_offset of 0: aligned groups need at least 5 s (ASAPQuery's allowed lateness). The generator uses 10 s.
• 5 s launch lead: now 5 s + 10 s per node, which covers one SSH retry each. An overrun still stops the feeders and fails the run.
• atexit under no_teardown: only registered when teardown is on.
• SQL groups without prebuilt configs: the generator refuses --sql-queries without --prebuilt-output-dir.
• Forwarding disabled: errors when other exporters run next to the feeder but none of their metrics are listed.

Not changed: the pgrep exact count (only matters when experiments share nodes), extra_flags quoting (quoting would break passing several flags), and moving get_exporter_target_nodes.