feat(asap-tools): feed every metric table of a sketch-bench workload into ClickHouse

[milindsrivastava1997]

Closes #868. Closes #873.

Summary

sketch_bench_workloads.py (#844) writes one ClickHouse table per metric, but the ClickHouse runner only bulk-loaded dataset.local_data_file into one table, so the others stayed empty. And since #841 the query client refuses align_to_interval groups without data_start_time, so #844's configs failed at client startup.

With dataset.paced_feeder set, experiment_run_clickhouse.py now feeds every metric table live, reusing #841's feeder plumbing:

• Feeder service (paced_feeder.py): a clickhouse sink (--sink clickhouse, one table per metric, records metrics too). PacedFeederService takes its target nodes directly; the table/type/duration checks are split into check_paced_feeder_feed for reuse.
• Runner: per mode, runs the init SQL (so baseline_mv's views exist before data), starts one feeder per metric on the ClickHouse node at an aligned start (aligned_interval_lcm / aligned_start_ms), writes data_start_time into the mode's client config (set_data_start) and rsyncs it, runs the queries, then stops the feeders. Configs without paced_feeder bulk-load as before. sketchdb mode is refused with a feeder (its engine ingests a data file); follow-up in asap-tools: feed the sketchdb arm in the ClickHouse runner #874.
• Validation: a fed config needs an existing paced_feeder.workload_table instead of local_data_file.
• Generator: sketch_bench_workloads.py takes --seed (and optional --samples-per-sec / --records-per-sec) instead of --data-file, and emits the feeder block for all metrics.
• Install: the feeder is now its own internal component (data-sources/paced-feeder/installation/install.sh, listed in components.conf), so only_install_internal_components.sh rebuilds it after a code sync; previously only the one-time common/setup_dependencies.sh built it.

Each ClickHouse arm gets its own feed; with the same seed and table, repetition k covers the same samples in every arm because each arm's start is aligned and its data_start_time matches its feed.

Testing

• Unit: tests/test_paced_feeder.py (ClickHouse sink command lines), tests/test_experiment_run_clickhouse.py (feeder config checks; init SQL before the feed, aligned start, data_start_time == start_ms // 1000, sink URL), tests/test_sketch_bench_workloads.py (feeder block for every metric; generated config passes validate_experiment_config, which failed before the fix). Experiments suite: 156 passed. black/flake8 clean.
• CloudLab (one node), sketch-bench e2e-rate10/synthetic-templatesall-shared1 (gauge, counter, two records metrics; ~200k rows/s + 300k records/s), seed 42, 600 s query phase:
  • baseline: 4 feeders, one shared start, all ran to the end without errors; queries on all four tables returned non-empty results (4500 results, 0 empty, 0 errors).
  • baseline_mv, run ~17 min later: all 4500 answers (96 queries x repetitions) equal baseline's within 1e-6 relative, so the aligned starts line up across runs.
  • At the table's full rate, flows/http overran some ticks (up to 2.5 s) in bursts while the 60 s query group (1-day windows) ran on the same node, then caught up. Scale with --records-per-sec or move the feeder off the ClickHouse node if that matters.

🤖 Generated with Claude Code

[milindsrivastava1997]

Code review: 3 bugs in stopping the feeders between modes (no_teardown, killed before stats are written, no wait for exit), plus robustness and cleanup notes inline.

[milindsrivastava1997]

Bug: with flow.no_teardown=true, a mode's feeders are never stopped before the next mode starts. The next mode's init SQL drops and recreates the tables while the old feeders keep inserting, and PacedFeederService.start then sees 8 processes instead of 4 and raises RuntimeError. experiment_run_e2e.py stops the feeder at the top of every mode whatever no_teardown says; do the same before _start_clickhouse_feed.

[milindsrivastava1997]

Fixed in 6e482c5. _start_clickhouse_feed now stops any feeders and waits until pgrep finds none (PacedFeederService.wait_for_exit) before the mode's init SQL, whatever no_teardown says, and raises if one outlives pkill by 30 s. Test: StartClickHouseFeedTest (order stop → wait → init SQL → start; feeders that survive stop the run before the init SQL).

[milindsrivastava1997]

Bug: two problems with stopping here.

1. pkill reaches each feeder about 60 s before its duration_secs ends. The feeder writes --stats-out only when it finishes, so the per-metric <sink>_<metric>_stats.json files (tick overruns and achieved rate) are never produced for ClickHouse runs.
2. stop() returns without waiting for the processes to exit. An INSERT already in flight can finish after the next mode's CREATE TABLE and put the previous arm's rows, at overlapping timestamps (same seed), into the new arm's tables. Wait until pgrep finds no feeders before running the init SQL.

[milindsrivastava1997]

Both fixed in 6e482c5.

1. After the queries, _finish_clickhouse_feed waits for the feeders to reach the end of duration_secs and exit on their own, so they write --stats-out; it stops any still running 30 s later. With the generator's 60 s slack this adds about a minute per mode. (experiment_run_e2e.py has the same kill-before-stats behaviour; left for a separate change.)
2. The next mode stops and then waits for no feeders to be running before its init SQL (see the reply above).
   Tests: FinishClickHouseFeedTest, WaitForExitTest.

[milindsrivastava1997]

The feed-duration check allows only the 2 s startup check before queries start. In this runner the query client starts 10–20 s later, after remote_monitor and the container launch, so with a hand-written config at the minimum duration_secs, groups without align_to_interval keep querying after the feed has stopped. The generator's 60 s slack hides this only for generated configs.

[milindsrivastava1997]

Fixed in 6e482c5: the minimum now adds QUERY_CLIENT_STARTUP_SECS = 30 (remote_monitor and the client container start after the feed; 10–20 s on CloudLab). The generator's 60 s slack still covers it. Test: test_feed_must_outlast_the_query_client_startup.

[milindsrivastava1997]

For fed configs, validation should also require dataset.init_sql_file, which is now the only thing that creates the metric tables; without it the run fails only after setup. A missing workload_table also raises an unhelpful ConfigAttributeError here instead of a clear ValueError.

[milindsrivastava1997]

Fixed in 6e482c5: fed configs now require an existing dataset.init_sql_file, and a missing workload_table raises a ValueError naming it. Tests: FedClickHouseConfigValidationTest.

[milindsrivastava1997]

run_init_sql is passed the hard-coded string "custom" instead of the configured dataset name, yet validation accepts feeders for clickbench and h2o. Either pass dataset_name or have validation reject feeders for non-custom datasets.

[milindsrivastava1997]

Fixed in 6e482c5: validation now rejects paced_feeder unless dataset.name is custom (the feeder only produces sketch-bench tables), and the runner passes dataset_name through.

[milindsrivastava1997]

run_init_sql calls _ensure_clickhouse_http_ready again, so the bulk-load path, which already ran it in start(), now checks readiness twice. Keep the check in the callers, or split out a helper without it.

[milindsrivastava1997]

Fixed in 6e482c5: the copy-and-execute steps are now a private _exec_local_sql_file with no readiness check. start() (one check) and the built-in DDL branch use it, and only the public run_init_sql checks readiness.

[milindsrivastava1997]

13 positional parameters, several of them derivable (the experiment and local dirs can be built from the root dirs plus experiment_mode). Neighbouring str arguments can be swapped silently at the call site. Compute the derivable ones inside, or call it with keyword arguments.

[milindsrivastava1997]

Fixed in 6e482c5: the parameters are keyword-only, the per-mode output dirs are built inside from the root dirs and experiment_mode, and the call site uses keywords.

[milindsrivastava1997]

aligned_interval_lcm enforces ASAPQuery's 5 s allowed-lateness rule on query_time_offset. That rule doesn't apply here, because this runner refuses sketchdb mode, so a ClickHouse-only config with query_time_offset=2 is rejected for no reason.

[milindsrivastava1997]

Leaving this as is. It only rejects hand-written configs with query_time_offset < 5 s; the generator uses 10 s. ClickHouse needs about 2 s anyway: a tick's rows are inserted after its second ends, so 5 s is a reasonable floor here too. Relaxing it would mean a parameter on the helper shared with experiment_run_e2e.py. Happy to revisit if a ClickHouse-only config needs a smaller offset.