Skip to content

ClickHouse runner: pass data_start_time to aligned query groups #873

Closed

Description

@milindsrivastava1997

Blocked by #868: experiment_run_clickhouse.py only bulk-loads a data file, so there is no live feed whose start to align to.

Problem

With align_to_interval, the query client should evaluate at multiples of T counted from where the data starts, so every arm (ASAPQuery, ClickHouse, ClickHouse + MV) answers over the same samples.

Example (T = 60 s): two arms run at different times, with feeds starting at 10:02:23 and 14:02:42. Rounding the wall clock gives windows that cover different samples in each arm (shifted by 19 s here), so accuracy comparisons mix input differences with sketch error.

The SQL rounds NOW() to a multiple of T, so windows are grid-aligned, but that doesn't give the arms a shared data origin.

Proposed fix

Once the ClickHouse runner feeds data live (#868), have it do what #841 does for experiment_run_e2e.py:

  1. Start the feed at the next multiple of the LCM of the aligned intervals (aligned_interval_lcm / aligned_start_ms in experiment_utils/services/paced_feeder.py).
  2. Write data_start_time (seconds) into each aligned query group's client_options before the client starts (set_data_start).

The client then evaluates repetition k at data_start_time + (max(1, ceil(starting_delay / T)) + k) * T, the same as for the PromQL runner.

Done when

  • ClickHouse-runner configs with align_to_interval carry data_start_time, and every arm's repetition k covers the same samples.
  • The wall-clock fallback for align_to_interval without data_start_time is removed from the client, and data_start_time becomes required.

Related: #841, #844, #862.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions