Activity

1. milindsrivastava1997 commented on Aug 20, 2026

   

   milindsrivastava1997

   on Aug 20, 2026

   ContributorAuthor

   More actions

   Sliding window manual override — grilling decisions log

   Status: Planning only. Nothing implemented. Blocked on a separate issue
   (window-serving execution support, "Issue B" below) before any code lands.

   Goal: run a tumbling-vs-sliding-window experiment via
   asap-tools/experiments/experiment_run_e2e.py, comparing window types (and
   sizes) at the cost/latency level. Arroyo is out of scope entirely — it's
   deprecated, and the planner's should_use_sliding_window() hardcoded-off
   switch (asap-planner-rs/src/planner/window.rs:11-14) exists only because
   sliding windows used to crash it.

   See [[sliding-window-multi-window-merge-blocker]] for the technical trace of
   why sliding windows aren't usable today beyond the narrowest case, and for
   the cross-reference to issue #405.

   This doc records the design decisions from the /grilling session, for the
   eventual issue descriptions and PR.

   ---

   Decisions

   D1 — Override granularity: global, not per-query-group

   One windowing: setting in config.yaml applies to the whole experiment
   run, same shape as the existing sketch_parameters: block. Not per-query-group
   (unlike t_repeat_ms/step_ms/range_duration_ms, which already support
   per-QueryGroup overrides).

   Why: the experiment sweeps window type/size across whole runs (one hydra
   multirun leg = one window config), not within a single run. Per-query-group
   control has no current use case and would roughly double the plumbing.

   Ideal case (not this work): per-aggregation-id overrides — deferred to
   Issue C below, since it applies equally to sketch_parameters (also
   global-only today) and to windowing.

   D2 — Slide interval specified as a divisor, not absolute ms

   slide_interval_ms = window_size_ms / slide_divisor, computed after each
   query's own window_size_ms is derived (window_size_ms differs per query —
   it's a function of that query's t_repeat_ms/step_ms, not a global
   constant). An absolute slide_interval_ms override wouldn't divide evenly
   into every query's window size in general.

   D3 — Config shape

   windowing:
     type: "tumbling"   # or "sliding" — matches WindowType's existing
                         # #[serde(rename_all = "snake_case")] convention,
                         # same as aggregate_cleanup.policy / CleanupPolicy
     slide_divisor: 4    # only meaningful when type == "sliding"

   Top-level key, mirroring sketch_parameters:'s placement. No new casing
   convention needed — WindowType already deserializes from
   "tumbling"/"sliding" directly.

   D4 — Scope: PromQL + SQL, not ElasticDSL

   • PromQL: ControllerConfig (config/input.rs:11-31) → SingleQueryProcessor
     (planner/promql.rs) → set_window_parameters() (planner/window.rs).
   • SQL: SQLControllerConfig (config/input.rs:141-147) →
     SQLSingleQueryProcessor (planner/sql.rs) → compute_sql_window()
     (structurally a duplicate of set_window_parameters(), same invariants,
     same relaxation, currently hardcoded WindowType::Tumbling at sql.rs:269).
   • Python: generate_controller_client_configs() (PromQL,
     experiment_utils/config.py:340) and generate_sql_planner_input() (SQL,
     config.py:830, already extended once for sketch_parameters by PR fix(tools): pin TZ=UTC across for ClickHouse experiments #514 —
     same pattern applies here).
   • ElasticDSL excluded: ElasticDSLControllerConfig exists in Rust
     (elastic_dsl/generator.rs, elastic_dsl/controller.rs) and is a real
     asap-planner --query-language elastic_querydsl CLI mode, but nothing in
     asap-tools/experiments generates that config — no
     experiment_run_elastic*.py, no Python generator function. Wiring it would
     touch code nothing in the experiment pipeline calls today.

   D5 — Validation: error loudly, never round

   If slide_divisor doesn't evenly divide a query's window_size_ms, that's a
   planner error (ControllerError/Result<_, String>), same style as every
   existing invariant check in window.rs/sql.rs ("must be a multiple of ..." etc.). No silent rounding to a nearby millisecond value.

   D6 — Failure mode: fail-fast across the whole plan

   If any one query in the workload can't legally take a sliding window under
   the chosen global slide_divisor, plan generation aborts entirely — it does
   not silently punt that query to the raw backend (unlike the existing
   enable_punting/should_be_performant path for genuinely unsupported query
   shapes). Rationale: windowing.type: sliding is a deliberate, explicit
   experiment setting, not an automatic optimizer decision — a query silently
   falling back to unaccelerated serving would quietly corrupt the cost/latency
   comparison the experiment exists to make.

   D7 — Sequencing: blocked on execution-side support

   Shipping the override before sliding-window queries can actually be served
   correctly (beyond the single-window-exact-match case) means
   windowing.type: sliding would produce plans that fail at query time for
   almost any real workload query. Building the override is held until that
   lands. See Issue split below.

   D8 — Relationship to issue #405

   #405 ("Optimization-based sketch/streaming config selection") is the tracked
   replacement for automatic planner decision-making generally (sketch type,
   params, window config, query method), via a MIP-based optimizer. It already
   documents Sliding: W < range_a → Infeasible as a deliberate v1 modeling
   choice in its own cost formulation — not just an unimplemented case.

   Per discussion: #405 stays the design/tracking issue for the optimizer
   project as a whole. The windowing work here gets its own issues (below),
   not folded into #405, because #405 is about automatically choosing configs
   via cost optimization, while this work is about manually controlling
   window type/size as an experiment knob — a different, complementary axis.
   Both new issues cross-reference #405 (Issue B extends its feasibility-table
   assumption about sliding; Issue A is the manual stand-in for what #405 would
   eventually choose automatically).

   ---

   Issue split

   Issue A — Manual windowing override (candidate generation / planner side)

   The config plumbing itself: D1–D6 above. windowing: {type, slide_divisor}
   in config.yaml → ControllerConfig/SQLControllerConfig → planner emits
   the right window_type/window_size_ms/slide_interval_ms per aggregation.
   Blocked by Issue B (D7) — the plan this generates isn't safely runnable
   until B lands.

   Definition of done:

   • Config plumbing + validation (D3, D5, D6), PromQL and SQL paths (D4).
   • Unit tests for the plumbing itself (config parses; divisor validation
     errors correctly; fail-fast behavior) — these don't depend on Issue B and
     can be written now.
   • Not required for A's DoD: an end-to-end "sliding query actually
     returns correct data" test — that's Issue B's responsibility, since it's
     the thing that proves execution works, not candidate generation.
     (Resolved during grilling: the split is exactly "generating candidates"
     (A) vs. "actually executing them" (B), so correctness-of-execution testing
     belongs to B.)

   Issue B — Sliding-window query execution (multi-window merge/subtract)

   The query-engine-side work: relax capability_matching.rs's
   window_compatible Sliding rule beyond exact-match, implement merge (for
   mergeable sketch types) / subtract (for subtractable ones) across multiple
   stored sliding-window buckets in simple_engine's query-serving path,
   update cleanup.rs's Sliding branch to be window-count-aware, and resolve
   the epoch/slide-interval alignment question for exact-match lookups. Full
   technical trace in [[sliding-window-multi-window-merge-blocker]]. Extends
   #405's feasibility-table assumption about sliding windows. Needs its own
   /grilling session — deliberately not designed further here.

   Definition of done (tentative, to be refined in its own session):

   • Multi-window sliding queries (data_range_ms a multiple of
     window_size_ms, mirroring tumbling's existing rule) return correct data.
   • End-to-end correctness test: plan → run → query a sliding-window
     aggregation → assert correct results. This is the test Issue A explicitly
     defers to B.

   Issue C — Per-aggregation-id parameter overrides (ideal case)

   Both sketch_parameters (today) and windowing (once A ships) are
   global-only overrides. The ideal is per-aggregation-id control — a detail
   the planner emits, not something users author queries around, but currently
   inaccessible as an override target. Tracks the gap; no design work done on
   it yet.

   ---

   Open items

   • Whether to actually file A/B/C on GitHub now, or keep this doc as the
     record until closer to implementation — pending.
2. milindsrivastava1997

   added a commit that references this issue on Aug 20, 2026

   docs(design): refine sliding-window alignment validation and PR stack

   ...

   1ef34e0
3. milindsrivastava1997

   self-assigned this

   on Aug 23, 2026
4. milindsrivastava1997

   closed this as completedin #643on Aug 28, 2026