Audit ASAPQuery precompute-engine changes for backend parity #532

Activity

1. milindsrivastava1997

   added

   meta: trackingUmbrella issue coordinating multiple implementation gapsUmbrella issue coordinating multiple implementation gaps

   area: data planeRuntime ingestion, storage, and query-serving data pathsRuntime ingestion, storage, and query-serving data paths

   on Sep 9, 2026
2. milindsrivastava1997 commented on Sep 9, 2026

   

   milindsrivastava1997

   on Sep 9, 2026

   Last edited by milindsrivastava1997

   ContributorAuthor

   More actions

   Parity audit (2026-09-09)

   Compared the listed ASAPQuery PRs against backend main at d5dc72f. This is a static source audit; no repository changes were made.

   Already present or represented by the current backend contract

   • feat(sketch_algebra): unify SketchKind onto asap_sketch::SummaryKind #400: idle wall-clock closure plus shutdown force-close.
   • backend-plan: construct BackendPlan from BackendStageConfig (Phase 1) #437: millisecond window/query handling; ingest units are explicit.
   • fix(precompute): separate event and closure watermarks #483: typed sketch dimensions and explicit heap weighting replace legacy parameter parsing.
   • Add a shared authoritative SummaryCatalog #536: pane last-touch prevents wall-clock closure while ingest remains active.
   • Reference the shared SummaryCatalog from execution plans #537/Bind physical-plan execution to the authoritative SummaryCatalog #538: deterministic fake-clock regression coverage exists at worker level.
   • Add offline cluster data exporter tool #704: PromQL (start, end] behavior is present via promql_right_closed: state identity carries the convention and the worker assigns boundary samples accordingly.
   • Lookback vs PromQL range selector: redundant window concepts (+ /query_range readout bug) #706: temporal Top-K and explicit heap weighting are planned and tested.
   • Retire the collector-facing emission pipeline #719: heap weighting is represented as parameters.weight_mode, rather than upstream subtype refactor.

   Intentionally superseded / outside the backend contract

   • chore(promql_utilities): relocate Statistic/KeyByLabelNames to asap_types #397: accumulator construction uses typed AccumulatorSpec rather than the legacy factory contract.
   • fix: acknowledge maintenance outputs after publication #605 and Compile actual finite multi-source maintenance plans #668: SetAggregator/DeltaSetAggregator were retired in backend refactor(types): retire SetAggregator + DeltaSetAggregator family #264. They were a never-wired exact key-membership route for count(distinct ...); HLL is currently the sole supported approximate cardinality answerer. No key-only accumulator remains.
   • Do not silently select raw fallback when logical candidate costs are unavailable #721: legacy Flink/Arroyo/Kafka infrastructure is not part of this backend supported precompute path.

   Required follow-up work

   1. Introduce WindowBoundaryType as the authoritative four-case contract: OpenOpen (start,end), OpenClosed (start,end], ClosedOpen [start,end), and ClosedClosed [start,end].

      • Migrate promql_right_closed: true to OpenClosed while preserving state-identity separation from existing half-open states.
      • Define tumbling/sliding, timestamp-zero, watermark-closure, and boundary-sample behavior; test every variant.

   2. Port Execute completed cohorts at every materialized maintenance frontier #665/Remove test-only query planning and unused backend APIs and dependencies #717 count semantics for raw precompute. Bare CMS and MultipleSum currently add sample values. Add an explicit count-vs-value contract so count mode adds one per observation, as heap Top-K already does through weight_mode. THIS IS DONE.

   3. Port Remove the unreferenced legacy planning endpoints #715 fail-fast configuration validation. Invalid HLL precision and invalid Min/Max subtype must be rejected at plan/config installation, rather than falling through to a warning/default accumulator path.

   4. Port refactor: split the MinMax aggregation family into Min and Max #716 ingest-throughput diagnostics. samples_ingested is maintained but not periodically reported; add a bounded periodic debug metric/log with a testable interval or metric surface.

   5. Re-evaluate whether SetAggregator and DeltaSetAggregator should return. Decide whether the system needs exact, enumerable key membership / exact distinct semantics. If yes, define the supported query path, state model, chronological-merge contract for deltas, persistence/late-data semantics, and tests before restoring either accumulator.

   6. Port docs(control_plane): design doc for BackendPlan wire format #389 before supporting SQL Top-K. Add SQL parsing/planning/execution for GROUP BY with COUNT(...), descending aggregate order, and LIMIT k, backed by a heap-bearing frequency sketch. This is separate from PromQL/query-plan Top-K.

   7. Port SummaryExecutor: permanent self-exclusions + upstream composition gaps leave some PromQL shapes always on the legacy/archive path #436 before supporting SQL materializations. Define a typed source-column-to-ingest-value contract so a SQL materialization uses its declared value_column rather than always using the wire scalar.

   No follow-up issues have been filed by this audit.
3. milindsrivastava1997 commented on Sep 10, 2026

   

   milindsrivastava1997

   on Sep 10, 2026

   ContributorAuthor

   More actions

   Audit update (2026-09-10)

   Follow-up item 4 is complete via #578: backend ingest throughput is now exposed as periodic diagnostics and Prometheus metrics.

   The merged change records monotonic accepted-sample, processed-update, and materialized-output counters; the 30-second log derives elapsed-time rates without resetting shared state. Regression coverage verifies non-destructive observations, correct interval deltas, and ticker cancellation.