Skip to content

feat(planner): emit streaming and inference configs from the MILP plan - #803

Merged
milindsrivastava1997 merged 2 commits into
mainfrom
790-a3-milp-plan-to-configs
Oct 7, 2026
Merged

milindsrivastava1997 merged 2 commits into
mainfrom
790-a3-milp-plan-to-configs

Conversation

@milindsrivastava1997

@milindsrivastava1997 milindsrivastava1997 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Implements A3 (plan → StreamingConfig / InferenceConfig) of #790 (session 3a). It builds on #801, which has the S8 plan output and the capability split.

What

  • New optimizer/milp_output.rs: plan_to_planner_output(config, workload, solution) -> PlannerOutput produces streaming and inference YAML in asap-planner's format. rollup_labels is left empty (the legacy planner fills it; the engine doesn't read it). It reuses the PromQL generator's build_streaming_yaml / build_inference_yaml.
  • asap-optimizer-cli --milp:
    • --output-dir <dir> writes streaming_config.yaml and inference_config.yaml.
    • --allow-undeployable-families passes true to build_all_candidates. It prints the plan, writes no configs, and can't be combined with --output-dir.
  • solve_milp takes allow_undeployable_families.

Mapping

sketch-bench variant (capability) Aggregation sub_type parameters
exact-sum (Sum / Count) MultipleSum sum / count —
exact-min / exact-max MultipleMinMax min / max —
exact-increase MultipleIncrease — —
kll-percall DatasketchesKLL — K = k
hll HLL — precision = lg_k
cms-heap-topk-fastpath-vector2d (TopKByValue / TopKByCount) CountMinSketchWithHeap sum / count depth = rows, width = cols, heapsize = largest k among served queries
  • Grouping and aggregated labels come from set_subpopulation_labels, the same split the legacy planner uses.
  • A deployment is tumbling if window == slide, otherwise sliding.
  • Aggregation ids start at 1 and follow the plan's deployment order.

Design decisions

  1. Unmapped variants are errors, including hydra-kll (not deployed: its DeltaSet key tracker has no ASAPQuery pairing). No DeltaSet pairing is built.
  2. avg queries are errors (A3b) when writing configs. --output-dir rejects them before the solve; print-only runs still plan them. The avg rewrite plans sum and count, but the user's avg query would get no query_config, and the engine has no Avg (Support Avg (and other multi-statistic aggregations) in query engine #463).
  3. A query string served by two deployments is an error. This can happen when the same leaf appears at two cadences or SLAs. One query_config can name only one aggregation (related: Query engine cannot distinguish occurrences of the same query string with different requirements #778).
  4. heapsize = the largest k, without the legacy × 4 multiplier. sketch-bench doesn't model heap size: cost rows use a fixed top-k = 32 wrapper.
  5. Cleanup: NoCleanup, the same as the greedy translator. Under NoCleanup the YAML carries no retention, so retained_instance_count and merged_instance_count aren't emitted yet and retention is unbounded. Moving to ReadBased is tracked in planner: MILP plan → InferenceConfig should use ReadBased cleanup instead of NoCleanup #800.
  6. DDSketch is not mapped yet. sketch-bench doesn't list dd as deployable; the follow-up, including value-range facts and the accuracy metric, is planner: deploy DDSketch from MILP plans #802.

Known risk

Count is served by MultipleSum with sub-type count. That works because each query_config names its aggregation id. The engine's fallback matching (compatible_agg_types(Count)) doesn't list MultipleSum, so a Count query without a query_config wouldn't find it. No end-to-end engine run yet.

Interface for 4a (A4+A5)

  • plan_to_planner_output is the plan → configs entry point. Its PlannerOutput has to_streaming_yaml_string / to_inference_yaml_string.
  • promql::generator::{build_streaming_yaml, build_inference_yaml} are now pub(crate).

Test plan

  • 8 new tests in milp_output.rs:
    • Full round trip: generated YAML → InferenceConfig / StreamingConfig as the engine parses it, checking type, sub_type, params, labels and no retention for sum, count, quantile, and value- and count-ranked topk.
    • Shared heap sized for the largest k.
    • Sliding window type.
    • Errors: avg query, hydra-kll, query split across deployments.
    • avg detection inside binary queries.
    • Parsing topk's k.
  • cargo test -p asap_planner, clippy with -D warnings, fmt; pre-commit ran workspace check, clippy and test.
  • CLI smoke run on sketch-bench/out_10_6_26_1342/rqe_atomic_costs.json (scrape 15s; sum, count, rate, quantile, both topk kinds, max[1h]) wrote 7 aggregations as in the table. --allow-undeployable-families prints the plan and is rejected with --output-dir.

🤖 Generated with Claude Code

milindsrivastava1997 and others added 2 commits October 7, 2026 10:59
plan_to_planner_output maps each planned deployment's sketch-bench variant
to an ASAPQuery aggregation and writes the planner's streaming and
inference YAML: exact-sum (sum/count) -> MultipleSum, exact-min/max ->
MultipleMinMax, exact-increase -> MultipleIncrease, kll-percall ->
DatasketchesKLL, hll -> HLL, cms-heap -> CountMinSketchWithHeap with
heapsize = the largest k it serves. Any other variant (e.g. hydra-kll),
avg queries, and a query split across deployments are errors. Cleanup is
NoCleanup.

asap-optimizer-cli --milp gains --output-dir and
--allow-undeployable-families (prints the plan, writes no configs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- asap-optimizer-cli --output-dir rejects avg queries before the MILP
  solve; print-only runs still plan them.
- Both YAML strings are serialized before either file is written.
- Each item is visited once per deployment, not once per occurrence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997
milindsrivastava1997 merged commit d54bcea into main Oct 7, 2026
8 of 9 checks passed
@milindsrivastava1997
milindsrivastava1997 deleted the 790-a3-milp-plan-to-configs branch October 7, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant