rqe-optimizer: let mergeable finer-group deployments serve coarser RAQEs

[milindsrivastava1997]

Closes #187.

A deployment grouped by G_d can now serve a RAQE grouped by a subset G_r when its family merges across groups. The read merges each G_r group's fine states into one (a roll-up), and the planner chooses it only when it is SLA-valid and cheapest. ASAPQuery can't run roll-ups yet; see Interface change.

Design decisions (please approve)

These are the decisions from the design discussion on #187 (Q&A, Q1 change). Please approve each one or push back. Example used throughout: KLL by (service, endpoint) (50 groups) serving p99 by (service) (5 groups), L = 1h, x = 1m.

Scope

• Rate/increase, top-k and Hydra don't roll up:
  • Rate/increase: a coarse increase is the sum of per-series increases. The accumulator's merge joins two pieces of one counter in time, so merging two groups' counters is wrong. For example, A going 100 → 110 and B going 5 → 8 over the same minute merge to 10, not 13. ASAPQuery keys MultipleIncrease per series, so there is no engine bug there.
  • Top-k: heaps drop items that are in the parent's top-k but in no child's.
  • Hydra: it inserts every label subset of its schema, so it answers a coarse grouping directly from the grid rather than by merging fine groups. Hydra's admission design is in docs: Hydra admission design (accuracy, cost, candidates, sketch-bench needs) #193, stacked on this PR; this PR stays roll-up only.
• No feature flag. ASAPQuery pins a sketch-bench rev, and until it supports roll-ups it errors on them (see Interface change). A flag would have to be threaded through about 8 signatures to reach is_eligible.
• Uneven fan-out (one service owning most endpoints) is rqe-optimizer: read accuracy at the largest group's sample count, not the average #189. This PR uses averages.

Q1. Which families roll up? New FamilyProperties::mergeable_across_groups.

Family	Roll up?
exact-sum (Sum, Count), exact-min, exact-max	yes
hll, univmon-cardinality	yes (union)
kll-percall, dd	yes
exact-increase (RateOrIncrease)	no: its merge concatenates one counter in time; a coarse increase needs per-series state and a sum
cms/countsketch heap top-k, univmon-topk	no
hydra-kll (and every Hydra variant)	no: answers subsets of its schema directly

is_eligible: the groupings are equal, or the family is mergeable_across_groups and r.grouping ⊂ d.grouping. A coarse deployment never serves a finer RAQE.

Q2. Candidate windows. For families that roll up, a grouping's windows, slides and heaps come from every RAQE with the same capability, metric and filter whose grouping is a subset of it. That way a fine x can divide a coarse RAQE's L. Other families keep their exact-grouping inputs, so top-k candidates don't grow. No new groupings are invented.

Q3. Cost of a roll-up read. Each merge folds two states into one. A query reads card(G_d)·L/x states and ends with card(G_r):

merges = card(G_d) · L/x − card(G_r)

Phase	Before	Now
merge CPU	card(G)·(L/x − 1)·c_mrg / T	merges · c_mrg / T
merge memory	card(G)·m(L)	card(G_r)·m(L)
query CPU / output	card(G)·…	card(G_r)·…
latency	card(G)·c_qry + card(G)·(L/x−1)·c_mrg	card(G_r)·c_qry + merges·c_mrg
ingest, storage	G_d	unchanged

• With G_r = G_d, each row reduces to the old formula.
• A direct (L == x) roll-up still merges card(G_d) − card(G_r) states.
• DD's accumulator is sized from a coarse group's values.
• Assumption: a merge across groups costs the measured c_mrg.

Q4. Accuracy of a roll-up. It reads data_shape[G_r] and N = λ·L / card(G_r). KLL and univmon-cardinality merge lossily (#131; UnivMon rebuilds its heavy-hitter heaps from the union on merge), so they read their merge curves at ⌈card(G_d)/card(G_r)⌉ · L/x merged instances. That's the average fan-out; the worst group's is in #189. Past the measured merge counts, KLL takes its guarantee. univmon-cardinality has no guarantee, so it has no accuracy there; it isn't in the study or DEPLOYABLE_FAMILIES today, so nothing changes in practice.

Q5. Plan output. No new field. A RAQE rolls up exactly when its grouping differs from its deployment's.

Q6. Fact validation. validate_facts requires card(X) ≤ card(Y) for every pair X ⊂ Y among the label sets a plan can use: every RAQE grouping on the metric, plus its full label set. Label sets no RAQE groups by are not checked. This replaces the "more groups than series" check. Otherwise bad facts would make the merge count negative, and the MILP would treat a roll-up as a discount.

Interface change (ASAPQuery)

• Plans may now contain roll-ups: a RAQE whose grouping_labels differs from its deployment's. Until ASAPQuery can merge fine groups into coarse ones at read time, it should error on such a plan when it bumps the pinned rev.
• validate_facts rejects more inputs: a RAQE grouping with more groups than another RAQE grouping, or than the full label set, that is a superset of it.
• The MILP planner must refuse roll-ups until the engine reads them: MILP planner writes a roll-up plan at the first RAQE's grouping ASAPQuery#813 / #814.

Tests

• tests/mixed_grouping.rs (new, end to end):
  • two quantile RAQEs by (service, endpoint) and (service) share one fine KLL deployment;
  • top-k at the same groupings keeps one deployment per grouping.
  • It was written first and failed before the change.
• candidates.rs:
  • roll-ups go only to a subset grouping and only for mergeable families;
  • top-k and rate serve only their own grouping;
  • fine KLL candidates get a coarse RAQE's 90-minute window, top-k candidates don't.
• analytical_cost_model.rs: roll-up merge count, accumulators and latency, including a direct roll-up.
• saturation.rs: KLL roll-up reads the coarse shape, coarse N and the fan-out × windows merge curve.
• lib.rs: subset-cardinality validation.
• candidates.rs (pruning): retains_candidate_with_cheaper_compaction and prunes_the_candidate_with_larger_query_output.
• analytical_cost_model.rs: a_dd_roll_up_sizes_its_accumulator_at_the_coarse_grouping.
• saturation.rs: univmon_cardinality_merges_lossily_hll_and_dd_do_not.
• lib.rs: rejects_a_used_label_set_with_more_groups_than_a_used_superset, rejects_a_grouping_with_more_groups_than_series and ignores_label_sets_no_raqe_uses. These replace the old rejects_more_groups_than_series.
• Each rule's test fails when the rule is removed. The other existing tests use one grouping per RAQE and pass unchanged.

cargo test (104 + 2), clippy --workspace -D warnings and fmt all pass, rebased on main (#192).

examples/workload_scenarios.rs against main: the only change is the mixed-grouping sweep's "both" case (p99 by (service) plus p99 by (service, endpoint)), which now runs on 1 DDSketch deployment instead of 2:

• CPU drops from 9.25e-5 to 8.47e-5 (−8.4%);
• memory drops from 7.5 MB to 6.9 MB;
• the by-service query's latency rises from 0.16 ms to 1.6 ms (about 10 endpoints merged per service).

Every other sweep is byte-identical.

Also in this PR (found in review)

• Dominance pruning compares every cost that cost by use bills. It used to compare only ingest, query latency, merge memory and storage. It now also compares compaction, per window (each window starts a chain) and per slide (CPU and byte-seconds), and query memory, merge plus output. The old pruning could drop a candidate that was cheaper by use, so plans can change even without a roll-up, and only toward lower cost.
• validate_facts checks group counts only among the label sets a plan can use: every RAQE grouping on the metric, plus its full label set.
• Docs. The optimizer docs are split into rqe_optimizer_candidates.md and rqe_optimizer_cost_model.md, with one cost table covering every phase, roll-ups included. They now say why rate/increase, top-k and Hydra don't roll up.

Open questions

1. Cardinality's counted key. A union of fine HLLs equals the coarse answer only if the counted key doesn't depend on the grouping (series identity or the value). If the key is the non-grouping labels, a roll-up undercounts. Raqe doesn't record the counted key yet.
2. AutoSketch-vs-ASAP eval: decided to keep roll-ups on. Its synthetic and trace tables currently model every stream at a single grouping, so roll-ups never trigger there.

examples/small_problem.rs adds latency_p99_1h_by_service. With --milp --saturation-dir study_final030k, it rolls up onto the per-endpoint DD deployment:

latency_p99_1h_by_service -> deployment 1 merged_instances=12 query_latency=3.179e-1 ms rolls up {"endpoint", "service"} -> {"service"}

🤖 Generated with Claude Code

[milindsrivastava1997]

@zzylol These are part of the old cost model. We should check these against the new cost model and merge tables?

[zzylol]

Yes we should update and merge

[zzylol]

Merged in 332ed6a. Cost by use §3 is now the one cost table: ingest, compaction, query job (merge, then estimate) and storage, each with CPU per event, mean vCPUs, memory, and how long the memory is held. It is written in terms of I_d (instances per window, card(G_d) or 1 for a shared sketch) and I_r (states a query ends with, card(G_r) or 1), summed over parts (sketch + key tracker).

The old snapshot table is gone. The snapshot-AUC section (milp::minimize, still supported) now just lists how it accounts §3's table differently: k_D = 1 so no compaction; query memory held all the time; per-RAQE serial latency against latency_sla_ms instead of batch latency. The DD m definition stayed there and now distinguishes card(G_d) (window) from card(G_r) (merge accumulator).

[milindsrivastava1997]

@zzylol Did you ever review this section? This was present earlier. I believe this should help with Hydra's cost model

[zzylol]

Let me review.

[zzylol]

Reviewed. The shape/law part is right, and it is in fact the Hydra skeleton: Shared + Fixed is I_d = 1. Changes in 332ed6a:

• the section uses I_d / card(G_d) (and card(G_r) for probes), matching §3;
• the side-by-side table no longer repeats every phase (its latency row was the old serial G·(c_qry + (n−1)·c_mrg) with no compaction, and its merge rows ignored roll-ups). It now gives what each family puts into §3's table: m/c_mrg basis, I_d, I_r, probes, key tracker, whether it rolls up;
• the HydraKLL column is marked as in the code but not in the evaluation, pointing to the Hydra section, since the subpopulations ceiling here conflicts with the Hydra section saying one ceiling is insufficient.

One thing I didn't change: KLL is still treated as Fixed at the 1e6-values size. Worth confirming that still matches the #174 study.

[zzylol]

Checked the KLL point myself (3060337). The old text was wrong in both directions:

• The cost table's KLL m is not "the size measured at 1e6 values". It is sketch-bench's nominal footprint kll_footprint = 4·k·sizeof(T): 1600 B at k = 50, 6400 B at k = 200. approxbench reports the same bytes at 1e3, 1e6 and 1e8 items (run on clnode109 with the final030 binary). So "Fixed" is exactly what the model gets, and nothing grows with n.
• It is also not what asap_sketchlib::KLL allocates: the lib preallocates max capacity over 61 levels (aqpbm-planeval's kll_max_capacity replica), which is 4640 B at k = 50 (2.9×), 8032 B at k = 200 (1.25×) and 22240 B at k = 800 (0.87×).

The doc now says this. Whether the cost table should switch KLL's footprint to the allocated capacity is a sketch-bench change outside this PR; it would raise small-k KLL memory about 3×.

[milindsrivastava1997]

@milindsrivastava1997 : This should be merged with the above tables. We should just have one cost model table with all phases and CPU and memory costs both.

[zzylol]

Merged in 332ed6a: the proposed Hydra model is now §3's table with I_d = I_r = 1 and c_qry → c_probe,h(G_r), shown as an instance rather than a separate formula. Differences from the old Hydra table:

• added compaction (k_D > 1 workers produce k_D partial grids: (k_D − 1)·c_mrg,h, holding k_D·m_h) and k_D copies of open windows at ingest;
• symbols follow the doc: lookback S_i (not L, which is the SLA), card(G_r)/card(G_d) (not C_r/C_d), no ceil on storage;
• the key tracker is a part, so its cost adds in every phase, as the text already required.
  The anchor #proposed-hydra-cost-model is unchanged, so the link from the candidates doc still works.

[zzylol]

Also fixed in 332ed6a: the cost-by-use §3/§4 formulas were out of sync with this PR's roll-ups. The code already prices them (usage.rs calls query_latency_ms and merge(...).memory_bytes, which use merge_count and G_r), but the doc still had ℓ = card(G)·c_qry + Σ inst·(n−1)·c_mrg and the merge accumulators at card(G). They now read:

• ℓ_{i,D} = card(G_r)·c_qry + (I_d·n − I_r)·c_mrg
• q_{i,D} = I_r·m(S_i)·[merges > 0] + card(G_r)·output bytes

Ingest, compaction and storage stay at G_d. With G_r = G_d both reduce to the old formulas. A direct roll-up (n = 1) still merges I_d − I_r times. §1 now defines G_d, G_r, I_d, I_r, a_D, k_D; ingest memory is renamed ingest_D so it doesn't clash with I_d. Doc-only change; no code touched.

🤖 Generated with Claude Code

[zzylol]

Two review rounds (code, docs, Hydra), fixes pushed in 81c46b7 (code) and 00fd9a9 (docs). cargo test -p rqe-optimizer (99 + 2), clippy -D warnings and fmt pass.

Code

• Dominance pruning now compares everything cost by use bills. Before, it skipped compaction (per window in every chain, and per slide in CPU and byte-seconds) and query output memory, which differs between top-k configs answering different k. A direct RAQE never merges at query time, so a candidate with cheaper ingest but costlier merges could prune one that is cheaper by use. The new test retains_candidate_with_cheaper_compaction fails without the fix.
• Stale comments fixed: workload_scenarios's module doc, the exact-increase reason, merged_instance_count, and the §N citations, which now name the docs.

Docs

• rqe_optimizer_candidates.md:

  • eligibility allows subset groupings;
  • generation shows the roll-up RAQE sets;
  • accuracy is read at G_r, with KLL's fan-out merge count;
  • past the measured merge counts, the guarantee applies, floored by the last measurement (the old text said "no accuracy");
  • pruning lists every compared cost;
  • the MILP lives only in the cost doc.

• rqe_optimizer_cost_model.md: top-k k scaling, k_D over parts, mergeable_across_groups, and KLL's footprint (kll-percall (lib) reports a nominal 4·k footprint, not asap_sketchlib's allocation #191/kll: report asap_sketchlib's allocation as the lib rows' footprint #192).

• rqe_sketch_deployment_v1.md and rqe_optimizer_TODO.md: phases and TODOs now match cost by use.

• New rqe_optimizer_hydra.md replaces both Hydra sections:

  • Hydra as sketch-bench runs it;
  • accuracy bounds per combination (CMS, CountSketch, HLL, UnivMon, KLL), checked against the paper's Theorem 2 and the library;
  • cost model as §3's table with I_d = I_r = 1;
  • candidates and MILP edges (MILP unchanged, still linear);
  • 10 sketch-bench requirements and 6 open decisions.

  In short: no combination has a planning-grade closed-form bound for every group. Contamination is additive in the global fanned-out mass, inner errors are correlated across rows (cells share seeds), and the canonical grids have 3 rows (univmon 2). So admission has to be empirical.

Open design questions (not changed in code)

1. HLL / univmon-cardinality roll-up: a union of fine HLLs equals the coarse answer only if the counted key doesn't depend on the grouping. The optimizer doesn't define the counted key. If it is the non-grouping labels, roll-ups undercount.
2. univmon-cardinality merges are lossy (heaps are rebuilt), but it isn't in merges_lossily. Roll-ups multiply the merge count.
3. autosketch_vs_asap can now plan roll-ups (a {g} deployment serving a {} RQE on the same stream) that PerQuery, AutoSketch and ASAPQuery can't run. Should the eval disable roll-ups?
4. validate_facts now checks every pair of label sets, including ones no RAQE uses, so it can reject facts that passed before.
5. exact-increase: summing per-series increases is exact, so it could roll up if ASAPQuery's accumulator keeps per-series increases. Can you confirm?

🤖 Generated with Claude Code

[zzylol]

Decision on open question 3 (from @zzylol): the AutoSketch-vs-ASAP eval keeps roll-ups on. ASAP may serve an ungrouped RQE from a grouped deployment on the same stream; that sharing is part of what the planner offers. The results on main (#138) were planned before roll-ups, so they will be rerun once this PR lands.

🤖 Generated with Claude Code