Skip to content

eval: e2e 10/s plans with one-scrape instant RQEs and one RQE per query - #217

Merged
milindsrivastava1997 merged 1 commit into
mainfrom
eval/e2e-rate10-regen
Oct 14, 2026
Merged

milindsrivastava1997 merged 1 commit into
mainfrom
eval/e2e-rate10-regen

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Summary

Regenerates rqe-optimizer/results/e2e-rate10/ (classic, multigroup and all at shared = 1, 10 samples/s) on main. That picks up #213's cost table and curves, #215's one-scrape instant RQEs, and #216's one RQE per schema query. These are the plans for the end-to-end run.

  • Tables: export_autosketch_eval_table.py --synthetic --samples-per-sec 10, without --keep-duplicate-queries.
  • Results: --runs 3 --no-chosen. Plans: --runs 1, then extract_autosketch_plans.py.
  • README: RQE counts, costs, classic deployments, the change from the previous 10/s plans, ASAPQuery conversion, and Reproduce.

RQEs

Workload RQEs at 200/s (§6.3) Duplicate queries Below the curves RQEs at 10/s, all served
classic 42 0 0 42
multigroup 62 10 2 50
all 104 10 2 92

Every chosen accuracy is measured, and no Hydra grid is chosen. Every e2e query's RQEs are served by one deployment in every plan.

Cost (cheapest plan, ASAP, CPU weights), previous 10/s plans → now

Workload CPU (vCPU) Memory (GiB) Fargate ($/h) Deployments Rolled up
classic 0.309 → 0.318 15.3 → 10.7 0.0787 → 0.0585 6 → 7 0 → 0
multigroup 0.0950 → 0.0911 1.34 → 1.33 0.00872 → 0.00846 9 → 9 41 → 33
all 0.404 → 0.409 16.7 → 12.0 0.0874 → 0.0670 15 → 16 41 → 33
  • Memory falls ~30% and Fargate ~25%. With study: denser KLL/DDSketch grid and Hydra W 65536; §6.3 rerun #213's denser grid, plans pick DDSketch α = 0.1.
  • CPU rises ~3% on classic and all. sum by (job) (sum_over_time) no longer shares the instant sum by (job) deployment, which is now a 100 ms window.
  • The tightest feasible latency bounds are unchanged: classic 21 ms, multigroup and all 490 ms.

ASAPQuery

--planner saved needs --data-ingestion-interval-ms 100 on these tables, the scrape interval. With ASAPQuery main (ProjectASAP/ASAPQuery#866), every ASAP plan converts: classic PromQL 12/12, multigroup SQL 8/8, all SQL 8/8 and PromQL 8/8.

Verification

  • An independent re-plan in another session gave identical tables (classic, all, multigroup) and identical plans in all 120 (method, weights, kind, SLA) combinations: classic 48, all 36, multigroup 36.
  • Runs were on a 16-core desktop, not an idle cluster node, so planning_secs are indicative only.

🤖 Generated with Claude Code

Regenerates rqe-optimizer/results/e2e-rate10 (classic, all, multigroup at
shared = 1, 10 samples/s) on main: the denser grid's cost table and curves
(#213), instant RQEs reading one scrape (#215), and one RQE per schema query
(#216). Tables, --runs 3 results and --runs 1 plans; the README gives the
RQE counts, costs, classic deployments and ASAPQuery's
--data-ingestion-interval-ms 100.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997
milindsrivastava1997 merged commit a4de956 into main Oct 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant