Skip to content

docs: propose workload-wide planning, summary sharing, and materialization - #509

Merged
zzylol merged 56 commits into
mainfrom
docs/planner-layering-contract
Oct 2, 2026
Merged

docs: propose workload-wide planning, summary sharing, and materialization#509
zzylol merged 56 commits into
mainfrom
docs/planner-layering-contract

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Propose ASAPPlanner's planning stages, from language-specific frontends through logical and physical optimization to selection of one physical plan for the whole workload. The deployment supplies empirical cost and accuracy models and execution capabilities, then executes the selected plan.

Add docs/design_docs/proposals/planner-layering.md and link it from the proposals index.

Before this PR

The planning boundary needs an explicit contract for workload-wide candidates, summary sharing, materialization, and plan selection.

After this PR

  • Frontends produce CandidateLogicalDAGs; logical optimization produces CandidateLogicalASAPDAGs; physical optimization produces CandidatePhysicalASAPDAGs; selection returns one PhysicalASAPDAG.
  • Logical optimization generates local exact and summary alternatives, then applies ASAP-aware common-subexpression elimination across computations and windows.
  • Physical optimization explores materialization, retention and operator implementations. Outputs may be maintained at ingestion time, retained at query time, or rebuilt without materialization.
  • Selection evaluates complete workload candidates using deployment cost and accuracy models, rejects candidates that fail requirements or capabilities, and chooses the cheapest remaining plan.

Four worked examples cover candidate growth (3 local, 54 logical and 156 physical alternatives), UnivMon sharing across three computations, KLL sharing across windows, and ingestion/query-time materialization choices.

Scope and validation

Documentation-only proposal; no runtime or API changes. Deployment-input types, summary subtract/delete design, and physical parallelism, partitioning and resource management remain TODO.

The branch's file contents have been restored to commit 8c29c2e376800ad0823d4917e121eed3b60ed3ab and verified identical with git diff. No runtime tests run.

zzylol and others added 3 commits September 30, 2026 17:59
@zzylol @claude
docs: define the Planner and deployment layering contract
b400e54 
Add the layering proposal, its proposals-index entry, and an Output layers
section in the input/output/workflow design. Every Planner layer outputs all
legal candidates; the deployment keeps every summary-family candidate and
selects with its own costs. Design DAG names are marked as the target API with
the current main type alongside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: treat source binding as a PhysicalDAG execution step
f1c13d4 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: place summary materialization under physical planning
2bc772f 
Split physical planning into physical design (summary materialization,
workload-level, like materialized-view selection) and physical
implementation (per-node lowering and cutting). The annotated graph stays a
Post-ASAP DAG; materialization is decided in design and realized by the cut.
State node-by-node correspondence and the Fallback exception that the
operator-flattening proposal removes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Sep 30, 2026
Closed
zzylol and others added 5 commits September 30, 2026 18:27
@zzylol @claude
docs: point per-operator export at #481
6702d3a 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: state family retention and timing-sensitive lowering plainly
1bb8ec5 
Describe current Planner behaviour (no family pruning) and the backend gap,
and replace the Binary 'exception' with the general rule that a timing-
sensitive node needs one compilation per distinct timing, with an example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: drop Binary lowering detail from the layering proposal
1e84c15 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: drop mixed-placement and compatibility sections from the layeri…
cd6a18d 
…ng proposal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: drop the implementation-status section from the layering proposal
aa0b959 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Selvomega
Selvomega self-requested a review September 30, 2026 18:36
@zzylol @claude
docs: leave input-output-workflow unchanged in the layering PR
62ae3e7 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
Selvomega requested changes Sep 30, 2026
View reviewed changes
Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated
Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
@zzylol @claude
docs: move selection into ASAPPlanner; deployment supplies prices
3607270 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
Selvomega requested changes Sep 30, 2026
View reviewed changes
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
@zzylol @claude
docs: address review of layering proposal
dc275fd 
Planner takes the query and data workloads plus the deployment's cost
model, accuracy requirements and capabilities, and returns one optimal
PhysicalDAG; candidate sets stay internal. Name the annotated stage
MaterializedPostASAPDAG, tabulate what each DAG encodes, fold timing and
selection into the layer descriptions, rename physical design to summary
materialization, and make the example trace one query through every DAG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
Selvomega requested changes Sep 30, 2026
View reviewed changes
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
@zzylol @claude
docs: name layer 2 summary lifecycle planning; drop deployment rankin…
a095ce5 
…g row

A lifecycle fixes materialization, timing, maintenance, retention and
window framework together, so the stage and its DAG are named after the
lifecycle (LifecyclePostASAPDAG) rather than one of those aspects. The
deployment row no longer mentions ranking or selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
@zzylol @claude
docs: match the DAG API docs to the implemented collection boundaries
25d54e9 
Describe the two plain candidate collections, the view-based compile, the
caller-typed physical candidate errors, lifecycle_guarantee, and
PhysicalExecution as an execution handle. Remove APIs the docs said were
removed but never existed (compile_timed_candidates, PhysicalDAGCandidate),
the agent instructions in the alignment proposal's baseline, and the
ingestion-time Binary exception from design docs, where it is an
implementation detail (the developer migration guide keeps it).

Restore #485's statement that candidates do not choose placement and #508's
CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the
split test table in physical-planning-and-deployment.md, and take
planner-backend-layering.md verbatim from #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
@zzylol @claude
docs: sync the layering proposal with #509
7fde378 
Copies docs/design_docs/proposals/planner-backend-layering.md from #509's
head 3607270 so this file merges cleanly after #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
@zzylol @claude
refactor!: merge the lifecycle plan into LifecyclePostASAPDAG
c8ff36b 
The layering proposal (#509) names a single annotated DAG,
LifecyclePostASAPDAG: a PostASAPDAG plus its lifecycle assignment. The
code kept a SummaryMaintenanceLifecyclePlan beside the DAG and named the
timed collection CandidatePostASAPDAGsWithTiming.

- Rename SummaryMaintenanceLifecyclePlan to LifecyclePostASAPDAG and its
  error to LifecyclePostASAPDAGError. The type already held the root and
  each state's lifecycle, retention and window framework; per-node timing
  stays derived by execution_assignment as the existing
  PostASAPDAGAssignment overlay, so no timed graph is stored beside it and
  no new type is added.
- Rename CandidatePostASAPDAGsWithTiming to CandidateLifecyclePostASAPDAGs;
  CandidatePostASAPDAGs stays the logical collection.
- Call layer 2 "summary lifecycle planning" and reword comments that had
  the deployment choose or rank; selection is Planner's, over the
  deployment's cost model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
@zzylol @claude
docs: make selection Planner's, over the deployment's cost model
e7453ee 
Several docs still said the deployment or "downstream" selects, ranks or
chooses the plan, and the glossary said physical plans are owned by
downstream systems. Per the layering proposal (#509), Planner compiles
and selects the physical plan using the deployment's cost model; the
deployment supplies prices and executes the selected plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
@zzylol @claude
docs: copy the layering proposal from #509 byte for byte
c3f10eb 
#509 carries the proposal; #480's copy now matches its head a095ce5
exactly, so merging either PR first leaves no conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Sep 30, 2026
Closed
@zzylol
zzylol requested a review from Selvomega September 30, 2026 20:37
Selvomega
Selvomega previously approved these changes Sep 30, 2026
View reviewed changes
Comment thread docs/design_docs/proposals/planner-layering.md Outdated
@zzylol @claude
docs: rename layering proposal to planner-layering.md
a3c6dbc 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
@zzylol @claude
docs: follow #509's rename to planner-layering.md
67d01c9 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: name the DAG stages LogicalPostASAPDAG and PhysicalPostASAPDAG
41076ca 
Every stage after the frontend is a Post-ASAP DAG; the prefix says how
far planning has gone: Logical, Lifecycle, Physical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 4, 2026
Draft
Draft
Draft
zzylol added a commit that referenced this pull request Oct 4, 2026
@zzylol @claude
feat(dag-viewer): add three-lane Stages view for asap-stage-pipeline/v1
d494595 
Load an asap-stage-pipeline/v1 document (file picker or ?doc=<path>) and
show Logical, Logical ASAP and Physical ASAP lanes side by side, with
candidate switchers, per-node timing and cost, a cost-ranked list of
physical candidates, and the stage-3 selection and rejection reasons.
Selecting a physical candidate shows its from_logical candidate in lane 2.
Pre/Post-ASAP stays the default unless a stage document is loaded.

stages.js holds the DOM-free validation, lane construction and ranking,
tested headless against a hand-written #509 Example 1 Q2 fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 4, 2026
@zzylol @claude
feat(dag-viewer): move the Stages view to stage-pipeline contract v2
812f572 
Contract v2 (Q28-Q30) gives each DAG one root per workload query and
moves cost out of Stage 2 into stage3_selection.costs.

- Read `roots` (still accepting a single `root`); mark every root and
  label it with its query id.
- Take node badges, totals and ranking only from Stage 3 costs, labelled
  as a Stage 3 result; without Stage 3 the physical lane has no cost.
- Load partial documents; missing later stages render "not produced".
- Require every Stage 2 candidate to be selected or rejected, and show
  valid-but-costlier apart from invalid.
- Show optional per-query requirements in the query list and on roots.
- Rewrite the sample as #509 Example 1 with both queries, and label
  sort/limit physical operators.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 4, 2026
@zzylol @claude
feat(dag-viewer): say when a stage document carries only some plans
ed7522e 
stage_pipeline now prices every plan and writes the cheapest
--max-candidates (#613); the Stages view says "showing N of M plans" from
the document's shown_of section. The README shows how to generate all six
#509 examples into an ignored out/ directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 4, 2026
@zzylol @claude
feat(dag-viewer): redesign the viewer around the stage pipeline
7516caa 
Pre/post-ASAP DAGs no longer exist after #509/#572, so the viewer now
reads only asap-stage-pipeline/v1 documents and uses the layout of the
published Stage Viewer: example tabs with what each shows, the workload's
queries and the deployment inputs on top, the Stage 3 plans ranked by cost
beside three DAG lanes (Stage 0, the chosen Stage 1 candidate, the chosen
Stage 2 candidate with costs), and node and edge details below.

- index.html and app.js replace viewer.js; stages.js is unchanged and
  node-style.js keeps only the kind table the contract test reads, plus
  the three node groups.
- examples.json lists the six #509 examples; server.py writes any
  missing document into out/ with stage_pipeline.
- The query editor (editor.js) plans PromQL queries with optional ε/δ
  through /api/plan, which now runs stage_pipeline.
- Removed: the Pre/Post-ASAP view, WorkloadDAG loading, render.py's
  standalone HTML, the dag_export-based sample and fixtures, screenshots.
- test_viewer.py replaces test_render.py: the stage-document tests, the
  page and server checks, and app.js run against a stub DOM in V8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 5, 2026
Open
Open
Draft
zzylol added a commit that referenced this pull request Oct 5, 2026
@zzylol @claude
docs: describe the stage pipeline instead of the removed legacy search
64e122d 
Library API, architecture overview, planner pipeline and the accuracy
and cost developer docs now describe the #509 stage pipeline. Docs
that only describe removed APIs (legacy search, CostModel,
explanation, physical-plan cost adapter) are marked historical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Oct 5, 2026
Draft
2 tasks done
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(planner): remove the summary maintenance lifecycle
21cc044 
Stage 2 materialization (#509) will decide per sub-DAG whether and when to
materialize, so the Planner no longer chooses a maintenance lifecycle.

- MajorPass now runs search_workload_with_targets -> global_selection ->
  assemble_selected_dag per root -> share_common_sub_dags.
- QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's
  execution_timed_dag times the roots directly. LifecycleInput, the
  lifecycle errors and UserInput's `lifecycle` field are gone (public API
  break).
- Delete summary_maintenance_lifecycle, summary_maintenance_cost,
  summary_maintenance_dag_export, post_asap::{summary_maintenance,
  summary_maintenance_lifecycle} and SummaryWindowFramework (the pane
  primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides
  and EmpiricalEvidenceProvider::lifecycle_cost_inputs.
- Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire
  e2e_plan, summary_sharing, operator_design_examples and
  weighted_topk_binding to the new pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(ir): default to query-time materialization
c14c60e 
Rename LifecycleAssignment to MaterializationAssignment and
apply_lifecycle_timings to apply_materialization_timings. The default
assignment is now all_query_time(): nothing is materialized until Stage 2
materialization (#509) decides per sub-DAG. all_ingestion_time() and set()
assign maintenance explicitly.

- validate_default becomes validate_maintained: candidate legality is still
  checked with every summary maintained, so candidate generation is
  unchanged. planned_data_state and fixed_window_rate_candidates use the
  same maintained assumption, and the maintained precompute compilers in
  promql_rows assign ingestion time explicitly.
- PlanOutput::execution_timed_dag, show_post_asap_ir and the default test
  helpers now emit query-time summaries.
- Tests that exercise maintenance assign it explicitly (new `maintained`
  helpers); a new unit test pins the query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
docs: drop the summary maintenance lifecycle and document the query-t…
c20b6cf 
…ime default

Remove the lifecycle APIs, recipes and viewer section from the docs, delete
the workload-demand-and-summary-lifecycle proposal, and point materialization
questions to Stage 2 (#509). Rename LifecycleAssignment/apply_lifecycle_timings/
validate_default to their new names and describe the all-query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(planner): plan one shared unified workload DAG
166e386 
Cut today's planner over to the unified OperatorNode IR. Frontends return
OperatorNode / QueryRoot, ParsedWorkload keeps scalar roots, Pass 1
(ASAPStrategies), the existing identical-sub-DAG sharing, selection, DAG
assembly and lifecycle costing all run on OperatorNode, and PlanOutput exposes
the whole workload DAG. The native compiler from #541 becomes the canonical
physical_planner and consumes the PhysicalASAPDAG export.

Ported from the earlier #542 (95eef55) without new #509 stage logic. Legacy
QueryExpr/SummaryNode modules stay compiled for their own tests but are no
longer re-exported from post_asap; the cleanup PR removes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(planner): remove the summary maintenance lifecycle
20abea6 
Stage 2 materialization (#509) will decide per sub-DAG whether and when to
materialize, so the Planner no longer chooses a maintenance lifecycle.

- MajorPass now runs search_workload_with_targets -> global_selection ->
  assemble_selected_dag per root -> share_common_sub_dags.
- QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's
  execution_timed_dag times the roots directly. LifecycleInput, the
  lifecycle errors and UserInput's `lifecycle` field are gone (public API
  break).
- Delete summary_maintenance_lifecycle, summary_maintenance_cost,
  summary_maintenance_dag_export, post_asap::{summary_maintenance,
  summary_maintenance_lifecycle} and SummaryWindowFramework (the pane
  primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides
  and EmpiricalEvidenceProvider::lifecycle_cost_inputs.
- Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire
  e2e_plan, summary_sharing, operator_design_examples and
  weighted_topk_binding to the new pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(ir): default to query-time materialization
69caa82 
Rename LifecycleAssignment to MaterializationAssignment and
apply_lifecycle_timings to apply_materialization_timings. The default
assignment is now all_query_time(): nothing is materialized until Stage 2
materialization (#509) decides per sub-DAG. all_ingestion_time() and set()
assign maintenance explicitly.

- validate_default becomes validate_maintained: candidate legality is still
  checked with every summary maintained, so candidate generation is
  unchanged. planned_data_state and fixed_window_rate_candidates use the
  same maintained assumption, and the maintained precompute compilers in
  promql_rows assign ingestion time explicitly.
- PlanOutput::execution_timed_dag, show_post_asap_ir and the default test
  helpers now emit query-time summaries.
- Tests that exercise maintenance assign it explicitly (new `maintained`
  helpers); a new unit test pins the query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
docs: drop the summary maintenance lifecycle and document the query-t…
5c1560e 
…ime default

Remove the lifecycle APIs, recipes and viewer section from the docs, delete
the workload-demand-and-summary-lifecycle proposal, and point materialization
questions to Stage 2 (#509). Rename LifecycleAssignment/apply_lifecycle_timings/
validate_default to their new names and describe the all-query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(planner): remove the summary maintenance lifecycle
823e248 
Stage 2 materialization (#509) will decide per sub-DAG whether and when to
materialize, so the Planner no longer chooses a maintenance lifecycle.

- MajorPass now runs search_workload_with_targets -> global_selection ->
  assemble_selected_dag per root -> share_common_sub_dags.
- QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's
  execution_timed_dag times the roots directly. LifecycleInput, the
  lifecycle errors and UserInput's `lifecycle` field are gone (public API
  break).
- Delete summary_maintenance_lifecycle, summary_maintenance_cost,
  summary_maintenance_dag_export, post_asap::{summary_maintenance,
  summary_maintenance_lifecycle} and SummaryWindowFramework (the pane
  primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides
  and EmpiricalEvidenceProvider::lifecycle_cost_inputs.
- Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire
  e2e_plan, summary_sharing, operator_design_examples and
  weighted_topk_binding to the new pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(ir): default to query-time materialization
0d3b750 
Rename LifecycleAssignment to MaterializationAssignment and
apply_lifecycle_timings to apply_materialization_timings. The default
assignment is now all_query_time(): nothing is materialized until Stage 2
materialization (#509) decides per sub-DAG. all_ingestion_time() and set()
assign maintenance explicitly.

- validate_default becomes validate_maintained: candidate legality is still
  checked with every summary maintained, so candidate generation is
  unchanged. planned_data_state and fixed_window_rate_candidates use the
  same maintained assumption, and the maintained precompute compilers in
  promql_rows assign ingestion time explicitly.
- PlanOutput::execution_timed_dag, show_post_asap_ir and the default test
  helpers now emit query-time summaries.
- Tests that exercise maintenance assign it explicitly (new `maintained`
  helpers); a new unit test pins the query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
docs: drop the summary maintenance lifecycle and document the query-t…
877e5d1 
…ime default

Remove the lifecycle APIs, recipes and viewer section from the docs, delete
the workload-demand-and-summary-lifecycle proposal, and point materialization
questions to Stage 2 (#509). Rename LifecycleAssignment/apply_lifecycle_timings/
validate_default to their new names and describe the all-query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Oct 6, 2026
Merged
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
feat(ir): add canonicalization, sub-DAG sharing and flat DAGs on the …
76e713b 
…unified IR (#645)

#536 defined the unified OperatorNode DAG, but canonicalization, common
sub-DAG sharing and a flat, serializable form existed only for the old
split IRs. Add them on OperatorNode:

- ir::canonicalize: heavy-hitter promotion and EXISTS / NOT EXISTS / IN
  subquery lowering, bottom-up and memoized so shared sub-DAGs stay shared.
- ir::cse: hash-consing over a workload batch (the identical-expression
  rule of #509 Pass 2), following scalar-referenced operator nodes too.
- Generic child references: Operator, NonASAPOp, ASAPOp, ScalarExpr,
  Predicate, ProjectItem, SortKey and QueryRoot take the child reference
  as a type parameter (default Rc<OperatorNode>). ir::flat::flatten writes
  a DAG as nodes whose operators are Operator<NodeId>.

Summary coverage (#567) and SummaryMerge (#560) build on this.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(planner): remove the summary maintenance lifecycle
5c3f5f4 
Stage 2 materialization (#509) will decide per sub-DAG whether and when to
materialize, so the Planner no longer chooses a maintenance lifecycle.

- MajorPass now runs search_workload_with_targets -> global_selection ->
  assemble_selected_dag per root -> share_common_sub_dags.
- QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's
  execution_timed_dag times the roots directly. LifecycleInput, the
  lifecycle errors and UserInput's `lifecycle` field are gone (public API
  break).
- Delete summary_maintenance_lifecycle, summary_maintenance_cost,
  summary_maintenance_dag_export, post_asap::{summary_maintenance,
  summary_maintenance_lifecycle} and SummaryWindowFramework (the pane
  primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides
  and EmpiricalEvidenceProvider::lifecycle_cost_inputs.
- Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire
  e2e_plan, summary_sharing, operator_design_examples and
  weighted_topk_binding to the new pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(ir): default to query-time materialization
38ff0e5 
Rename LifecycleAssignment to MaterializationAssignment and
apply_lifecycle_timings to apply_materialization_timings. The default
assignment is now all_query_time(): nothing is materialized until Stage 2
materialization (#509) decides per sub-DAG. all_ingestion_time() and set()
assign maintenance explicitly.

- validate_default becomes validate_maintained: candidate legality is still
  checked with every summary maintained, so candidate generation is
  unchanged. planned_data_state and fixed_window_rate_candidates use the
  same maintained assumption, and the maintained precompute compilers in
  promql_rows assign ingestion time explicitly.
- PlanOutput::execution_timed_dag, show_post_asap_ir and the default test
  helpers now emit query-time summaries.
- Tests that exercise maintenance assign it explicitly (new `maintained`
  helpers); a new unit test pins the query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
docs: drop the summary maintenance lifecycle and document the query-t…
44a7839 
…ime default

Remove the lifecycle APIs, recipes and viewer section from the docs, delete
the workload-demand-and-summary-lifecycle proposal, and point materialization
questions to Stage 2 (#509). Rename LifecycleAssignment/apply_lifecycle_timings/
validate_default to their new names and describe the all-query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(planner): remove the summary maintenance lifecycle
479cf3a 
Stage 2 materialization (#509) will decide per sub-DAG whether and when to
materialize, so the Planner no longer chooses a maintenance lifecycle.

- MajorPass now runs search_workload_with_targets -> global_selection ->
  assemble_selected_dag per root -> share_common_sub_dags.
- QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's
  execution_timed_dag times the roots directly. LifecycleInput, the
  lifecycle errors and UserInput's `lifecycle` field are gone (public API
  break).
- Delete summary_maintenance_lifecycle, summary_maintenance_cost,
  summary_maintenance_dag_export, post_asap::{summary_maintenance,
  summary_maintenance_lifecycle} and SummaryWindowFramework (the pane
  primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides
  and EmpiricalEvidenceProvider::lifecycle_cost_inputs.
- Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire
  e2e_plan, summary_sharing, operator_design_examples and
  weighted_topk_binding to the new pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
refactor(ir): default to query-time materialization
597325b 
Rename LifecycleAssignment to MaterializationAssignment and
apply_lifecycle_timings to apply_materialization_timings. The default
assignment is now all_query_time(): nothing is materialized until Stage 2
materialization (#509) decides per sub-DAG. all_ingestion_time() and set()
assign maintenance explicitly.

- validate_default becomes validate_maintained: candidate legality is still
  checked with every summary maintained, so candidate generation is
  unchanged. planned_data_state and fixed_window_rate_candidates use the
  same maintained assumption, and the maintained precompute compilers in
  promql_rows assign ingestion time explicitly.
- PlanOutput::execution_timed_dag, show_post_asap_ir and the default test
  helpers now emit query-time summaries.
- Tests that exercise maintenance assign it explicitly (new `maintained`
  helpers); a new unit test pins the query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
docs: drop the summary maintenance lifecycle and document the query-t…
13628f9 
…ime default

Remove the lifecycle APIs, recipes and viewer section from the docs, delete
the workload-demand-and-summary-lifecycle proposal, and point materialization
questions to Stage 2 (#509). Rename LifecycleAssignment/apply_lifecycle_timings/
validate_default to their new names and describe the all-query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Selvomega Selvomega Selvomega left review comments
@milindsrivastava1997 milindsrivastava1997 milindsrivastava1997 left review comments
@GordonYuanyc GordonYuanyc Awaiting requested review from GordonYuanyc

Assignees

@zzylol zzylol

Labels

documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zzylol @milindsrivastava1997 @Selvomega