Skip to content

docs: design for ASAP primitive schema and summary semantics - #573

Open
zzylol wants to merge 11 commits into
mainfrom
docs/asap-primitive-schema
Open

docs: design for ASAP primitive schema and summary semantics#573
zzylol wants to merge 11 commits into
mainfrom
docs/asap-primitive-schema

Conversation

@zzylol

@zzylol zzylol commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why

ASAP primitives (summary states such as KLL, Count-Min or exact accumulators) flow along DAG edges next to raw values. A Schema describes what an edge is (fields and state types), but not which data a summary summarizes. So two states with identical schemas can still be wrong to merge or reuse, for example overlapping time windows or the same population counted twice. The design for this was split across several docs; this PR collects it into one design document.

What

  • New: docs/design_docs/proposals/asap-primitive-schema.md. It covers:
    1. Goal, problem and requirements, with equal-schema examples.
    2. The schema as the DAG edge.
    3. Fields with ASAP primitive types (FieldDataType) and the state vs value boundary.
    4. Summary coverage as definition + selection, based on Goldstein & Larson's view matching (SIGMOD 2001): what a state computes, which rows of that computation it took, how both are derived from the sub-DAG, and the rules for merge, rollup, slice, reuse and subtract.
    5. Examples for every summary operator (SummaryAgg, SummaryEstimate, FinalizeExactAccumulator, MaintainPopulation, EvaluatePopulation, SummaryMerge, and the reserved ones).
    6. Key code interfaces.
  • Moved here, leaving a summary and a link behind: operator-sharing.md §2.1, pre-asap-ir.md (fields and column references), asap-aware-mapping-contracts.md (sketch family identity), and physical-planning-and-deployment.md (input vs summary semantics).
  • Link only: decoupling_op_and_expr.md, concepts/post-asap-ir.md, univmon-frequency-summary.md and the proposals index.

Before / after

  • Before: schema and state-type design is spread over four docs, and none of them says what a summary state covers.
  • After: one design doc defines it end to end, and the other docs point to it.

Related code: #567 added the first SummaryCoverage and #560 enables SummaryMerge. The design here supersedes the coverage shape in #646 (declared time, source/regions/input/group_by) and the declared pane coverage in the Stage 2 stack (#601, #604); those PRs will be reworked to follow it. This PR is documentation only and based on main.

🤖 Generated with Claude Code

@zzylol @claude
docs: design for ASAP primitive schema and summary semantics
273de4d 
One design document for the edge schema of ASAP primitives: fields that carry
summary state, the summary semantics a schema must preserve, coverage, and
per-operator examples. Related docs keep a summary and link to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 3, 2026
@zzylol @claude
docs: move the ASAP primitive schema design to #573
4a91371 
The design document and the docs it consolidates are reviewed separately on
main. This PR keeps code, tests and the ScanSelection rename in docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Oct 3, 2026
Merged
@zzylol @claude
docs: restore the hand-written sections before the examples
1f93e4d 
Keep the author's sections 1–4 as written; the operator examples (§5) and
key code interfaces (§6) stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Oct 3, 2026
Draft
@zzylol @claude
docs: a top-k readout returns selected rows
085cae0 
Follows #579: the top-k sketch readout derives one row per selected item
instead of a packed Utf8 column, matching the exact Sort → Limit shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Oct 4, 2026
Draft
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
docs: move the ASAP primitive schema design to #573
89804d5 
The design document and the docs it consolidates are reviewed separately on
main. This PR keeps code, tests and the ScanSelection rename in docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: explain why summary coverage is a node field, not part of the s…
a0db033 
…chema

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 6, 2026
@zzylol @claude
feat(ir): add summary coverage metadata for time and population (#567)
c3471c0 
* feat(ir): define joint summary observation coverage

* refactor(ir): name represented observations ObservationExtent

* refactor(ir): rename observation extent to SummaryCoverage and trim its fields

Name the metadata coverage to match #560 and the docs; drop the single-variant
multiplicity and deployment-specific revision; rename grouping to reduction to
match SummaryAgg; report failures through SchemaDerivationError::Coverage; revert
the unrelated PaneCoverageError rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(ir): describe coverage source as any observation data source

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(ir): require coverage on summary nodes; allow regions without time bounds

validate_structure rejects a SummaryAgg without coverage (CoverageError::Missing).
CoverageRegion time bounds become optional so tabular sources without a time
column can declare coverage. Population stays trusted; #570 tracks checking it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: explain the summary coverage problem with examples

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(ir): keep only time and population in SummaryCoverage

Given required coverage on summary nodes, input and reduction duplicated the
producing SummaryAgg fields; drop them along with ProducerMismatch. Type source
as Source, matching Scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cost): rename SourceCoverage to ScanSelection

SourceCoverage names the rows a physical scan reads for cost comparison, not
which observations a summary state holds; rename it so it is not confused with
SummaryCoverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: turn summary coverage into a design document

Move it to docs/design_docs/proposals with problem and motivation,
requirements, design, alternatives and key code interfaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: design doc for ASAP primitive schema and summary semantics

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update asap-primitive-schema.md

* Update asap-primitive-schema.md

* Update asap-primitive-schema.md

* Update asap-primitive-schema.md

* docs: add code interfaces and per-operator examples to the ASAP primitive schema design

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: move the ASAP primitive schema design to #573

The design document and the docs it consolidates are reviewed separately on
main. This PR keeps code, tests and the ScanSelection rename in docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(ir): say coverage time is absolute and population is Utf8 equality

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(ir): carry summary coverage through CSE and flat DAGs

Coverage is part of a summary state's identity: two states over different
observations are never shared, so CSE hashes and compares it, and a flat
node keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 2 commits October 7, 2026 14:21
@zzylol @claude
docs: summary coverage records rows and columns
7b0a991 
SummaryCoverage records which columns a state summarizes (input,
group_by) as well as which rows. Update the SummaryAgg and SummaryMerge
examples to #560: merges compare coverage columns, carry no coverage
until #646, and merged_coverage is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol @claude
docs: summary coverage as definition + selection, based on Goldstein-…
63acde8 
…Larson view matching

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 7, 2026
@zzylol @claude
refactor(ir): keep SummaryMerge structural; leave coverage to #646
4a0eab7 
Remove the coverage columns (input, group_by), check_columns and the
merge's coverage checks. SummaryMerge now only checks structure: at least
one input, every input is State with one state column and an identical
schema. Whether a structurally valid merge is semantically valid (same
computation, disjoint selections) is decided by summary coverage in #646,
following the design in #573.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol zzylol mentioned this pull request Oct 7, 2026
Open
zzylol and others added 2 commits October 7, 2026 22:50
@zzylol @claude
docs: the 5.1 example's time index is a Timestamp column
fc5e9a9 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol @claude
docs: absolute time is a timestamp-column interval; match the coverag…
b1bc9c7 
…e interface in #646

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 7, 2026
@zzylol @claude
feat(ir): derive summary coverage as definition + selection
adbe786 
Coverage is derived from the node, never declared. For a SummaryAgg,
predicate conjuncts that lift through Filter, a range TimeRange, a
TimeShift without @ and direct-column Project items, and are a value set
or an interval on one column, form the selection; everything else stays
in the definition (the SummaryAgg with the selection removed). A
TimeRange(w) over TimeShift(s) gives the relative window (-(s+w), -s].

A SummaryMerge is valid only when its inputs have structurally equal
definitions (ignoring timing and guarantee) and pairwise disjoint
selections; OperatorNode::new and validate_structure enforce it. Its
coverage is the shared definition and the union of the selections,
joining adjacent windows and value sets.

OperatorNode.coverage becomes a private cache behind coverage():
ignored by equality, skipped by serde, emptied on clone. with_coverage,
requires_coverage and CoverageError::Missing are removed, as are the
coverage fields of CSE keys and FlatNode.

Design: docs/design_docs/proposals/asap-primitive-schema.md §4 (#573).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol zzylol mentioned this pull request Oct 7, 2026
Open
@zzylol
zzylol marked this pull request as ready for review October 8, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zzylol