Skip to content

types: evaluation-relative coverage time and mergeable summary families - #592

Draft
zzylol wants to merge 2 commits into
stack/509-w6-counter-prooffrom
stack/509-w3-relative-coverage
Draft

zzylol wants to merge 2 commits into
stack/509-w6-counter-prooffrom
stack/509-w3-relative-coverage

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Restacked on #646 (coverage = definition + selection, derived). Chain: … #594 → #593 → #592 → #591 → … (Phase C tail, #528).

Step 3 of the #580 plan (item A prerequisites). Part of #509.

Why

Stage 1 tumbling windows (W1) compose N SummaryAgg panes with one SummaryMerge. #646 already derives where a pane sits in time: TimeRange(w) over TimeShift(s) gives the selection's relative time (-(s+w), -s], and a merge is accepted only for equal definitions and disjoint selections. Two facts are still missing:

  • Whether a family may merge. SummaryMerge accepted any state family with matching schemas, including Rate/Increase accumulators and CMS/CountSketch heaps. W6 says merged heaps are rejected until there is an accuracy model.
  • How the inputs' selections may relate. Disjointness is right for counting families, but idempotent families (HLL, exact Min/Max) give the same state when a row is seen twice, so overlapping inputs are fine (docs: design for ASAP primitive schema and summary semantics #573 §4.4).

What

  • FieldDataType::family_merges() returns true for exact Sum/Count/Min/Max, KLL, DDSketch, HLL, CMS, CountSketch and UnivMon. Everything else fails closed: Rate/Increase/IRate, CmsWithHeap, CountSketchWithHeap, KMV/Theta, Sample/Wavelet/StatModel and plain values. SummaryMerge::validate_inputs rejects a family that does not merge.
  • FieldDataType::merge_relation() -> Option<SelectionRelation>, next to family_merges: Disjoint for counting families, OverlapAllowed for HLL and exact Min/Max, None when the family does not merge. SelectionRelation::Contained names what the reserved SummarySubtract will require; nothing produces it yet.
  • SummaryCoverage::derive skips the disjointness check for OverlapAllowed families, and the union of identical selection boxes is one box.

The earlier version of this PR added CoverageTime::{Absolute, RelativeToEvaluation} to declared coverage; that commit is dropped because #646 derives relative time from the sub-DAG.

SummaryMerge(KLL us, KLL eu)          -> selection {region ∈ {us}} ∪ {region ∈ {eu}}
SummaryMerge(KLL us, KLL us)          -> PossibleOverlap
SummaryMerge(HLL us, HLL us)          -> accepted, selection {region ∈ {us}}
SummaryMerge over CmsWithHeap states  -> "summary merge over Sketch(CmsWithHeap…) is unsupported: the family has no sound merge"

Tests

Written first; idempotent_families_merge_overlapping_states and family_merge_selection_relation fail without the change.

  • summary_merge_structure.rs: heap states are rejected and KLL states accepted (summary_merge_requires_a_mergeable_family); table tests of family_merges and merge_relation; HLL/Max states over the same rows merge and Sum/KLL ones do not.

Open questions for review

  • KMV and Theta merge soundly in the literature but were not on the W6 list, so they currently fail closed.

Gate

cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings and cargo test --workspace pass at this head.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 390d792 to 0f9db83 Compare October 4, 2026 18:25
zzylol added a commit that referenced this pull request Oct 4, 2026
Pane coverage time is a CoverageTime since #592; the tumbling pane-merge test
keeps absolute time ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol changed the base branch from stack/509-e2-keyed-additive-topk to stack/509-w6-counter-proof October 4, 2026 18:25
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from ee6f881 to 761b2ce Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 0f9db83 to 6b31d2d Compare October 5, 2026 06:21
zzylol and others added 2 commits October 8, 2026 14:30
FieldDataType::family_merges() is true for exact Sum/Count/Min/Max,
KLL, DDSketch, HLL, CMS, CountSketch and UnivMon; every other family
fails closed (Rate/Increase accumulators and heap top-k sketches
included, per W6). SummaryMerge::validate_inputs rejects the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FieldDataType::merge_relation() returns the SelectionRelation a merge
needs (#573 §4.4): Disjoint for counting families (exact Sum/Count, KLL,
DDSketch, CMS, CountSketch, UnivMon), OverlapAllowed for idempotent ones
(exact Min/Max, HLL), and None when the family does not merge. Contained
names what the reserved SummarySubtract will require.

SummaryCoverage::derive skips the disjointness check for OverlapAllowed
families, and the union of identical selection boxes is one box.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from 761b2ce to f6a38fb Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 6b31d2d to 8e9b5f9 Compare October 8, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant