Skip to content

fixed_grain for level metrics - #2532

Open
robinld wants to merge 3 commits into
DataJunction:semi-additive-metricsfrom
robinld:robind/fixed-grain
Open

fixed_grain for level metrics#2532
robinld wants to merge 3 commits into
DataJunction:semi-additive-metricsfrom
robinld:robind/fixed-grain

Conversation

@robinld

@robinld robinld commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Implements the fixed_grain half of #2245. A base metric can declare the grain at which its aggregate is computed, and DJ broadcasts that value when the query requests a finer grain.

This PR is stacked on #2502, which implements the complementary reaggregate direction. It can be retargeted to main after #2502 lands.

- name: revenue
  query: SELECT SUM(revenue) FROM sales_f

- name: total_revenue
  query: SELECT SUM(revenue) FROM sales_f
  fixed_grain: []

- name: pct_of_total
  query: SELECT revenue / total_revenue

fixed_grain has three forms:

  • omitted: compute at the query grain, preserving current behavior
  • []: compute at the global grain
  • [dimension, ...]: compute at the declared partition grain

For example, querying total_revenue at a finer grain generates the equivalent of:

SUM(SUM(sales_f_0.revenue_sum)) OVER () AS total_revenue

The inner aggregate computes the query-group contribution. The outer window re-merges those contributions within the declared fixed grain. A non-empty grain produces OVER (PARTITION BY ...) instead.

Implementation

  • sql/decompose.py validates that the declaration belongs to one freely aggregatable base component, propagates it through decomposition, and rejects a later window over a broadcast value.
  • The broadcast is written into the combiner during decomposition, naming partition dimensions in unresolved form. Each consumer resolves them as it does any dimension reference — a CTE-qualified column for a query, a bare column for a cube — so the window reaches both query SQL and the SQL stored on a materialized cube.
  • construction/build_v3/cube_matcher.py skips a cube that does not materialize a metric's partition dimension, mirroring validate_cube_covers_reaggregate_dimensions. A cube that does carry it is used normally.
  • The declaration is carried through storage, deployment specs, namespace export/copy, REST and GraphQL output, loader allowlists, change classification, and semantic fingerprints.

Current constraints

The initial implementation accepts only cases where broadcasting can be expressed without changing the metric's meaning.

Situation Behavior
COUNT(DISTINCT ...), a multi-component metric (AVG, variance), or a non-decomposable aggregate Refused; the result cannot be re-merged as a window. Multi-component is unimplemented rather than impossible — it needs every merge windowed, not just the first
A declared grain dimension is absent from the query grain Joined and grouped on as private grain, then projected away — the same treatment a protected reaggregate dimension gets. The result stays the shape that was requested
A non-empty fixed grain is combined with another fact grain group Refused; the joined SELECT groups over a COALESCE, so a partition naming one group's column is ungrouped
A derived metric declares its own fixed grain Refused; the declaration belongs to the base aggregate
A frame accumulates a fixed-grain metric, directly or transitively Refused; the frame would add the same broadcast value once per row. MAX/AVG over it are allowed, as is referencing it beside a window — e.g. a trailing sum divided by a global total
A cube does not materialize the partition dimension Cube skipped; DJ builds from the underlying graph. A cube that carries the dimension is used normally

Related fixes

  • Normalizes both frozen-measure creation paths so metric-level reaggregate and fixed_grain declarations do not become part of shared measure state or identity.
  • Adds both declarations to the GraphQL extractedMeasures loader's load_only. Reading a field that loader omits raises MissingGreenlet on a cold session, which the batch handler turns into a silent null rather than an error.

Out of scope

  • Relative grain declarations such as {hierarchy, drill_up}
  • The proposed frame primitive
  • UI support. reaggregate gained create/edit fields and node-page display in feat: semi-additive measures #2502; the equivalent for fixed_grain is deferred to a follow-up, so the declaration is currently settable only through deployment specs and the API.

Known gaps

  • Direct POST /nodes/metric validation does not yet reject an unresolvable fixed-grain column, although deployment validation does. The invalid declaration is rejected later when a query attempts to use it.

Testing

Coverage includes global and partitioned broadcasts, derived ratios, declaration round trips, namespace copying, semantic identity, frozen-measure reuse, GraphQL decomposition loading, window refusal, and cube matching and materialization carrying the broadcast.

@robinld
robinld marked this pull request as ready for review September 9, 2026 22:09
@robinld
robinld requested a review from doulam September 9, 2026 22:10
@robinld robinld linked an issue Sep 10, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: Level (and other Derived) Metrics

1 participant