Skip to content

adapter: add the first curated metric sinks (sizes and errors) (SQL-556) - #38528

Open
mtabebe wants to merge 1 commit into
MaterializeInc:mainfrom
mtabebe:ma/prom-metrics/sql-556-curated-metrics
Open

adapter: add the first curated metric sinks (sizes and errors) (SQL-556)#38528
mtabebe wants to merge 1 commit into
MaterializeInc:mainfrom
mtabebe:ma/prom-metrics/sql-556-curated-metrics

Conversation

@mtabebe

@mtabebe mtabebe commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

SQL-555 landed the machinery to install curated metric sinks on every
replica, but CURATED was empty, so nothing ran at boot.

Solution:

Add the two v1 definitions: mz_metric_arrangement_sizes (three gauge
families) and mz_metric_dataflow_errors (one gauge). Both read the raw
..._raw logs, not the derived views, which re-aggregate expensively.

The sizes sink labels each series with the exported object's GlobalId, not the
timely dataflow id, which churns across re-renders. A dataflow with no export,
like a subscribe, falls back to an unattributable bucket so its bytes still count.

The errors sink reads the raw log rather than mz_compute_error_counts
because that view joins in mz_compute_dependencies, a storage-managed
source that ensure_reads_only_logs rejects: reading it would couple the
sink's frontier to envd. Elapsed stays out of v1, its log churns
continuously and cost 14% of a 2-worker replica even raw-sourced.

curated_definitions_plan only called plan_source, so a definition that
broke the log-only contract still passed the test but crash-looped the
coordinator at boot. It now runs ensure_reads_only_logs too.

Testing:

  • Unit tests plan each definition and run the log-only gate.
  • metric-sink.td gains a curated section and a multi-cluster,
    multi-replica section.
  • metric-sink.td covers both the attributable and unattributable id paths.
  • The MetricSink platform check now asserts the curated sinks survive a boot,
    which covers restart, upgrade, and zero-downtime.

@mtabebe
mtabebe force-pushed the ma/prom-metrics/sql-556-curated-metrics branch from 3235c00 to 62b0af0 Compare August 27, 2026 18:00
@mtabebe
mtabebe force-pushed the ma/prom-metrics/sql-556-curated-metrics branch from 62b0af0 to bea5072 Compare August 27, 2026 20:10
@mtabebe
mtabebe force-pushed the ma/prom-metrics/sql-556-curated-metrics branch 3 times, most recently from 9d5e103 to 89ca9ec Compare September 3, 2026 00:49
Problem:

SQL-555 landed the machinery to install curated metric sinks on every
replica, but `CURATED` was empty, so nothing ran at boot.

Solution:

Add the two v1 definitions: `mz_metric_arrangement_sizes` (three gauge
families) and `mz_metric_dataflow_errors` (one gauge). Both read the raw
`..._raw` logs, not the derived views, which re-aggregate expensively.

The sizes sink labels each series with the exported object's `GlobalId`, not the
timely dataflow id, which churns across re-renders. A dataflow with no export,
like a subscribe, falls back to an `unattributable` bucket so its bytes still count.

The errors sink reads the raw log rather than `mz_compute_error_counts`
because that view joins in `mz_compute_dependencies`, a storage-managed
source that `ensure_reads_only_logs` rejects: reading it would couple the
sink's frontier to envd. Elapsed stays out of v1, its log churns
continuously and cost 14% of a 2-worker replica even raw-sourced.

`curated_definitions_plan` only called `plan_source`, so a definition that
broke the log-only contract still passed the test but crash-looped the
coordinator at boot. It now runs `ensure_reads_only_logs` too.

Testing:

- Unit tests plan each definition and run the log-only gate.
- metric-sink.td gains a curated section and a multi-cluster,
multi-replica section.
- metric-sink.td covers both the attributable and `unattributable` id paths.
- The MetricSink platform check now asserts the curated sinks survive a boot,
which covers restart, upgrade, and zero-downtime.

Co-Authored-By: Moritz Hoffmann <antiguru@gmail.com>
@mtabebe
mtabebe force-pushed the ma/prom-metrics/sql-556-curated-metrics branch from 89ca9ec to 1610837 Compare September 3, 2026 01:29
@mtabebe
mtabebe marked this pull request as ready for review September 3, 2026 09:28
@mtabebe
mtabebe requested review from a team as code owners September 3, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant