Skip to content

fix(metrics): publish no restore-drill verdict before a drill has ever run - #1203

Open
HugoXK wants to merge 1 commit into
mainfrom
dev4/1199-drill-alert-fresh-install
Open

HugoXK wants to merge 1 commit into
mainfrom
dev4/1199-drill-alert-fresh-install

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Written-by: Fledge Platform Dev-4

Fixes #1199

The defect

Shape (a): the source published a verdict it did not have. With no backup-drill.json, publishBackupMetrics set fledge_restore_drill_ok 0 unconditionally (src/lib/backupMetrics.ts:78), and the shipped FledgeRestoreDrillFailing rule selects exactly that value (deploy/prometheus/fledge-alerts.yml:115, expr: fledge_restore_drill_ok == 0, for: 30m). So a host that had never drilled paged half an hour after its first scrape and kept paging. docs/observability.md promised a fresh install says nothing, and the in-app ops alert was already guarded by readings.drill !== null (src/jobs/service.ts:3806); only the scrape and the rule disagreed with them.

What changed

  • src/lib/backupMetrics.ts: when readings.drill === null, the verdict is cleared instead of set to 0 — the same treatment fledge_backup_age_hours, fledge_restore_drill_age_hours and fledge_clock_skew_seconds already take. A record that exists still publishes 0 when it failed or aged past BACKUP_EXPECTED_HOURS, beside its age series.
  • The rule's expression is unchanged. Absence means Prometheus has no series to compare, so the fresh-install case disappears without a presence guard every future rule would have to remember. Only the rule's comment changed, because it claimed a missing record "reads 0 like a fresh install does".
  • docs/observability.md and docs/deploy.md now say the verdict and the age series are absent before the first drill, and why a 0 is still the failed/aged drill case.

Why (a) rather than (b)

(a) fixes the producer, so the invariant holds for every consumer and every future rule; (b) would have made the single shipped rule correct while leaving a 0 on the scrape that reads as "drilled and failed" to anything else that scrapes it. I checked the consumers before choosing: the only reader of the series is /metrics and this one rule; the /admin card reads readings.drillOk from the same reader, not the gauge, and the in-app ops alert is already guarded by readings.drill !== null. Nothing else in the repo reads fledge_restore_drill_ok.

The case that must not regress

A drill that ran and failed, and a drill that passed but aged out, both still publish fledge_restore_drill_ok 0 and an age series. test/backupMetrics.test.ts now pins both, so silencing the fresh install cannot silence the drill that stopped running.

Ran:

  • npm run check — pass (lint 11 baseline warnings in 8 entries, typecheck, 3265/3265 tests, build)
  • node scripts/version.mjs check — pass (version: 0.4.4 ok)
  • npx tsx --test test/backupMetrics.test.ts at the base f1c48ed3 — red: no drill record is unknown, not a zero verdict, expected: ~ (absent), actual: 0
  • npm run mock:snapshot — no diff (no view change)
    Not run: Docker image, a real Prometheus with a 30-minute for: window, a production deploy.

Uncertainty and residual risk

  • The rule is severity: warning (deploy/prometheus/fledge-alerts.yml:118), so "pages" only holds through Alertmanager routing; the change does not touch severity.
  • A record that exists but whose at cannot be parsed still publishes 0 with no age series, so it alerts. That is deliberate — a record exists and cannot prove a passing drill — but it means a malformed record still counts as a failure rather than as unknown.
  • A scrape reads the gauge list at scrape time; between the drill file being deleted and the next scrape, a previously published 0 could still be in the scrape response. The series is rebuilt per scrape, so the window is one request.
  • No new metric, exporter, scheduler or rule; FledgeBackupStale, the 26-hour threshold, the in-app ops alert and the /admin card are untouched.

Alternatives considered

  • Keep the gauge and add absent_over_time-style presence logic to the rule (rejected: every future consumer of the series keeps seeing a misleading 0, and the guard has to be remembered by each new rule).
  • Publish fledge_restore_drill_ok 1 on a fresh install (rejected outright: a green nobody proved).

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS at cf90782

  • A fresh install is now silent at the producer: src/lib/backupMetrics.ts:84 clears the verdict when readings.drill === null, so /metrics has no fledge_restore_drill_ok series and expr: fledge_restore_drill_ok == 0 (deploy/prometheus/fledge-alerts.yml:116) cannot select it. readRestoreDrill returns null for a missing, unreadable or malformed record (src/lib/backupStatus.ts:83-90), which is its documented "no record".
  • The must-not-regress cases hold: a record that failed and one that aged past BACKUP_EXPECTED_HOURS still publish 0 beside the age series, pinned by the new assertions (test/backupMetrics.test.ts:119-147); I ran 6/6 at this tip, and the updated fresh-install assertion is red at base f1c48ed3 exactly as the body says (expected: null, actual: 0).
  • The rule change is comment-only: expr, for: 30m, labels and annotations are unchanged at :116-122; the old comment ("a missing record reads 0 like a fresh install") was the wrong claim and is replaced, and the new text matches the cleared-gauge behaviour.
  • No other reader of the series: grep fledge_restore_drill_ok over src, scripts and test finds only the metrics lib, this rule and its tests; /admin reads readings.drillOk and the in-app ops alert keeps the readings.drill !== null guard (src/jobs/service.ts:3806), so scrape and card now agree.
  • Non-blocking: the body's residual-risk bullet says a record "whose at cannot be parsed still publishes 0 … so it alerts" — that holds only for a shape-valid record with an unparsable date string; malformed JSON or a missing at/ok reads as null at src/lib/backupStatus.ts:83-90 and now clears the verdict, so that narrower subset is silent (it was already silent in the in-app alert). The body's rule citations also drifted by its own +1 comment line: at this tip expr is :116 and severity: warning is :119, not :115/:118.
    Ran: npx tsx --test test/backupMetrics.test.ts test/sloDefinitions.test.ts at this tip (6/6 and 3/3 pass); the updated backupMetrics test at base f1c48ed3 fails once — "no drill record is unknown, not a zero verdict" expected null, actual 0; read the rule block at the tip to confirm expression, for, labels and annotations unchanged.
    Not run: npm run check locally (CI ci-ok 136734444700 and coverage green at this tip); a real Prometheus with the 30-minute for: window; Docker and a production deploy (out of scope per the body).
    Written-by: HugoXK-bot

@HugoXK-bot HugoXK-bot added the review:approved Issue reviewed and approved by the Fledge Platform Reviewer label Nov 21, 2026
…r run

A fresh install published `fledge_restore_drill_ok 0`, and the shipped
`FledgeRestoreDrillFailing` rule selects exactly that value, so every deployment
paged half an hour after its first scrape and kept paging until someone ran a
drill. `docs/observability.md` promised a fresh install says nothing, and the
in-app ops alert was already guarded by `readings.drill !== null`; only the
scrape and the rule disagreed with them.

The gauge is now cleared when there is no drill record, the treatment
`fledge_backup_age_hours`, `fledge_restore_drill_age_hours` and
`fledge_clock_skew_seconds` already use. A record that exists still publishes 0
when it failed or aged past the expected interval, beside its age series, so the
case the rule was written for keeps paging. The rule's expression is unchanged;
its comment, `docs/observability.md` and `docs/deploy.md` now describe the series
as absent rather than zero.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved Issue reviewed and approved by the Fledge Platform Reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: the shipped restore-drill alert pages every fresh install that has never run a drill

2 participants