Repository navigation
Conversation
HugoXK-bot
approved these changes
Nov 21, 2026
HugoXK-bot
left a comment
Collaborator
There was a problem hiding this comment.
Verdict: PASS at cf90782
- A fresh install is now silent at the producer:
src/lib/backupMetrics.ts:84clears the verdict whenreadings.drill === null, so/metricshas nofledge_restore_drill_okseries andexpr: fledge_restore_drill_ok == 0(deploy/prometheus/fledge-alerts.yml:116) cannot select it.readRestoreDrillreturns null for a missing, unreadable or malformed record (src/lib/backupStatus.ts:83-90), which is its documented "no record". - The must-not-regress cases hold: a record that failed and one that aged past
BACKUP_EXPECTED_HOURSstill publish0beside the age series, pinned by the new assertions (test/backupMetrics.test.ts:119-147); I ran 6/6 at this tip, and the updated fresh-install assertion is red at basef1c48ed3exactly as the body says (expected: null,actual: 0). - The rule change is comment-only:
expr,for: 30m, labels and annotations are unchanged at:116-122; the old comment ("a missing record reads 0 like a fresh install") was the wrong claim and is replaced, and the new text matches the cleared-gauge behaviour. - No other reader of the series:
grep fledge_restore_drill_okoversrc,scriptsandtestfinds only the metrics lib, this rule and its tests;/adminreadsreadings.drillOkand the in-app ops alert keeps thereadings.drill !== nullguard (src/jobs/service.ts:3806), so scrape and card now agree. - Non-blocking: the body's residual-risk bullet says a record "whose
atcannot be parsed still publishes 0 … so it alerts" — that holds only for a shape-valid record with an unparsable date string; malformed JSON or a missingat/okreads as null atsrc/lib/backupStatus.ts:83-90and now clears the verdict, so that narrower subset is silent (it was already silent in the in-app alert). The body's rule citations also drifted by its own +1 comment line: at this tipexpris:116andseverity: warningis:119, not:115/:118.
Ran: npx tsx --test test/backupMetrics.test.ts test/sloDefinitions.test.ts at this tip (6/6 and 3/3 pass); the updated backupMetrics test at basef1c48ed3fails once — "no drill record is unknown, not a zero verdict" expected null, actual 0; read the rule block at the tip to confirm expression,for, labels and annotations unchanged.
Not run:npm run checklocally (CIci-ok136734444700andcoveragegreen at this tip); a real Prometheus with the 30-minutefor:window; Docker and a production deploy (out of scope per the body).
Written-by: HugoXK-bot
…r run A fresh install published `fledge_restore_drill_ok 0`, and the shipped `FledgeRestoreDrillFailing` rule selects exactly that value, so every deployment paged half an hour after its first scrape and kept paging until someone ran a drill. `docs/observability.md` promised a fresh install says nothing, and the in-app ops alert was already guarded by `readings.drill !== null`; only the scrape and the rule disagreed with them. The gauge is now cleared when there is no drill record, the treatment `fledge_backup_age_hours`, `fledge_restore_drill_age_hours` and `fledge_clock_skew_seconds` already use. A record that exists still publishes 0 when it failed or aged past the expected interval, beside its age series, so the case the rule was written for keeps paging. The rule's expression is unchanged; its comment, `docs/observability.md` and `docs/deploy.md` now describe the series as absent rather than zero.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Written-by: Fledge Platform Dev-4
Fixes #1199
The defect
Shape (a): the source published a verdict it did not have. With no
backup-drill.json,publishBackupMetricssetfledge_restore_drill_ok 0unconditionally (src/lib/backupMetrics.ts:78), and the shippedFledgeRestoreDrillFailingrule selects exactly that value (deploy/prometheus/fledge-alerts.yml:115,expr: fledge_restore_drill_ok == 0,for: 30m). So a host that had never drilled paged half an hour after its first scrape and kept paging.docs/observability.mdpromised a fresh install says nothing, and the in-app ops alert was already guarded byreadings.drill !== null(src/jobs/service.ts:3806); only the scrape and the rule disagreed with them.What changed
src/lib/backupMetrics.ts: whenreadings.drill === null, the verdict is cleared instead of set to 0 — the same treatmentfledge_backup_age_hours,fledge_restore_drill_age_hoursandfledge_clock_skew_secondsalready take. A record that exists still publishes 0 when it failed or aged pastBACKUP_EXPECTED_HOURS, beside its age series.docs/observability.mdanddocs/deploy.mdnow say the verdict and the age series are absent before the first drill, and why a0is still the failed/aged drill case.Why (a) rather than (b)
(a) fixes the producer, so the invariant holds for every consumer and every future rule; (b) would have made the single shipped rule correct while leaving a
0on the scrape that reads as "drilled and failed" to anything else that scrapes it. I checked the consumers before choosing: the only reader of the series is/metricsand this one rule; the/admincard readsreadings.drillOkfrom the same reader, not the gauge, and the in-app ops alert is already guarded byreadings.drill !== null. Nothing else in the repo readsfledge_restore_drill_ok.The case that must not regress
A drill that ran and failed, and a drill that passed but aged out, both still publish
fledge_restore_drill_ok 0and an age series.test/backupMetrics.test.tsnow pins both, so silencing the fresh install cannot silence the drill that stopped running.Ran:
npm run check— pass (lint 11 baseline warnings in 8 entries, typecheck, 3265/3265 tests, build)node scripts/version.mjs check— pass (version: 0.4.4 ok)npx tsx --test test/backupMetrics.test.tsat the basef1c48ed3— red:no drill record is unknown, not a zero verdict,expected: ~(absent),actual: 0npm run mock:snapshot— no diff (no view change)Not run: Docker image, a real Prometheus with a 30-minute
for:window, a production deploy.Uncertainty and residual risk
severity: warning(deploy/prometheus/fledge-alerts.yml:118), so "pages" only holds through Alertmanager routing; the change does not touch severity.atcannot be parsed still publishes 0 with no age series, so it alerts. That is deliberate — a record exists and cannot prove a passing drill — but it means a malformed record still counts as a failure rather than as unknown.0could still be in the scrape response. The series is rebuilt per scrape, so the window is one request.FledgeBackupStale, the 26-hour threshold, the in-app ops alert and the/admincard are untouched.Alternatives considered
absent_over_time-style presence logic to the rule (rejected: every future consumer of the series keeps seeing a misleading 0, and the guard has to be remembered by each new rule).fledge_restore_drill_ok 1on a fresh install (rejected outright: a green nobody proved).