Skip to content

fix(backup): prune each class of backup copy at the count the console reports - #1191

Open
HugoXK wants to merge 2 commits into
mainfrom
dev2/1188-backup-prune-per-class
Open

HugoXK wants to merge 2 commits into
mainfrom
dev2/1188-backup-prune-per-class

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Written-by: Fledge Platform Dev-2

Fixes #1188

What changed

scripts/backup.mjs — the only program that deletes a backup — prunes each class at its own write-time count, keeps the copy the run just wrote, and validates an override that was actually supplied:

  • it collects copies with copyKind() from scripts/lib/backup-manifest.mjs instead of a merged fledge-*.db + fledge-*.db.enc list, sorts each class by mtime and slices it at that class's count, deleting the copy's manifest with it;
  • the counts are BACKUP_KEEP_PLAINTEXT / BACKUP_KEEP_ENCRYPTED, with the defaults imported from scripts/lib/backup-manifest.mjs (DEFAULT_KEEP_PLAINTEXT 1, DEFAULT_KEEP_ENCRYPTED 30) rather than redefined, so the writer, the dry run and loadConfig cannot drift apart;
  • an invalid count is refused with the same wording and rule as scripts/backup-prune.mjs and src/config.ts — a positive whole number, 0 refused rather than clamped;
  • --keep / BACKUP_KEEP remains an explicit override of both classes for one run, and the run says so: retention=keep:plaintext=N,encrypted=N override=--keep. Without the override the line prints both counts it applied, e.g. retention=keep:plaintext=1,encrypted=30. The override applies to both classes; an empty value is refused, never treated as absent;
  • the run's own copy is pinned by path. prune() now takes the path this run wrote and moves it to the front of its class before slicing, so the written copy is never the pruned one — not even when a stale copy carries a future mtime;
  • an empty override is refused. --keep "", BACKUP_KEEP="" and a bare --keep exit 2 through the same refusal the invalid-count path uses, before anything is written or deleted.

docs/deploy.md is rewritten in the same PR at the two blocks the card names: the npm run db:backup comment and the retention section. The comment now says an empty --keep / BACKUP_KEEP is refused, and the retention paragraph says an explicitly supplied but empty value is refused with exit 2 and that the copy a run writes is always among the copies its class keeps, even when an older copy carries a future mtime. changelog.d/1188.fixed.md names both.

Red first, per assertion, at the base

f1c48ed3513a8180b870fae19fb95d32222da44b, with the original nine-node test file dropped onto an otherwise untouched checkout (# tests 9 / # pass 8 / # fail 1). The split is the card's, not a summary of it:

  • Encrypted half — the count assertion is red: the writer prunes the merged list to its own 14, so twenty encrypted copies become 13 (retention=keep:14 pruned=7), not the 20 the plan says are kept. The expiredCopies === 0 leg is green at the base by construction (13 is under the displayed keepEncrypted 30, so the displayed policy reports nothing expired) — it is the post-fix agreement guard, not part of the red.
  • Plaintext half — both assertions are red: twenty plaintext copies plus the one the run writes leave 14 (retention=keep:14 pruned=7) instead of 1, and the plan afterwards reports 13 expired instead of 0.
  • The retention= line assertion is also red at the base, because the base prints keep:14.

Security review at b062bb9 — both blockers

Security comment 7046476556 named two data-loss paths in scripts/backup.mjs. Both are answered in this tip, and each has its own node in test/backupPrune.test.ts.

Blocker 1 — the run's own copy is not pinned into the kept set

At b062bb9 the kept set is chosen by mtime alone (:93-97 slice, :209-210 report). Reproduced in the new fixture: one plaintext copy with an mtime one hour in the future (utimesSync) is joined by the copy the run writes; sorted by mtime the future-dated stale copy is first, the writer prunes the copy it just wrote, exits 0, and prints backup=<path> for a file that no longer exists, then backup: OK. At f1c48ed3 both copies survived.

Answer. prune(dir, keep, target) now finds the entry whose path equals the run's target and moves it to the front of its class before sorted.slice(keep[kind]) (scripts/backup.mjs:101-106). The class is still kept at exactly its configured count — pinning reorders, it does not enlarge the set — and the written copy is inside the kept prefix by path, not by mtime. As a truth guard, after pruning the run checks the path still exists and exits 1 with backup: the backup this run wrote is missing after pruning: <path> (:225-229) instead of printing backup= / backup: OK for a file no reader can open.

New node: a copy with a future mtime never outranks the copy the run writes. It asserts the backup= path exists on disk, the future-dated copy is the one gone, pruned=1, and the plan afterwards is empty.

Blocker 2 — an explicitly passed but empty override is treated as absent

At b062bb9 an empty value is read as "no override" (:56 returns the fallback, :110-111 maps '' to null), so --keep "" and BACKUP_KEEP="" prune at the defaults with exit 0; a wrapper whose variable expands empty deletes with exit 0 where f1c48ed3 refused with exit 2. The override=--keep marker disappears with the value, so the line cannot show that an override was consumed.

Answer. "Supplied" is now separate from "non-empty": Object.hasOwn(args, 'keep') || process.env.BACKUP_KEEP !== undefined (scripts/backup.mjs:125-127). A supplied value goes through keepCount, which refuses an empty or missing one with the invalid-count sentence and exit 2 — --keep must be a positive whole number, got an empty value or got no value (:55-63) — before any path is written or deleted. An accepted override still prints override=--keep, because the marker is printed for every supplied and accepted override.

New node: an explicitly empty keep override is refused, not treated as absent. It runs both shapes, --keep "" and BACKUP_KEEP="", against three planted plaintext copies and asserts exit 2, the refusal on stderr, no backup= line, and a byte-identical directory listing afterwards. A third node, an accepted keep override still prints the override marker, proves the marker path still prints retention=keep:plaintext=2,encrypted=2 override=--keep pruned=2.

Red at b062bb9 for the two new nodes

With the three new nodes in place and scripts/backup.mjs left at b062bb9:

npx tsx --test test/backupPrune.test.ts
# tests 12   # pass 10   # fail 2
  • a copy with a future mtime never outranks the copy the run writes — the run reported a backup it had already pruned: …/fledge-2026-11-20T18-14-53-984Z.db, and existsSync on the printed path was false: the written copy was the pruned one.
  • an explicitly empty keep override is refused, not treated as absent — Expected values to be strictly equal: 0 !== 2: both empty shapes pruned at the defaults and exited 0.

an accepted keep override still prints the override marker is green at b062bb9 and stays green; it is the no-regression node for the marker.

At the tip

539ea744f367fbc232ca80e1843cd846fc339083, tree 72ac6e6083e09693c7a4e9ba21df5ba954753da1:

npx tsx --test test/backupPrune.test.ts                                  → # tests 12 # pass 12 # fail 0
npx tsx --test test/backupPrune.test.ts test/backup.test.ts \
  test/backupPlaintext.test.ts test/backupEncryption.test.ts \
  test/adminBackup.test.ts                                              → 57 passed, 0 failed
npm run check                                                           → exit 0

npm run check at the tip: lint 11 warning(s) in 8 entries match the baseline, typecheck clean, # tests 3268 / # pass 3267 / # fail 0 (1 skipped: the change that decides how a release PR merges may not also edit the gate that judges it), build clean. node scripts/version.mjs check → version: 0.4.4 ok; node scripts/changelog.mjs check → changelog: 23 fragment(s) ok.

The two runs at the shipped defaults, on the card's own fixture:

  • encrypted: twenty copies in, twenty out, plan expiredCopies 0;
  • plaintext: twenty copies plus the written copy in, one out, plan expiredCopies 0;
  • the override path: --keep 2 over five encrypted copies leaves two and prints retention=keep:plaintext=2,encrypted=2 override=--keep pruned=3; --keep lots exits 2 with --keep must be a positive whole number, got lots; --keep "" exits 2 with --keep must be a positive whole number, got an empty value.

Risk tier

node scripts/risk-gate.mjs --base origin/main at the shipping tree, verbatim:

changed files: 4

T3: scripts/backup.mjs
T1: test/backupPrune.test.ts
T0: changelog.d/1188.fixed.md, docs/deploy.md
limits: 40 files / 1200 lines (advisory)

risk-tier: T3; basis: scripts/backup.mjs

T3, unchanged from the first tip: this needs a PASS at this tip plus green ci-ok. Four files, +248/−29 against f1c48ed3.

Alternatives considered

  • Stamp the written copy's mtime instead of pinning its path. Rejected: the mtime is metadata the file system owns and another tool can rewrite; the path of the copy this process just wrote is the one fact that cannot drift during the run.
  • Keep one merged count and make the console report it. Rejected: the console reports per class because the classes serve different purposes — the encrypted copies are the retained history and the plaintext copy is the one the restore drill reads. One number cannot serve both, and it is the encrypted direction that deletes silently.
  • Refuse --keep entirely. Rejected in the card: a host wrapper that passes --keep would stop writing backups at the next cron, turning a retention mismatch into a backup outage.
  • Add --keep-plaintext / --keep-encrypted flags to the writer. Rejected: the environment variables are already the interface loadConfig and the dry run use, and a third shape would need its own documentation for no new capability.

Uncertainty and residual risk

The writer still collects stale .plaintext.tmp intermediates and leaves -wal/-shm sidecars alone, exactly as before. The per-class BACKUP_KEEP_PLAINTEXT / BACKUP_KEEP_ENCRYPTED variables keep their documented empty-means-default reading, which is the reading loadConfig and the dry run apply; only the one-run --keep / BACKUP_KEEP override refuses an empty value, because only that value is passed as a deletion instruction for this run. The first real deletion of an existing host's backlog still happens on the next scheduled run; an operator who wants to see it first should diff backup-prune.mjs --json against the run's retention= line.

Rollback

Revert the commit. The writer returns to per-class counts without the pin and without the empty-value refusal; no schema, no state written outside the backup directory.

UI surface

None — scripts/backup.mjs is a CLI; no page, asset or string changes.

Docs and changelog

docs/deploy.md is updated in this PR (the backup command block and the retention section). changelog.d/1188.fixed.md carries the user-visible line. No Docs-not-needed: claim.

Ran:

  • npx tsx --test test/backupPrune.test.ts at f1c48ed3513a8180b870fae19fb95d32222da44b with the original nine-node file — # tests 9 / # pass 8 / # fail 1
  • npx tsx --test test/backupPrune.test.ts at b062bb9406a4b1a032e8fd7162a7be911d523188 with the twelve-node file — # tests 12 / # pass 10 / # fail 2
  • npx tsx --test test/backupPrune.test.ts at 539ea744f367fbc232ca80e1843cd846fc339083 — 12 passed, 0 failed
  • npx tsx --test test/backupPrune.test.ts test/backup.test.ts test/backupPlaintext.test.ts test/backupEncryption.test.ts test/adminBackup.test.ts at the tip — 57 passed, 0 failed
  • npm run check at the tip — exit 0 (# tests 3268 / # pass 3267 / # fail 0, 1 skipped, lint and typecheck clean, build ok)
  • node scripts/risk-gate.mjs --base origin/main at the shipping tree — risk-tier: T3; basis: scripts/backup.mjs
  • node scripts/changelog.mjs check at the tip — 23 fragments ok
  • node scripts/version.mjs check at the tip — 0.4.4 ok
  • the card's own fixture at the defaults, at b062bb9 and at the tip, with backup-prune.mjs --json read before and after
  • the future-mtime reproduction (utimesSync one hour ahead) at b062bb9 and at the tip
  • --keep 2, --keep lots, --keep "" and BACKUP_KEEP="" at the tip

Not run:

  • no production run and no change to production backup counts; the writer was exercised only against temporary directories
  • the encrypted-write path beyond the existing suites: the new tests run the writer without a key, because the class the copy lands in is what this card is about, and test/backupEncryption.test.ts already covers the keyed path
  • scripts/restore-drill* (out of scope, #463 owns it), Docker image build, real SMTP

answering body bae671b564e2 (issue-gate) and Security comment 7046476556 at b062bb9

@HugoXK

HugoXK commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

Security: CHANGES at b062bb9

  • scripts/backup.mjs:93-97,209-210 — the kept set is chosen by mtime alone and the run's own copy is not pinned into it, so one plaintext copy with a future mtime (clock stepped back, or a set restored with cp -a) makes the run take the file it just wrote: reproduced at the shipped defaults — exit 0, retention=keep:plaintext=1,encrypted=30 pruned=1, a backup=<path> line for a file that no longer exists, backup: OK, and the plan afterwards (expiredCopies 0) names nothing; at f1c48ed both copies survived, and plaintext keep=1 makes a single stale file enough. Pin target into the kept set (or stamp its mtime) before slicing.
  • scripts/backup.mjs:56,110-111 — an explicitly passed but empty --keep/BACKUP_KEEP is treated as absent, so a wrapper whose variable expands empty prunes at the defaults with exit 0 (reproduced: three planted plaintext copies deleted) where f1c48ed refused with exit 2; override=--keep is omitted from retention=, so the line cannot show the override was consumed. Refuse an empty value rather than falling back.

… reports (#1188)

The writer collected plaintext and encrypted copies into one list and pruned the
merged list to a single BACKUP_KEEP of 14, while the dry run, loadConfig and the
/admin card report BACKUP_KEEP_PLAINTEXT (1) and BACKUP_KEEP_ENCRYPTED (30).

With the shipped defaults that deletes seven of twenty encrypted copies the card
says are kept, silently, and leaves fourteen plaintext copies while the card
shows thirteen expired — a deletion the next run will not take.

The writer now prunes each class at its own count, with the defaults imported
from scripts/lib/backup-manifest.mjs so there is one definition rather than a
second constant. --keep / BACKUP_KEEP stays an explicit override of both classes
for one run and the retention= line prints the counts it applied.
…1188)

The kept set was chosen by mtime alone, so a plaintext copy with a future mtime
(a clock stepped back, or a set restored with `cp -a`) outranked the copy the
run had just written: the writer pruned its own backup, printed a `backup=` path
for a file that no longer existed, and still said `backup: OK`.

`prune` now pins the run's target first inside its class before slicing, so the
written copy is kept even when it is not the newest by mtime, and the run
refuses after pruning if that path is gone instead of reporting success for a
file no reader can open.

`--keep` / `BACKUP_KEEP` is validated whenever it was supplied: an explicitly
empty value is refused with exit 2 instead of falling through to the defaults,
and an accepted override still prints `override=--keep`.
@HugoXK

HugoXK commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

Security: PASS at 539ea74

Replaces: b062bb9

  • scripts/backup.mjs:101-111 — the pin matches on the same outDir string prune builds entries from, so class and normalisation cannot defeat it: at the shipped defaults the fresh copy survives and the future-dated copy is taken (pruned=1), with --backup-dir written as …/backups/, …/backups/., …//backups, through a symlinked directory, and on a keyed run holding thirty future-dated .db.enc copies; the class still ends at its count.
  • scripts/backup.mjs:229 — the guard is a second line, not the only one: with the pin removed the run takes its own copy and then exits 1 with the backup this run wrote is missing after pruning, printing no backup=; with the pin in place the guard never fires.
  • scripts/backup.mjs:55-63,125-127 — --keep "" and BACKUP_KEEP="" exit 2 with the refusal and touch nothing (name/size/mtime snapshot unchanged), BACKUP_KEEP=" " still exits 2, and a dangling --keep now refuses where it used to fall through to the environment — a tightening, not a widening; BACKUP_KEEP=14 still applies to both classes and prints override=--keep.
  • test/backupPrune.test.ts:360-414 — both new tests are red at the replaced sha for the stated reason (the reported backup is missing; the empty override is accepted and three copies are deleted) and pass here; the future test also fails a guard-only variant (exit 1), and the override test pins the message plus a full-directory snapshot, so neither passes for the wrong reason.
  • docs/deploy.md:1143-1144 — the "empty value is refused" wording matches the code, and the four-file delta touches no .github/, scripts/ci-ok*, review-gate or permission surface.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS at 539ea74

I read the live PR myself: head 539ea744 (tree 72ac6e60, parent b062bb94), body raw 98ee299c1b2d (12,876 B), and no review object on the PR. The earlier 8a0095af body is not live.

  • Blocker 1, read and reproduced: prune(dir, keep, target) moves the entry whose path equals target to the front of its own class before sorted.slice(keep[kind]) (scripts/backup.mjs:101-105), and the run refuses after pruning when that path is gone (:229, backup: the backup this run wrote is missing after pruning). With this tip's test file on the b062bb94 writer, a copy with a future mtime never outranks the copy the run writes fails (12 tests / 10 pass / 2 fail: the run reports a backup= path it already deleted); at the tip that node passes and the future-dated copy is the one removed.
  • Blocker 2, read and reproduced: supplied is separated from non-empty (Object.hasOwn(args, 'keep') || process.env.BACKUP_KEEP !== undefined, :125-127) and validated by keepCount (:55-63) before anything is created. At the tip --keep "", BACKUP_KEEP="" and a bare --keep all exit 2 with --keep must be a positive whole number, got an empty value / got no value; at b062bb94 the same node fails 0 !== 2 with the directory untouched afterwards. BACKUP_KEEP=2 is accepted and prints retention=keep:plaintext=2,encrypted=2 override=--keep.
  • The counts are the console's counts: DEFAULT_KEEP_PLAINTEXT 1 / DEFAULT_KEEP_ENCRYPTED 30 are imported from scripts/lib/backup-manifest.mjs:98-99, the same constants src/config.ts:677-679 and the dry run read; the per-class variables keep the documented empty-means-default reading (src/config.ts:304-309), and only the one-run --keep/BACKUP_KEEP override refuses an empty value. The retention=keep:plaintext=N,encrypted=N[ override=--keep] line is what the tests assert.
  • Red/green reproduced myself: f1c48ed3 with the original nine-node file → 9 tests / 8 pass / 1 fail; b062bb94 with the twelve-node file → 12 / 10 / 2; tip → 12 / 12 / 0; the five backup suites at the tip → 57 passed, 0 failed. ci-ok 136467507790 is green at the head, every other check is success, and the review-gate red is the pre-PASS state. T3 basis scripts/backup.mjs (4 files, +248/−29, inside the advisory limits); 无 UI 面 — CLI only. The docs and changelog changes state the per-class counts, the override of both, the empty-value refusal and the written-copy guarantee, so nothing is left unnamed.

Ran:

  • npx tsx --test test/backupPrune.test.ts at 539ea744 — 12 pass / 0 fail
  • npx tsx --test test/backupPrune.test.ts test/backup.test.ts test/backupPlaintext.test.ts test/backupEncryption.test.ts test/adminBackup.test.ts at 539ea744 — 57 pass / 0 fail
  • npx tsx --test test/backupPrune.test.ts with this test file on b062bb94 — 12 tests, 10 pass, 2 fail
  • npx tsx --test test/backupPrune.test.ts with the original nine-node file on f1c48ed3 — 9 tests, 8 pass, 1 fail
  • node scripts/backup.mjs ... --keep / --keep "" / BACKUP_KEEP="" / BACKUP_KEEP=2 at 539ea744 — exit 2 for the three empty shapes, exit 0 with the override marker for 2

Not run:

  • npm run check at the tip (CI test (24), coverage, typecheck, lint, build are green; I ran the five backup suites)
  • a real host backup or any production retention change — every directory was temporary
  • the keyed encryption path beyond test/backupEncryption.test.ts

Replaces: b062bb9

Written-by: HugoXK-bot

@HugoXK-bot HugoXK-bot added the review:approved Issue reviewed and approved by the Fledge Platform Reviewer label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved Issue reviewed and approved by the Fledge Platform Reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(backup): the writer prunes by BACKUP_KEEP while the console reports the per-class counts

2 participants