Skip to content

test(deploy): read the deploy-lock holder instead of probing it per iteration - #1197

Open
HugoXK wants to merge 1 commit into
mainfrom
dev1/1193-deploy-lock-wait-seam
Open

HugoXK wants to merge 1 commit into
mainfrom
dev1/1193-deploy-lock-wait-seam

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Written-by: Fledge Platform Dev-1

Fixes #1193

What changed

test/deployAutomation.test.ts only: one readiness wait, one new case, +159/−13. deploy/fledge-deploy-tag.sh is untouched, so production semantics are unchanged — LOCK=${FLEDGE_DEPLOY_LOCK:-/tmp/fledge-deploy-tag.lock} and the flock -n early exit 0 still stand, and the case still asserts both (:349, :371-372).

The readiness wait no longer spawns a process per iteration (#1193). The holder is flock -n <lock> sh -c 'printf "ready\n"; exec sleep 30': it prints one readiness line only after flock owns the lock, and the wait resolves on that child's 'data' event, so observing readiness costs no fork. The deadline is the named constant LOCK_READY_TIMEOUT_MS (:272, 20 s) whose comment says what it is for; LOCK_READY_TICK_MS (:273) is the state-observation tick, not a probe interval.

On give-up the message names what it saw instead of concluding from an absence — elapsed ms, poll iterations, the last probe status, the holder pid and alive/exited:

the lock holder never signalled readiness after 5 poll(s) in 61 ms; last probe status 0, holder pid 10 alive

The probe exists only on that path (probeLock, :283): zero probes when the holder is ready, one when it gives up.

Red evidence, assertion by assertion

The new case cannot run at all against the literal f1c48ed3 file — there is no seam to call — so the red was taken against the base mechanism expressed through the same seam: a throwaway copy of the base file plus a waitForLockReady that runs the old spawnSync probe loop, 5 s budget and bare message, with the three new assertions as three separate cases. All three red, each on its own assertion:

  • A slow holder still passes → the wait must outlast the old 5 s budget, reported 1000 ms. The old loop measures against its own budget and ends as soon as one probe is refused, so it cannot run past the budget it is given.
  • A real timeout names what it saw → The input did not match the regular expression /after \d+ poll\(s\) in \d+ ms; last probe status \d+, holder pid \d+ (alive|exited \d+)/. Input: 'the lock holder never took the lock' — the literal defect message from the two CI failures.
  • A ready holder costs no probe → a readiness wait on a ready holder must not spawn a probe — the old loop probes once per iteration by construction.

Green at this tip: the same three assertions pass inside the new case (all three run, none skipped), and the whole file is 15/15.

Assertions kept

The old case is now async and keeps every assertion with its meaning: :349 (production lock line), :351, :353 (two hosts do not share a lock), :371-372 (a lock-blocked tick exits 0 and logs nothing), :380-382 (another host still reaches its refusal). The restore and rollback cases are untouched (:241-263, :461-467). No skip, no retry wrapper, no process.env.CI/NODE_ENV/version branch, no second attempt inside the case, and test/deployAutomation.test.ts:3 is untouched (spawnSync is still used by tick and by the give-up probe).

One consequence worth naming: each holder in the new case gets its own lock path (ready.lock), because kill() on a holder does not release its lock within the same process tick on this host — the next case must not be decided by a kill that has not landed yet.

Ran:

  • npm run check at 1fbb525 — exit 0, # tests 3265 / # pass 3265 / # fail 0, lint 11 warnings at the baseline, tsc --noEmit and build clean.
  • npx tsx --test test/deployAutomation.test.ts at this tip — 15/15, duration ~34 s.
  • The red harness above — 3/3 red on the old mechanism, each on its own assertion.
  • node scripts/risk-gate.mjs --base origin/main — risk-tier: T1; basis: test/deployAutomation.test.ts.

Not run:

  • No docs/ change and no changelog fragment: this is a test-determinism fix with no user-visible behaviour (the #992 exemption). Docs-not-needed: test-only change, the deploy runbook's behaviour is unchanged. Changelog-not-needed: no user-visible change.
  • Not run: a real CI runner under load (the evidence for the flake is the body of #1193), Docker image, and any production host.

Risk

T1, test-only. Residual: the deadline is still a wall-clock budget, so a runner stalled for longer than 20 s inside the wait still fails — but it now fails with a message that separates "lock never seen" from "probe saw a holder it never heard from", which is the part that cost three CI attempts to learn nothing.

Tip 1fbb525729ef3434506334f051f9fde85a9f14d7, tree 6b366ed2b78bb8806c901814fe0170a4729f7e88, base f1c48ed3513a8180b870fae19fb95d32222da44b.

#1193)

The deploy-lock case proved ownership with a `spawnSync` probe per iteration
under a hard 5 s deadline, so a loaded runner could spend the whole budget
without ever seeing the holder; the give-up then said only that the holder
"never took the lock", a conclusion drawn from an absence. `ci-ok` went red
twice on the same sha for this case and green on a third attempt of the same
sha.

Readiness now arrives on the holder's own stdout, which costs no process per
iteration. The deadline is the named constant `LOCK_READY_TIMEOUT_MS`, and the
give-up probes once so its message names elapsed ms, poll iterations, the last
probe status, the holder's pid and whether it was alive. Every existing
assertion keeps its meaning, and `deploy/fledge-deploy-tag.sh` is untouched.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(deploy): the deploy-lock readiness wait is a spawn poll under a hard 5 s deadline

1 participant