fix(aster-srv): serve /healthz while migrations wait for table locks

[calvin-archastro]

Review on ArchCode

Problem

Production incident 2026-10-17 (firstlanding auto_apply_production run 39067438593, rolling aster-srv 0.20.0 to 0.21.0):

new pod starts -> App::start -> db::migrate -> migration 5: ALTER TABLE
  test_results.classifier_state / quarantine   (needs ACCESS EXCLUSIVE)
  blocked by 0.20.0 pods still running their classifier pass (open transaction on those tables)
  5s lock_timeout x 11 attempts -> "canceling statement due to lock timeout"
no listener bound meanwhile -> liveness /healthz fails (10s + 3x20s) -> SIGKILL (137)
after 12 attempts the process exits 1
kubelet restarts it (8 restarts); each restart repeats the wait
Pulumi's 600s Deployment wait times out and the apply fails

It converged once the old pods' passes stopped holding the tables. Invariant broken: the process listened only after migrations finished, so any lock wait longer than the liveness window killed it, and the retry cap turned a transient wait into an exit.

What changed

Before: App::start migrates, then main binds. After: App::connect builds the app with no DB schema work, main binds immediately, and migrations run in a background task that retries every 5s without exiting, then starts the retention, classifier and webhook loops.

• /healthz is 200 throughout, so the liveness probe does not kill the pod.
• /readyz and every other route return 503 until App::migrate succeeds (App::schema_ready), so the pod receives no traffic against an old schema.
• App::start* still migrates before returning, so tests and other callers are unchanged.
• The retry loop is App::migrate_until_ready in lib.rs (main only wires logging and the loops to start). Table-lock waits and deadlocks (db::is_lock_contention) log a warning until they have lasted 10 minutes, then error. Any other migration error logs at error immediately. It never exits.
• Pacing: 5s after the first failed pass, 30s after later ones. Each pass is already up to 12 attempts of a 5s lock_timeout, and a waiting ALTER queues other sessions behind it, so the longer pause cuts how often the previous release's readers stall.
• Unchanged: migration SQL, per-attempt lock_timeout, the advisory lock.

Scope / risk / impact

• Backend only, crates/aster-srv. Low risk. One behavior change: a persistent migration error (for example a bad SQL migration) now keeps the pod running but not ready and logs migrations not applied yet; retrying, instead of exiting.
• No user impact. Rolling deploys no longer crash-loop on lock waits.

Testing

• Canonical end-to-end proof: crates/aster-srv/tests/startup.rs, the_server_stays_alive_while_a_migration_waits_for_a_table_lock. It uses real Postgres 16 and real HTTP. A second connection holds ACCESS SHARE on test_results.quarantine (the old release) while migration 5 runs. It asserts /healthz 200, /readyz 503 and a data route 503, then releases the lock and asserts /readyz 200.
• startup.rs also has: background_loops_start_only_after_the_schema_is_ready (lock held: warnings only, on_ready not called, then called after release) and a_migration_error_that_is_not_a_lock_wait_logs_an_error_and_stays_unready (no DB listening: error severity on each pass, keeps retrying, unready, loops never start; needs no Postgres).
• cargo test --locked -p aster-srv with ASTER_SRV_TEST_DATABASE_URL set: all pass. cargo clippy -p aster-srv --all-targets -D warnings and cargo fmt are clean.

Follow-ups

• The lock holder is the classifier pass, one long transaction touching classifier_state/quarantine. 0.21.0 already shortens it (4977ffb); this PR does not change it.
• firstlanding infra/services/aster-srv: a startupProbe is optional after this change.

🤖 Generated with Claude Code

[calvin-archastro]

ArchDev is watching this pull request. State: settled (cycle 2 of 2 for head 243ec42).

when	trigger	decision	result
2026-10-18T17:41:24.242Z	review feedback	enqueue job_232e5be9be54723d547deb75	published 6f02036
2026-10-18T18:06:02.477Z	review feedback	enqueue job_23834ec66fee0557eac14b36	published 243ec42

Paused by the no-autofix label or a draft; never resolves reviewer threads. Jobs: job_232e5be9be54723d547deb75, job_23834ec66fee0557eac14b36.