[Marketen]

Draft. Stacked on #70. Shows what a PostgreSQL 14 → 18 upgrade looks like for the package. Independent of #71.

Why

• Every variant runs postgres:14.1 (Nov 2021), which is missing about 23 minor releases of fixes.
• PostgreSQL 14 reaches end of life in Nov 2026.

What the DB holds

Only web3signer's slashing-protection data:

• validator public keys;
• signed blocks and attestations;
• watermarks, genesis validators root, and schema version.

The data is pruned to 500 epochs, so it stays small. Keystores (web3signer_data) and brain data (brain_data) live in other volumes and are not touched.

Changes

• services/postgres/Dockerfile.debian (mainnet, gnosis, lukso):
  • Base image is now postgres:18.6-bookworm. Still Debian, still uid 999.
  • The PostgreSQL 14 binaries are installed too (from the PGDG repo the image already uses). They're used only to read old clusters.
• services/postgres/upgrade-from-14.sh, a wrapper around the official docker-entrypoint.sh:
  1. If it finds a PG 14 cluster at the volume root, it starts PG 14 socket-only and pg_dumps every database.
  2. It initdbs a PG 18 cluster in a temporary directory and pg_restores into it.
  3. It compares exact row counts of every table in both clusters. On any mismatch it aborts and leaves PG 14 untouched.
  4. It atomically renames the new cluster into PGDATA.
  5. Fail closed: it renames the old cluster's PG_VERSION, so an older package version can't start Postgres on stale slashing history. The dumps are kept in /var/lib/postgresql/upgrade-14-to-18/ for manual recovery.
  • On any error the script exits non-zero before step 4. Postgres stays down, so web3signer can't sign, and the next restart retries from scratch.
• Volume mount: postgres_data is now mounted at /var/lib/postgresql, the layout the 18+ images require. They refuse to start with anything mounted at /var/lib/postgresql/data. The mount moved from the root compose into each variant.
• Hoodi keeps Alpine 14 at /var/lib/postgresql/data. Its uid-70 cluster would need its own migration.
• Why dump/restore and not pg_upgrade: the DB is tiny, and a logical dump has no on-disk, checksum or collation coupling between versions.

Testing (harness from #70: real on-chain 2.0.17 images, DAppNode-like network aliases)

Scenario	Result
2.0.17 (PG 14.1) with key and slashing history → this build	✅ upgrade log shows dump, restore, matching row counts, old cluster disabled. Server is 18.6, slashing DB identical, flyway exits 0, key loaded, double/surround votes from PG 14 history → 412, new attestation signed and persisted on 18, brain reaches the signer
Restart postgres after the upgrade	✅ upgrade not re-run, data intact
Roll back to 2.0.17 after the upgrade	✅ fails closed. 2.0.17's Postgres refuses to start (initdb: directory "/var/lib/postgresql/data" exists but is not empty). Web3signer crash-loops (Failed to initialize pool). Every signing request fails, including a double vote against history recorded only on PG 18. Volume contents unchanged. (2.0.17's flyway hides the psql failure and exits 0, so web3signer does start, but it can't run without its DB.)
Re-upgrade after that rollback	✅ healthy, PG 18 history still protects (double vote → 412), new signing works
Injected failure mid-upgrade (unreadable signed_attestations file)	✅ pg_dump fails with Permission denied, wrapper aborts. PG 14 cluster untouched (PG_VERSION still 14), no PG 18 cluster in place, web3signer not running. After the fix, the next start retries and upgrades cleanly with an identical slashing DB
Fresh install	✅ PG 18.6, schema V12 from the initdb migrations, upgrade path not taken, import + signing + surround refusal work
Flyway V11 → V12 migration on PG 18	✅ baselines at 11 and applies V12. ⚠️ Flyway 11.10 warns PostgreSQL 18.6 is newer than this version of Flyway and support has not been tested. See open questions

Open questions before this leaves draft

• Rollback: after the upgrade, going back to a PG 14 package version needs manual recovery (rename PG_VERSION.upgraded-to-18 back, then restore from the kept dump). Is that acceptable, or should the release that ships this be marked non-rollbackable?
• Hoodi: bring it to Debian + 18 too (needs a uid 70 → 999 chown step)?
• Flyway: bump flyway/flyway (11.10 → a release that officially supports PostgreSQL 18; current tags are 13.x) and re-run the V11 → V12 test.
• Mainnet: ship Fix mainnet variant so legacy web3signer users can migrate to the generic package #70 first and let it settle before adding a Postgres major upgrade on top.

Real-node notes

• The automatic DAppManager rollback after a downgrade attempt brings the package back on Postgres 18 with all history. It takes the full 15-minute compose timeout, during which the signer is down (fail-closed).
• Unclean shutdown before the upgrade (power loss leaves postmaster.pid with PID 1) was tested locally: the upgrade still succeeds.

Validation of every release order (#70, #71, #72)

I tested every state main can end up in:

State	Contains
S1	#70
S2	#70 + #71
S3	#70 + #72
S4	all three

Local matrix (arm64), for mainnet, gnosis, lukso and hoodi:

• Each variant starts from its latest on-chain release (2.0.17 / 1.0.15 / 0.1.5 / 0.1.3), seeded with a key and an EIP-3076 slashing history.
• 6 upgrade chains per variant cover every merge order. Each ends with a rollback and a re-upgrade.
• Extras per state: fresh install, flyway V11→V12 migration, and an injected failure during the Postgres upgrade.
• At every hop I check: slashing DB identical, keyfiles and brain-db preserved, double/surround votes refused (412), new attestation signed and saved, flyway never restarted, allowlist accepts/rejects as expected.
• 41 runs, 2,075 checks, 0 failures.

Real DAppNode (amd64, DAppManager 0.2.129, Docker 29.9, Compose v5.6), installed through the DAppManager installer:

Test	Result
Live web3signer-hoodi 0.1.5 (20 remote keys in Lighthouse) → S4	✅ 31/31: all 14 container DNS names accepted (0.1.5 rejected 9), keys/brain-db/slashing DB unchanged, Lighthouse still sees its 20 remote keys, 0 rejections
lukso 0.1.5 (published bytes) → S4	✅ 17/17: Postgres 14→18 upgrade, data identical, slashing protection kept
lukso S4 → published 0.1.5 (downgrade)	✅ fails closed: old Postgres refuses the disabled cluster, web3signer never starts, nothing signs. DAppManager times out (15 min) and restores S4 by itself; 5/5 after
mainnet legacy 2.0.17 (on-chain registry install) → S1	✅ migration: same Postgres 14 cluster (uid 999), slashing DB identical, legacy keystore loaded, double/surround refused, flyway 0 restarts
mainnet S1 → S4	✅ 17/17: Postgres 14→18 of the cluster legacy 2.0.17 created, strict allowlist, all 13 container DNS names accepted

Release risk outside these PRs: DAppNode's IPFS gateways sometimes cut the trustless ?format=car download of large images partway (terminated). On the test node, 3 registry installs failed this way. I reproduced it once from another machine, and later attempts succeeded. It fails safe (nothing is installed), but expect some "download failed" reports right after publishing; the infra team should look at it.

🤖 Generated with Claude Code