docs(genesis-writer): replace the rollout runbook with the sequenced cutover plan - #550
Open
rickyrombo wants to merge 13 commits into
Open
docs(genesis-writer): replace the rollout runbook with the sequenced cutover plan#550rickyrombo wants to merge 13 commits into
rickyrombo wants to merge 13 commits into
Conversation
A verify node started against the pristine chain DB with a binary that did not embed the artifact's genesis took its "generate new genesis" path and ran the core migrations down, destroying the writer output. OPENAUDIO_ENV=dev loads the embedded dev.json, so the binary looks normal right up until it drops the tables. Document sealing the artifact with ALLOW_CONNECTIONS false and serving from a TEMPLATE copy, so no connection string can reach the pristine write, and note that artifact verification belongs after services start, not before.
…s real prerequisites Five fixes from a read-through: - The runbook told the operator to confirm block 5,819. Both verification artifacts end at 5,839, so the check inverted: a healthy chain reads as the dead-chain failure mode. Every prod run ends at a different height, so the check now points at the writer's reported height instead of any literal. - The writer's prerequisites were undocumented. rewards is its FIRST step and needs --core-dsn to the old core chain, the launchpad secrets from env, and the mints file; the destination database must exist and be empty in both Postgres and its CometBFT directory. Anything missing kills a ten-hour run in the first second. - Step 3 had no pass criterion, and parity legitimately reports divergences -- including several where the migration is more correct than the source. Lists the expected set so a reviewer does not flag the migration for being right. - The ~10h writer estimate is unsourced; verification runs took 3h09m and 3h36m. Says to re-derive rather than substituting another machine's number. - Step 13 fell outside the reversibility framing, but it retires the two old-chain state-sync servers and so ends the cheap rollback path.
…plan The previous runbook ordered the work but left the two hardest transitions underspecified: how plays avoid being split across chains while the fleet is migrating, and how the indexer moves chains without duplicating or dropping writes. Sixteen steps now, with the indexer switch pinned to a height L chosen in the future so it does not race the config rollout, and the flusher filtered below L so the boundary is exact rather than inferred. Plays are routed through two old-network nodes for the duration, which keeps them on the chain the indexer is actually reading. Records what each irreversible step costs: 14 is the point of no return, 15 retires the rollback anchors, and jailing stops entirely once the old network is down to ~30 active validators -- after which a halt cannot be recovered by waiting.
…d with the code Six places where the reference either contradicted the runbook or stated something the code does not do: - The facts table gave prod's validator count as 9. That is the genesis list; the live set is 67 of 72 registered nodes, and the runbook's quorum and jailing math is over the live set. - Section 7 said to leave BlockInterval at its default, which the runbook now overrides with 20,000. Records why the default is wrong here, that it is a producer-side setting, and that retention is Keep x BlockInterval so it cannot be lowered freely. - The state-sync truncation table listed blobs but not uploads, audio_previews or qm_audio_analyses, which are also wiped. Adds them with how each recovers, and notes #551 makes the row moot. - Section 3 asserted the bootstrap validator holds power 100. Its own registration rewrites that to ValidatorVotingPower, 10 on mainnet, because the writer never seeds core_registered_nodes. - The cursor discussion now points at api#1018, which is open and must land before flushing is first enabled. - Section 11 omitted the chain-aware fallback to core_indexed_blocks, which is only reached on ErrNoRows and so never runs on a database that has indexed the old chain. That is why the failure is a silent stall.
…ld block the rollout The reference suggested refusing to start when the validator key disagrees with genesis. New-chain genesis lists one validator and every other node joins by registration, so that check would stop the entire fleet from ever reaching the new chain -- it is the rollout mechanism, not a fault. It also named ensureGenesisFiles, which does not exist. The genesis-file handling is setup.go:114-126 and compares nothing, correctly. The check worth having is already there: ensurePrivValidator compares the key file against the key derived from the delegate key and refuses only when prior signing history makes a mismatch a double-sign risk. The T7 node that motivated the suggestion was behaving correctly -- a node whose key is not in genesis block-syncs and does not propose. It read as a dead chain only because it was the sole node. Replaces the recommendation with a diagnostic: log when the key is absent from the genesis set and say the node will not propose until registered.
…e new binary Reference SS5 said the bootstrap node would dial old-network hosts and should suppress them with OPENAUDIO_PERSISTENT_PEERS, no release needed. That assumed the node runs the existing binary with only genesis swapped. Runbook step 3 builds a binary whose ProdPersistentPeers is already the new bootstrap nodes, so there are no old-network hosts in the list and nothing to suppress. Outbound dialing of the old network does not happen on the step 3 binary. Inbound does, and no env var here changes it: old nodes hold this host in their address books and keep dialing until those entries age out. Those connections are rejected at the handshake on the network mismatch. The env override at step 4 is still worth setting, but for the narrower reason now stated there: the second bootstrap node does not exist until step 7, so the baked list would otherwise churn against a host that is not up.
Neither bound is consumed by the ETL; both are re-read on every startup. Left set, a start height makes each restart re-index from it and duplicate plays, and an end height stalls the indexer at L permanently.
Sixteen steps across three phases is more than fits in someone's head, and the irreversible ones are buried at the end of a long document. One table, the three failure modes that actually bite, and the PRs that must merge first.
Every step that cannot be executed without a code change now names it, so the dependency is visible where the work happens rather than only in a list at the top: #553 for the binary (step 3), api#1029 for play routing (5 and 13), #551 before the fleet state syncs (10), api#1018 before flushing (11), api#1028 for the indexer bounds (12). Also records two things found while deriving the bootstrap's identity for #553: the node key is the comet key, so a node's P2P id is its validator address and the bootstrap's is derivable before it runs; and the bootstrap must not inherit node_key.json from the writer output, or its id will not match the peer list.
Someone reading only this document would have hit each of these cold: - how to actually run the replay and parity, including that --db must be an ETL database and the replay must point at the serve copy, not the sealed original - calibration figures from the 2026-08-25 run, framed as sanity checks rather than expected values, since a fresh snapshot changes all of them - the row-level parity mismatches that are not in the divergence table: genre re-casing, musical_key dropped by the allowlist, and track_downloads disagreeing on parent_track_id for deleted tracks. All pre-existing ETL behaviour; every missing row should reconcile to a listed cause - why plays need routing at all: logTrackListen runs when a node serves audio, and plays never enter the relay queue, so a migrated node writes them to a chain nobody is indexing - the arithmetic behind ten-at-a-time: eligibility is ~4.5h of proposing nothing, not warden throughput, and attestation quorum never binds before the chain has already halted - that the routine DP backup is not restorable as taken, and that user_balance_history is ~85 GB the migration never reads
The document had grown to where the instructions and the reasoning were the same text, and an operator working through it had to read an argument to find an action. Three parts now. The overview says what the migration is and why it is awkward, with a linked table of contents. The steps are imperative -- 22 to 110 words each, pointing at the appendix rather than explaining inline. The appendix holds the reasoning: nine sections keyed to the steps that need them, then the existing reference material. No content dropped. Every explanation that was inline is now an appendix section with an anchor, and all 26 cross-links resolve.
The overview described the mechanism without saying what it is for. audius-mainnet-alpha-beta began part-way through the protocol's life, so most of the network's history -- the majority of users, tracks, playlists, follows, saves, reposts and plays -- is not on it and lives only in the application database. Reconstructing state from that chain alone yields a fraction of it. That is the reason the new chain is written rather than synced: syncing would carry the same gap forward. It is also why verification dominates step 1, since the artifact is the history and there is no second source once it ships. Also drops 'Discovery Provider' for 'production snapshot', per the terminology note in CLAUDE.md.
Thirteen references across the document were plain text, so following one meant knowing which repo it belonged to. api#N now resolves to AudiusProject/api and bare #N to OpenAudio/go-openaudio, which is also the disambiguation the plain text was silently relying on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rewrites the genesis migration runbook as a sequenced sixteen-step cutover, and reconciles the reference sections with it and with the code.
The runbook
The previous version ordered the work but left the two hardest transitions underspecified.
Plays were going to be split across chains. They're submitted by whichever node serves the audio — never through the relay — so once nodes start migrating, each one writes its plays to whichever chain it happens to be on while the indexer still reads the old one. The fleet migration takes days, so that's days of plays written where nobody is reading. Step 5 routes all plays through two old-network nodes for the duration; step 13 reverts it.
The indexer switch had no procedure. It's now pinned to a height
Lchosen in the future, with the flusher filtered below it:Lin the future so the stop doesn't race the config rollout. Choosing one at or below the current tip means the indexer sails past it, and those blocks get indexed from the old chain and flushed to the new one.confirmed_blockis only roughly ordered byid; stopping at the first row aboveLwould strand one below it.Lreach zero and stay zero across a settle window, since a straggler can appear after the queue first reads empty.That yields an exact boundary: the old indexer covers everything at or below
L, the new one everything aboveH, and the ceiling guarantees nothing aboveLreached the new chain belowH. No pause, and no dependence on the two switches happening at the same instant.Step 13 also records that reverting the play routing loses a small number of plays on either side of
Land duplicates none — plays have no unique key, so there is no dedupe path — and that after is the better default because the window is then bounded by the drain.Reference reconciliation
Six places contradicted the runbook or the code:
BlockInterval. Section 7 said to leave it at the default, which the runbook now overrides with 20,000. Adds why the default is wrong here (no snapshot exists until height 100,000), that it's producer-side so it needs no fleet agreement, and that retention isKeep × BlockIntervalso it can't be lowered freely.blobsbut notuploads,audio_previews, orqm_audio_analyses, which are also wiped. Adds them with how each recovers, and notes fix(core): scope the state-sync truncate to the tables the snapshot restores #551 makes the row moot.ValidatorVotingPower— 10 on mainnet — because the writer never seedscore_registered_nodes.core_indexed_blocks. It's only reached onErrNoRows, so it never runs on a database that has indexed the old chain — which is why the failure is a silent stall rather than an error.Earlier corrections in this branch
5,819; both verification artifacts end at 5,839, so an operator following it against a healthy chain would diagnose the dead-chain failure mode. Now points at the height the writer reports, since every run against fresh data ends somewhere different.ALLOW_CONNECTIONS falseplus aTEMPLATEcopy makes the pristine write unreachable by any connection string.--core-dsn, launchpad secrets, mints file, and a destination empty in both Postgres and its CometBFT directory.Scope
Documentation only. No code changes. Code prerequisites for steps 11 and 12 are deliberately out of the runbook and will be handled before it's run.