Repository navigation
fix(deploy)!: make the shipped deployment boot on mainnet, and cut the docs to what works - #696
Merged
Merged
Conversation
mfw78
force-pushed
the
docs/restore-operator-docs
branch
from
October 12, 2026 03:27
cfe70eb to
b9e30f4
Compare
Shepherd's `main` carried no `docs/` directory: the carve rewrote history and the operator documentation stayed behind on the pre-carve line. About twenty tracked files still pointed into it, and the compose observability profile mounted a `prometheus.yml` that did not exist. This restores the deployment guide, the Docker guide, the multi-chain guide, the Prometheus scrape config and the soak runbook, rewritten against the current tree. The old text described `[[subscription]]` tables, `module.toml`, `twap-monitor`, the upstream ComposableCoW registry and `shepherd_` metric names, none of which exist here now. Engine-level topics link to the nexum-runtime operator handbook instead of repeating it. The guides describe the shipped files as they are, including where they do not boot as committed: several `engine.*.toml` files declare only chain 1 while `ethflow-watcher` triggers on Sepolia, and the soak compose file forwards the wrong RPC variable. Those files are not changed here. The engine substitutes a dollar-brace token before it parses the config, comments included, so a comment that spelled one out as an example made `engine.docker.toml` and `engine.example.toml` refuse to boot for a missing variable. Those comments are reworded, which is the one change here that alters behaviour. Every other edit to a config, compose or build file is a comment that pointed at a missing document. AI Assistance: Claude Code used to restore and rewrite the documents and to verify the commands, refusal strings and compose files against the tree.
The engine prints the full RPC URL at each start and can repeat it in an RPC error line. The runbook now tells the operator to keep the raw log private and to attach a copy with the endpoints replaced. AI Assistance: Claude Code used for the runbook text and the scrub commands.
…e docs to what works The operator docs explained at length things that did not work: six engine configs that could not boot, a soak harness that needed local patches, and an EthFlow module on Sepolia beside a venue on mainnet. This fixes or removes those things, and the docs now describe one tested path. The shipped files now form one consistent mainnet deployment. `ethflow-watcher` reads the production EthFlow contract on chain 1, and its first boot opens at head, because it only observes and the contract's history is of no use to it. `engine.docker.toml`, `docker-compose.yml` and `.env.example` need one RPC endpoint, not five. Compose now waits 180 seconds on a stop, which is longer than the engine's drain, and bounds the container log. A test holds `engine.docker.toml` to the chain that the shipped manifests read. Removed, because they could not run: `engine.m2.toml`, `engine.m3.toml`, `engine.e2e.toml`, `engine.load.toml`, `engine.soak.toml`, `engine.soak.docker.toml`, `docker-compose.soak.yml`, and the e2e, load and soak scripts with their justfile recipes. The forked-mainnet engine test covers what the e2e scripts were for. The multi-chain guide is folded into a short section of the deployment guide. The soak runbook now records a run with the Compose Prometheus profile and a set of queries, which also apply to any other Prometheus. BREAKING CHANGE: the shipped `ethflow-watcher` manifest targets chain 1 where it targeted chain 11155111, and `engine.docker.toml` declares chain 1 only. AI Assistance: Claude Code used for the review, the changes, the tests and the documentation, and for the live Docker test of the documented steps.
…entation The chain section of the deployment guide becomes "Multi-chain deployment". It names the two patterns (one engine process for each chain, one process for many chains), gives a worked example that runs ethflow-watcher on Gnosis Chain, shows the chain tables for Ethereum Mainnet, Gnosis Chain, Base and Arbitrum One, and states what is tested and what is not. A new test holds those chain tables to the pinned config parser and makes sure that the venue accepts each of the four chains. The README gets a documentation map that links the operator guides here and the module tutorial, the SDK guide, the testing guide, the manifest reference and the operator handbook in nexum-runtime. The Gnosis Chain example was started one time by hand from a source build: the module became alive, its event trigger opened at the chain head, and the engine stopped cleanly. AI Assistance: Claude Code used for the documentation, the test and the manual run.
…ublic The `ghcr.io/nullislabs/shepherd` package is public, so `docker compose pull` needs no login. The "Run" and "Upgrade" sections of the Docker guide now pull the published image first and give the local build as the alternative. They also say to use a clone and an image of the same commit, because the config comes from the clone and the module manifests come from the image. An anonymous `docker compose pull shepherd` was run and it got the `latest` image. AI Assistance: Claude Code used for the documentation and the pull test.
mfw78
force-pushed
the
docs/restore-operator-docs
branch
from
October 12, 2026 11:35
d72b5e8 to
6bccee9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The first version of this pull request restored the operator docs, and in doing so it documented at length things that did not work: six shipped engine configs that could not boot, a soak harness that needed two local patch files before it started, and an EthFlow module that read Sepolia beside a venue on mainnet. A document that explains how to work round the repository is the wrong fix. This version repairs or removes those things, and the docs describe one path that was run for real.
Merge this after the re-pin in #698 (nexum-runtime
15e4a38): the soak runbook readsnexum_runtime_start_time_secondsandnexum_runtime_delivery_latency_seconds, which that runtime adds.What changed
The shipped files form one consistent mainnet deployment.
modules/ethflow-watcher/component.tomlreads the production EthFlow contract on chain 1. The address isETH_FLOW_PRODUCTIONin the pinned cowprotocol crate, and it has code on mainnet. The module takes its chain from each log, so no module code changes.engine.docker.toml,docker-compose.ymland.env.exampleneed one RPC endpoint (MAINNET_RPC_URL), not five.docker-compose.ymlwaits 180 seconds on a stop, longer than the engine's 150 second drain, where it waited 30. It also bounds the container log at 10 files of 100 MB.engine.example.tomlno longer names the wrong binary or a positional argument that does not exist.Removed, because they could not run.
engine.m2.toml,engine.m3.toml,engine.e2e.toml,engine.load.toml,engine.soak.toml,engine.soak.docker.tomlanddocker-compose.soak.yml; thee2e-*,load-*andsoak-snapshotscripts withlib.sh,env-template, the two Python helpers andscripts/README.md; and therun-m2,run-e2e,e2e,loadandsoak-snapshotrecipes. The configs named modules this repository does not build or chains they did not declare, the e2e scripts targeted a different chain and registry from their config and checked metric names that no longer exist, and the load script needed a tool that nexum-runtime retired. The forked-mainnet engine test (anvil_fork_e2e.rs) covers what the e2e scripts were for. Everything removed is in the history and at thepre-carvetag.Docs. Three files where there were four, and about 590 lines where there were about 1290.
docs/deployment.md: the engine config reference, the two modules, and a "Multi-chain deployment" section that replaces the multi-chain guide. It names the two patterns (one engine process for each chain, which the CoW venue requires, and one process for many chains), lists the supported chains, gives a worked example that runsethflow-watcheron Gnosis Chain, shows the chain tables for Ethereum Mainnet, Gnosis Chain, Base and Arbitrum One, and ends with what is tested and what is not.README.mdgains a documentation map: the operator guides here, and the module tutorial, SDK guide, testing guide, manifest reference and operator handbook in nexum-runtime.docs/deployment/docker.md: build, run, verify, pin the digests, Prometheus, stop, upgrade, backup.docs/operations/soak-runbook.md: a soak is the Compose deployment with its Prometheus profile, plus a table of queries (uptime, engine restarts, module state, quarantine, faults, delivery latency, handler latency, memory, reconnects, RPC errors) and the steps that save the evidence. The queries apply to any Prometheus that scrapes the engine. It keeps the step that removes the RPC endpoint from a log before the log is published.Tests.
operator_configs.rsgains a test that holdsengine.docker.tomlto the shipped manifests: every chain a trigger reads is declared, and it is the venue's chain. Parsing alone let a config through that named the wrong chain.operator_configs.rsalso holds the chain tables of the multi-chain section to the pinned parser, and makes sure that the venue accepts each of Ethereum Mainnet, Gnosis Chain, Base and Arbitrum One.module_manifests.rsno longer demands astart_blockon the EthFlow trigger. It pins the registry's deployment block forccow-monitoras before, and it pins the EthFlow trigger to chain 1, the production address,resume, and nostart_block.Worth a second look
ethflow-watcheropens at head on its first boot. The earlier manifest backfilled from the contract's Sepolia deployment block. On mainnet that would read the contract's whole history and put every past order under watch, all long settled or expired. The module only observes, so the first boot starts at head andresumecovers every later restart. An order that was open at the moment of the very first boot is not observed.docker stats.docker compose pull shepherdand gives the local build as the alternative. An anonymous pull was run and it worked. The verification below used an image built from this branch: thelatestimage on the registry is the build ofmainbefore this pull request, and its EthFlow manifest still reads Sepolia, so the full pull-then-run path can be run only after this merges andmainpublishes.tools/baseline-latencyis untouched. It targets Sepolia and was not run for this change.Verification
The image was built from this branch on top of the re-pin, and every command in the Docker guide and the soak runbook was run against it with a mainnet node:
/readyzansweredready: truewith both modulesalive, the container wenthealthy,ccow-monitorfinished its first-boot backfill of about 500,000 blocks in 35 seconds, andethflow-watcheropened at head.changes(nexum_runtime_start_time_seconds[7d])answered 1.graceful shutdown complete. After the start that followed, each event trigger opened at its cursor.require_component_digest = false, thesupervisor readyline had"verified":2and noloading unverifiedwarning./readyzansweredready: truewithethflow-watcher: alive, the event trigger opened at the chain head, and the engine exited with code 0.just ciwithCIset passes on this branch: fmt, clippy at-D warnings, the orderbook-only gate, rustdoc, both module wasms, nextest and the doctests.Breaking
The shipped
ethflow-watchermanifest targets chain 1 where it targeted chain 11155111, andengine.docker.tomldeclares chain 1 only. A deployment that used the committed files with a Sepolia endpoint needs its own manifest and config.AI Assistance: Claude Code used for the review, the changes, the tests and the documentation, and for the live Docker test of the documented steps.