Skip to content

fix(deploy)!: make the shipped deployment boot on mainnet, and cut the docs to what works - #696

Merged
mfw78 merged 5 commits into
mainfrom
docs/restore-operator-docs
Oct 12, 2026
Merged

mfw78 merged 5 commits into
mainfrom
docs/restore-operator-docs

Conversation

@mfw78

@mfw78 mfw78 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Why

The first version of this pull request restored the operator docs, and in doing so it documented at length things that did not work: six shipped engine configs that could not boot, a soak harness that needed two local patch files before it started, and an EthFlow module that read Sepolia beside a venue on mainnet. A document that explains how to work round the repository is the wrong fix. This version repairs or removes those things, and the docs describe one path that was run for real.

Merge this after the re-pin in #698 (nexum-runtime 15e4a38): the soak runbook reads nexum_runtime_start_time_seconds and nexum_runtime_delivery_latency_seconds, which that runtime adds.

What changed

The shipped files form one consistent mainnet deployment.

  • modules/ethflow-watcher/component.toml reads the production EthFlow contract on chain 1. The address is ETH_FLOW_PRODUCTION in the pinned cowprotocol crate, and it has code on mainnet. The module takes its chain from each log, so no module code changes.
  • engine.docker.toml, docker-compose.yml and .env.example need one RPC endpoint (MAINNET_RPC_URL), not five.
  • docker-compose.yml waits 180 seconds on a stop, longer than the engine's 150 second drain, where it waited 30. It also bounds the container log at 10 files of 100 MB.
  • engine.example.toml no longer names the wrong binary or a positional argument that does not exist.

Removed, because they could not run. engine.m2.toml, engine.m3.toml, engine.e2e.toml, engine.load.toml, engine.soak.toml, engine.soak.docker.toml and docker-compose.soak.yml; the e2e-*, load-* and soak-snapshot scripts with lib.sh, env-template, the two Python helpers and scripts/README.md; and the run-m2, run-e2e, e2e, load and soak-snapshot recipes. The configs named modules this repository does not build or chains they did not declare, the e2e scripts targeted a different chain and registry from their config and checked metric names that no longer exist, and the load script needed a tool that nexum-runtime retired. The forked-mainnet engine test (anvil_fork_e2e.rs) covers what the e2e scripts were for. Everything removed is in the history and at the pre-carve tag.

Docs. Three files where there were four, and about 590 lines where there were about 1290.

  • docs/deployment.md: the engine config reference, the two modules, and a "Multi-chain deployment" section that replaces the multi-chain guide. It names the two patterns (one engine process for each chain, which the CoW venue requires, and one process for many chains), lists the supported chains, gives a worked example that runs ethflow-watcher on Gnosis Chain, shows the chain tables for Ethereum Mainnet, Gnosis Chain, Base and Arbitrum One, and ends with what is tested and what is not.
  • README.md gains a documentation map: the operator guides here, and the module tutorial, SDK guide, testing guide, manifest reference and operator handbook in nexum-runtime.
  • docs/deployment/docker.md: build, run, verify, pin the digests, Prometheus, stop, upgrade, backup.
  • docs/operations/soak-runbook.md: a soak is the Compose deployment with its Prometheus profile, plus a table of queries (uptime, engine restarts, module state, quarantine, faults, delivery latency, handler latency, memory, reconnects, RPC errors) and the steps that save the evidence. The queries apply to any Prometheus that scrapes the engine. It keeps the step that removes the RPC endpoint from a log before the log is published.

Tests.

  • operator_configs.rs gains a test that holds engine.docker.toml to the shipped manifests: every chain a trigger reads is declared, and it is the venue's chain. Parsing alone let a config through that named the wrong chain.
  • operator_configs.rs also holds the chain tables of the multi-chain section to the pinned parser, and makes sure that the venue accepts each of Ethereum Mainnet, Gnosis Chain, Base and Arbitrum One.
  • module_manifests.rs no longer demands a start_block on the EthFlow trigger. It pins the registry's deployment block for ccow-monitor as before, and it pins the EthFlow trigger to chain 1, the production address, resume, and no start_block.

Worth a second look

  • ethflow-watcher opens at head on its first boot. The earlier manifest backfilled from the contract's Sepolia deployment block. On mainnet that would read the contract's whole history and put every past order under watch, all long settled or expired. The module only observes, so the first boot starts at head and resume covers every later restart. An order that was open at the moment of the very first boot is not observed.
  • The Compose soak harness is gone, with its hourly snapshot container. The runbook records the same evidence through Prometheus. Process memory still has no engine metric, so the runbook reads it from docker stats.
  • Published image. The package is public now, so the Docker guide leads with docker compose pull shepherd and gives the local build as the alternative. An anonymous pull was run and it worked. The verification below used an image built from this branch: the latest image on the registry is the build of main before this pull request, and its EthFlow manifest still reads Sepolia, so the full pull-then-run path can be run only after this merges and main publishes.
  • tools/baseline-latency is untouched. It targets Sepolia and was not run for this change.
  • Multi-chain operation is documented and held to the config parser and the venue by a test, and the Gnosis Chain example was started once by hand. No run of many days was made on a chain other than Ethereum Mainnet, and the guide says so.

Verification

The image was built from this branch on top of the re-pin, and every command in the Docker guide and the soak runbook was run against it with a mainnet node:

  • Boot: /readyz answered ready: true with both modules alive, the container went healthy, ccow-monitor finished its first-boot backfill of about 500,000 blocks in 35 seconds, and ethflow-watcher opened at head.
  • All eleven runbook queries parse and answer on the live Prometheus. After a deliberate restart, changes(nexum_runtime_start_time_seconds[7d]) answered 1.
  • Stop: the engine exited with code 0 and wrote graceful shutdown complete. After the start that followed, each event trigger opened at its cursor.
  • Digest pinning: with both pins and without require_component_digest = false, the supervisor ready line had "verified":2 and no loading unverified warning.
  • The log scrub left zero occurrences of the node's host, and the state backup command copied the store.
  • The Gnosis Chain example of the multi-chain section was run from a source build against a public Gnosis endpoint: /readyz answered ready: true with ethflow-watcher: alive, the event trigger opened at the chain head, and the engine exited with code 0.

just ci with CI set passes on this branch: fmt, clippy at -D warnings, the orderbook-only gate, rustdoc, both module wasms, nextest and the doctests.

Breaking

The shipped ethflow-watcher manifest targets chain 1 where it targeted chain 11155111, and engine.docker.toml declares chain 1 only. A deployment that used the committed files with a Sepolia endpoint needs its own manifest and config.

AI Assistance: Claude Code used for the review, the changes, the tests and the documentation, and for the live Docker test of the documented steps.

@mfw78
mfw78 force-pushed the docs/restore-operator-docs branch from cfe70eb to b9e30f4 Compare October 12, 2026 03:27
@mfw78 mfw78 changed the title docs: restore the operator, Docker, multi-chain and soak documentation fix(deploy)!: make the shipped deployment boot on mainnet, and cut the docs to what works Oct 12, 2026
mfw78 added 5 commits October 12, 2026 11:34
Shepherd's `main` carried no `docs/` directory: the carve rewrote history and the operator documentation stayed behind on the pre-carve line.
About twenty tracked files still pointed into it, and the compose observability profile mounted a `prometheus.yml` that did not exist.

This restores the deployment guide, the Docker guide, the multi-chain guide, the Prometheus scrape config and the soak runbook, rewritten against the current tree.
The old text described `[[subscription]]` tables, `module.toml`, `twap-monitor`, the upstream ComposableCoW registry and `shepherd_` metric names, none of which exist here now.
Engine-level topics link to the nexum-runtime operator handbook instead of repeating it.

The guides describe the shipped files as they are, including where they do not boot as committed: several `engine.*.toml` files declare only chain 1 while `ethflow-watcher` triggers on Sepolia, and the soak compose file forwards the wrong RPC variable.
Those files are not changed here.

The engine substitutes a dollar-brace token before it parses the config, comments included, so a comment that spelled one out as an example made `engine.docker.toml` and `engine.example.toml` refuse to boot for a missing variable.
Those comments are reworded, which is the one change here that alters behaviour.
Every other edit to a config, compose or build file is a comment that pointed at a missing document.

AI Assistance: Claude Code used to restore and rewrite the documents and to verify the commands, refusal strings and compose files against the tree.
The engine prints the full RPC URL at each start and can repeat it in an RPC error line.
The runbook now tells the operator to keep the raw log private and to attach a copy with the endpoints replaced.

AI Assistance: Claude Code used for the runbook text and the scrub commands.
…e docs to what works

The operator docs explained at length things that did not work: six engine configs that could not boot, a soak harness that needed local patches, and an EthFlow module on Sepolia beside a venue on mainnet.
This fixes or removes those things, and the docs now describe one tested path.

The shipped files now form one consistent mainnet deployment.
`ethflow-watcher` reads the production EthFlow contract on chain 1, and its first boot opens at head, because it only observes and the contract's history is of no use to it.
`engine.docker.toml`, `docker-compose.yml` and `.env.example` need one RPC endpoint, not five.
Compose now waits 180 seconds on a stop, which is longer than the engine's drain, and bounds the container log.
A test holds `engine.docker.toml` to the chain that the shipped manifests read.

Removed, because they could not run: `engine.m2.toml`, `engine.m3.toml`, `engine.e2e.toml`, `engine.load.toml`, `engine.soak.toml`, `engine.soak.docker.toml`, `docker-compose.soak.yml`, and the e2e, load and soak scripts with their justfile recipes.
The forked-mainnet engine test covers what the e2e scripts were for.

The multi-chain guide is folded into a short section of the deployment guide.
The soak runbook now records a run with the Compose Prometheus profile and a set of queries, which also apply to any other Prometheus.

BREAKING CHANGE: the shipped `ethflow-watcher` manifest targets chain 1 where it targeted chain 11155111, and `engine.docker.toml` declares chain 1 only.

AI Assistance: Claude Code used for the review, the changes, the tests and the documentation, and for the live Docker test of the documented steps.
…entation

The chain section of the deployment guide becomes "Multi-chain deployment". It names the two patterns (one engine process for each chain, one process for many chains), gives a worked example that runs ethflow-watcher on Gnosis Chain, shows the chain tables for Ethereum Mainnet, Gnosis Chain, Base and Arbitrum One, and states what is tested and what is not.

A new test holds those chain tables to the pinned config parser and makes sure that the venue accepts each of the four chains.

The README gets a documentation map that links the operator guides here and the module tutorial, the SDK guide, the testing guide, the manifest reference and the operator handbook in nexum-runtime.

The Gnosis Chain example was started one time by hand from a source build: the module became alive, its event trigger opened at the chain head, and the engine stopped cleanly.

AI Assistance: Claude Code used for the documentation, the test and the manual run.
…ublic

The `ghcr.io/nullislabs/shepherd` package is public, so `docker compose pull` needs no login. The "Run" and "Upgrade" sections of the Docker guide now pull the published image first and give the local build as the alternative. They also say to use a clone and an image of the same commit, because the config comes from the clone and the module manifests come from the image.

An anonymous `docker compose pull shepherd` was run and it got the `latest` image.

AI Assistance: Claude Code used for the documentation and the pull test.
@mfw78
mfw78 force-pushed the docs/restore-operator-docs branch from d72b5e8 to 6bccee9 Compare October 12, 2026 11:35
@mfw78
mfw78 merged commit a1ccd22 into main Oct 12, 2026
7 checks passed
@mfw78
mfw78 deleted the docs/restore-operator-docs branch October 12, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant