Skip to content

feat(deploy): add a Kustomize base and a guide for Kubernetes - #699

Merged
mfw78 merged 1 commit into
mainfrom
feat/kubernetes-manifests
Oct 12, 2026
Merged

mfw78 merged 1 commit into
mainfrom
feat/kubernetes-manifests

Conversation

@mfw78

@mfw78 mfw78 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Why

The operator docs cover a run from source and Docker Compose. This adds the third common way to run the engine: a Kubernetes cluster, from the published image.

What changed

  • deploy/kubernetes/ is a Kustomize base with two objects: a StatefulSet (one replica, one 2 GiB state volume) and a headless Service for the metrics listener.
  • The pod runs as the image's shepherd user (id 999) with a read-only root file system, no capabilities and no service account token, so it agrees with the restricted Pod Security level. It has HTTP probes on /healthz and /readyz, the prometheus.io/* scrape annotations, and a termination grace period of 180 seconds, which is longer than the engine's 150 second drain.
  • The base does not create the engine config ConfigMap or the RPC URL Secret. docs/deployment/kubernetes.md shows how to make them from engine.docker.toml, how to pin the image and the StorageClass in an overlay, how to read metrics, and how to stop and upgrade.
  • README.md lists the new directory and the new guide.

No Rust code changes.

Verification

The base was applied to a local kind cluster (Kubernetes 1.36), in a namespace with pod-security.kubernetes.io/enforce=restricted, with an image built from the tree of #696 and a mainnet node:

  • The pod went 1/1 Running. /readyz answered ready: true with both modules alive, and /metrics had nexum_runtime_start_time_seconds.
  • Inside the container the user was 999, the root file system refused a write, and the state volume held local-store.redb.
  • After kubectl delete pod, the engine wrote graceful shutdown complete. The new pod opened each event trigger at its cursor and did no backfill.
  • The overlay of the guide renders with kubectl kustomize, with the image pin and the StorageClass patch, against this branch by commit.

Worth a second look

  • Not tested: a network volume, a default-deny network policy, and a collector that scrapes by annotation. The guide says so. These get their first run in the soak deployment, and the "What is tested" list can gain a line after that.
  • The user id 999 comes from useradd -r in the Dockerfile, which does not name an id. The manifest names 999 because Kubernetes refuses runAsNonRoot with a non-numeric image user. A later change could set the id explicitly in the Dockerfile so that the two cannot drift.
  • CI does not render or apply the manifests.

AI Assistance: Claude Code used for the manifests, the guide and the test run.

`deploy/kubernetes` holds one StatefulSet (one replica, one 2 GiB state volume) and one headless Service for the metrics listener. The pod runs as the image's `shepherd` user with a read-only root file system and no capabilities, so it agrees with the `restricted` Pod Security level. It has HTTP probes on `/healthz` and `/readyz`, scrape annotations, and a termination grace period of 180 seconds, which is longer than the engine's drain.

The base does not create the engine config ConfigMap or the RPC URL Secret. `docs/deployment/kubernetes.md` shows how to make them, how to pin the image and the StorageClass in an overlay, and what was tested.

Tested on a local kind cluster (Kubernetes 1.36) in a namespace that enforces the `restricted` level, with an image built from this tree and a mainnet node: both modules became alive, the endpoints answered, and after a pod deletion the engine stopped cleanly and the new pod opened each event trigger at its cursor. A network volume and a default-deny network policy were not tested.

AI Assistance: Claude Code used for the manifests, the guide and the test run.
@mfw78
mfw78 merged commit a42d720 into main Oct 12, 2026
7 checks passed
@mfw78
mfw78 deleted the feat/kubernetes-manifests branch October 12, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant