Skip to content

fix: harden the deploy agent and its trigger, back up volumes through container labels - #45

Merged
AmrMsCLL merged 3 commits into
mainfrom
fix/agent-hardening
Nov 5, 2026
Merged

AmrMsCLL merged 3 commits into
mainfrom
fix/agent-hardening

Conversation

@AmrMsCLL

@AmrMsCLL AmrMsCLL commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Fixes from the review of the deploy agent, plus the volume backup labels that existing volumes could not take.

Agent (host/agent.py)

Finding Fix
A signed compose could ask for anything as root Every project except console, edge and backup is checked before anything is written: no privileged, pid, ipc, uts, userns_mode, network_mode, cgroup, cap_add, devices, volumes_from, sysctls, ports, build, env_file, no security_opt beyond no-new-privileges, bind mounts only under ./, no external, foreign or driver-option volumes, only the external edge network, config files only under ./ and never from the environment. The list of projects with host access lives in agent.py on the host, so CI cannot change it.
A route could claim another project's hostname A route may only use Host(...) rules, may not name a hostname another route file already holds, and may only send traffic to its own containers (http://<project>:port or <project>-*).
A compose could read the agent's token Not in the review: Compose interpolates from the process environment, so ${DEPLOY_TOKEN} would have reached a container. graphite-up and compose down now run with only PATH and HOME.
Unbounded body, single thread Content-Length is required and capped at 4 MiB; clients time out after 30 s; a threaded server keeps /healthz answering while one lock keeps deploys serial.
/reapply exposed Removed with CONSOLE_TOKEN; the console redeploys through a host unit instead (console PR). The public route now passes only /deploy and /healthz.
Legacy secrets action Removed, with scripts/agent.py's secrets command. The secrets repository's Apply workflow fails from now on; disable it.
Replay within five minutes Each accepted signature is remembered for ten minutes and refused again.

Containers on the host can still reach 172.17.0.1:9000; with /reapply gone, only signed /deploy requests do anything there.

Deploy trigger

The graphite-deploy app can push to main past review (it is a bypass actor) and every project's CI can mint its token, so a compromised project CI could push any file and Deploy would run it. The app has no workflows permission, so deploy.yml is the place to check: a push by a bot now fails Deploy unless it changes only IMAGE_TAG=sha-… lines in deploy/*/values. projects/** no longer triggers Deploy, so registry syncs never deploy on their own; deploy by hand after a hosting change.

Backups

Docker cannot add labels to existing volumes, so the deploys kept console_data and strata_files-data without graphite.backup and printed a recreate prompt each time. The label moves to the container: graphite.backup.volumes: <volume>[,<volume>] on the service that mounts them, which backup.py reads.

Redeploy hint

graphite-up touches /srv/graphite/<project>/.applied after a successful up. The console compares a project's env file with it, so a redeploy that recreates nothing (a secret no container reads) still clears "needs redeploy".

Rollout

After merge, install the new agent and edge route over SSH, then run Deploy with project all:

source=https://raw.githubusercontent.com/project-graphite/platform/main/host
sudo curl -fsSL -o /srv/graphite/bin/agent.py "$source/agent.py"
sudo curl -fsSL -o /srv/graphite/bin/graphite-up "$source/graphite-up"
sudo chmod 755 /srv/graphite/bin/graphite-up
sudo systemctl restart graphite-agent

CONSOLE_TOKEN can be deleted from /agent/ in the console.

Checks

The policy was run against the flattened compose and route of Strata, Tracker and PixelPocket (no findings) and against a crafted compose and route (each rule caught); a replayed signature is refused. deploy.py check, check-values.py, check-memory.py pass.

…ough container labels

The agent caps the request body, times out stalled clients, serves health while a
deploy runs, refuses a replayed signature, and drops /reapply, CONSOLE_TOKEN and
the secrets action. graphite-up and compose run without the agent's environment.
Every project but console, edge and backup gets a policy: no privileged or host
settings, no host paths, ports, env_file or foreign volumes and networks, and a
route only for its own hostnames and containers. deploy.py sends compose and the
route as JSON for it. The public route to the agent passes only /deploy and
/healthz. A bot push may change only IMAGE_TAG lines before Deploy runs.

Existing volumes cannot take new labels, so the backup service reads the volumes
to keep from graphite.backup.volumes on the container that mounts them.
@AmrMsCLL
AmrMsCLL requested a review from a team as a code owner November 5, 2026 02:48
@github-actions github-actions Bot added the size/m 51-200 changed lines, excluding lockfiles label Nov 5, 2026
@AmrMsCLL
AmrMsCLL merged commit 71c9140 into main Nov 5, 2026
5 checks passed
@AmrMsCLL
AmrMsCLL deleted the fix/agent-hardening branch November 5, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/l 201-600 changed lines, excluding lockfiles

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant