Skip to content

feat: secret store, Backups page and the registry workflow retired - #19

Merged
AmrMsCLL merged 2 commits into
mainfrom
feat/secret-store
Nov 5, 2026
Merged

feat: secret store, Backups page and the registry workflow retired#19
AmrMsCLL merged 2 commits into
mainfrom
feat/secret-store

Conversation

@AmrMsCLL

@AmrMsCLL AmrMsCLL commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Phase 1, step 1 of the Graphite Hub plan: the secret store inside the console, replacing the secrets repository and SOPS once every project is moved.

What it does

  • Store. /data/secrets.db holds secrets, secret_versions and secret_audit. Values are AES-256-GCM with a random nonce per write and <project>/<name> as associated data, so a tampered value or one moved to another name fails to decrypt.
  • Key rotation. SECRETS_KEY holds new or new,old (32-byte base64 keys). On start every version still on the old key is re-encrypted with the new one; a version whose key is missing stops the start with its key id.
  • Host files. Every change writes /etc/graphite/<project>.env on the host through HOST_ROOT (the console already runs privileged with pid: host), mode 600 in a 700 directory, through a temporary file and a rename, inside the same transaction as the database. The file is rewritten only when its values differ.
  • Reconcile. On start a hand-edited host file wins over the store: changed, added and removed names are applied as host and audited. console.env is never read and the console project is refused.
  • Audit. Create, update, restore, delete, reveal, redeploy and reconcile each write a row.
  • Access. Owners see everything. A member gets a project's secrets only through a new secrets:<slug> grant on Admin; * does not include it.
  • Compose awareness. The compose file the agent last applied (/srv/graphite/<project>/compose.yaml) gives the Usage tab, the "required by compose, missing" hint and the refusal to delete a name it requires with :?.
  • Needs redeploy. A project whose env file is newer than its newest container shows a red hint on the Secrets page, its Project page and its Overview row. Redeploy posts {"name", "by"} to ${AGENT_URL}/reapply with /agent/CONSOLE_TOKEN from the store.

UI

Secrets page for owners (path prefix and text filter, rows grouped by project, checkbox column for Delete and Copy names, Create and Import), a detail page with Overview (reveal for 30 seconds, copy), History (length change, restore) and Usage, and the same list on each Project page for members with the grant.

Backups (step 3)

The Backups page (owners) reads the shared backup service's archives in /var/lib/graphite/backups through HOST_ROOT:

  • every project's runs with date, verification result and errors, contents (dumps, volumes, SQLite copies) and size;
  • Download streams the archive; Back up now, Restore (behind a confirmation) and Import (upload an archive, which the service verifies before it can be restored) write request files the service picks up; recent requests show their state and output;
  • every download, run, import, verification and restore is appended to audit.log beside the archives.

The Overview shows the last run and how many projects verified. The Alerts check that read the old per-project backups volume now flags a project whose newest run is over 26 hours old or failed verification; its test is rewritten for the new summaries with the same requirement.

Registry

.github/workflows/registry.yml goes with actions' reusable-sync-registry.yml (Project-Graphite/actions#10); platform's Sync registry runs hourly.

Rollout

  • Without SECRETS_KEY the console runs as before and the Secrets page answers 503, so this can merge before the key exists.
  • Redeploy answers 502 until the agent has /reapply (platform, step 1.2).

New settings

SECRETS_KEY (optional), AGENT_URL (http://172.17.0.1:9000), GRAPHITE_HOME (/srv/graphite).

Checks

tsc for server and web, oxlint, npm run build, test/secrets.spec.ts, test/access.spec.ts, test/backups.spec.ts and test/alerts.spec.ts. The pages have not been looked at in a browser.

README.md is stale on the Admin access table, the API table, the Production settings, the Alerts section (backups) and the Layout table.

@github-actions github-actions Bot added the size/xl Over 600 changed lines, excluding lockfiles label Nov 5, 2026
@AmrMsCLL AmrMsCLL mentioned this pull request Nov 5, 2026
Merged
@AmrMsCLL
feat: a secret store with a Secrets page, per-project env files on th…
0260f0a 
…e host and a Redeploy hint

Values are encrypted with AES-256-GCM in /data/secrets.db, bound to their
project and name, versioned, restorable and audited. Every change writes
/etc/graphite/<project>.env on the host atomically through HOST_ROOT, and a
hand-edited file is reconciled into the store on start. SECRETS_KEY may hold
new,old to rotate. Secrets are granted per project with secrets:<slug>; * does
not include them.
@AmrMsCLL
feat: a Backups page over the shared backup service, and the registry…
70cb70f 
… workflow retired

The console lists every project's nightly archives from /var/lib/graphite/backups
with their verification, downloads them, asks the backup service to back a
project up now, to verify an uploaded archive and to restore one, and audits each
action in audit.log beside the archives. The Overview shows the last run and the
Alerts page flags a project whose newest backup is over 26 hours old or failed.
The registry workflow goes; platform syncs .graphite.yml on its own schedule.
@github-actions github-actions Bot added the ci label Nov 5, 2026
@AmrMsCLL AmrMsCLL changed the title feat: secret store with a Secrets page and per-project env files on the host feat: secret store, Backups page and the registry workflow retired Nov 5, 2026
AmrMsCLL added a commit to Project-Graphite/platform that referenced this pull request Nov 5, 2026
@AmrMsCLL
feat: plain compose with shared fragments, a shared backup service, t…
c171c68 
…he secret store's host side, and the chart, tags and people retired (#43)

Phase 1 of the Graphite Hub plan for `platform`: the host side of the
console's secret store (step 2), the shared backup service (step 3),
plain compose with shared fragments (steps 4 and 5), and the retirement
of the chart, tag records and people files (step 7). Goes with
Project-Graphite/console#19 and Project-Graphite/actions#10.

## Host and agent

- `graphite-up` and the agent use `/etc/graphite/<project>.env` when it
exists and `secrets.env` otherwise, so each project moves to the
console's store on its first rollout after its secrets are imported
there.
- The agent runs as root, reads `/etc/graphite/agent.env`, accepts
`values` and `files/<name>` in a payload, refuses a missing secret name
before it writes anything, and answers `POST /reapply` with
`Authorization: Bearer <CONSOLE_TOKEN>` for the console's Redeploy.
- `graphite-up` passes `--env-file values` before the secrets file.

## Plain compose

- `shared/app.yaml` (`service`: hardening, logging, health timing,
limits; `app`: plus the edge alias `${PROJECT}`, `APP_URL`,
`APP_PORT`/`APP_MEMORY`/`APP_CPUS`/`APP_RESERVE`),
`shared/postgres.yaml`, `shared/redis.yaml`,
`shared/redis-password.yaml`.
- Console, Strata, Tracker and PixelPocket are each
`deploy/<slug>/compose.yaml` plus `deploy/<slug>/values`; the Pages
projects keep only `values` with `IMAGE_TAG`. Volume names, container
names and the compose project name are unchanged, so data stays where it
is.
- `deploy.py` flattens the compose (`docker compose config
--no-interpolate`), renders it with the secret names stubbed, derives
the Traefik route from the service on the edge network, and sends
compose, values (with `PROJECT` and `HOST` from the registry) and
`files/` to the agent. A removed `deploy/<slug>/` takes a compose
project down.
- PR checks: `deploy.py check` (every compose renders),
`check-memory.py` (reservations against `deploy/host.yaml`),
`check-values.py` (no value equal to its default, `ghcr.io` images only
on `${IMAGE_TAG:?}` or `sha-`, no `${NAME}` without `:?` or `:-`).

### Rendered both ways

Each project rendered through the chart (origin/main) and through the
new compose, both interpolated with the same stubbed secrets, differs
only in:

| Project | Difference |
| :--- | :--- |
| console | `SECRETS_KEY: ${SECRETS_KEY:-}` added; `graphite.backup`
label on `data` |
| strata | backup sidecar and its `backups` volume gone; postgres flags
reordered and `work_mem=4MB` (the default) dropped; health check reads
`$POSTGRES_USER`/`$POSTGRES_DB`; backup labels on postgres and
`files-data` |
| graphite-tracker | postgres command states the defaults
`shared_buffers=128MB`, `max_connections=100`; health check as above;
backup label on postgres |
| pixelpocket | MinIO scripts as config files instead of inline content;
`APP_URL` on the frontend; the unused `MONGO_PASSWORD` gone from mongo |

Required secret names and routes are identical for all four.

## Backups

`edge/backup.yaml` runs `edge/backup.py` (Python standard library,
Docker API over the socket):

- Nightly at 03:00 UTC, per compose project: `pg_dump` of every
container labelled `graphite.backup.postgres`, run and checked with
`pg_restore --list` inside the container, pulled out through the archive
API and compared by checksum; a tar of every volume labelled
`graphite.backup`, with every SQLite database in it replaced by a
consistent copy through SQLite's online backup API (no endpoint in the
app). One `/var/lib/graphite/backups/<project>/<stamp>.tar` with a
manifest, and a `<stamp>.json` summary beside it.
- Every archive is reopened and verified: each member's checksum, every
volume tar read to its end, every SQLite copy passes `integrity_check`.
A failure marks the run failed, which the console alerts on.
- 14 days kept. The console asks for a run, a verification of an
uploaded archive, or a restore through request files in `.requests/`; a
restore verifies first, stops the project, restores the dumps with
`pg_restore --clean --single-transaction` and the volumes, starts it and
waits for health.

## Removed

`charts/`, `deploy/chart.yaml`, `tags/`, `people/`,
`scripts/compile.py`, `scripts/people.py`, `schema/person.schema.json`,
the Main workflow, sleep and wake jobs (no project uses them;
`edge/wake.yaml` goes), `release-core`, Helm in CI, `sync.py`'s
single-manifest check. `validate.py` checks owners against organisation
membership. Sync registry runs hourly as well as on dispatch.

`deploy/graphite-message-lab/` is removed, which **takes Message Lab
down** on merge (volumes kept). `deploy/traceboard/` (disabled) is
removed.

## Rollout

1. Merge this, then Project-Graphite/actions#10 straight after. If
`main` gained `tags/` commits meanwhile, rebase and copy the newest tags
into each `values` `IMAGE_TAG`.
2. Deploy runs on merge: the edge and Message Lab's takedown work with
the current agent; the backup service and the compose projects are
refused by it before anything is written.
3. Install the new agent over SSH:

   ```sh
   sudo install -d -m 700 /etc/graphite
   sudo mv /etc/graphite-agent.env /etc/graphite/agent.env

source=https://raw.githubusercontent.com/project-graphite/platform/main/host
   sudo curl -fsSL -o /srv/graphite/bin/agent.py "$source/agent.py"
sudo curl -fsSL -o /srv/graphite/bin/graphite-up "$source/graphite-up"
   sudo chmod 755 /srv/graphite/bin/graphite-up
sudo curl -fsSL -o /etc/systemd/system/graphite-agent.service
"$source/graphite-agent.service"
   sudo systemctl daemon-reload
   sudo systemctl restart graphite-agent
   ```

4. Run Deploy by hand with project `all`.
5. Back up one project from the console's Backups page, download it,
import it and restore it: the phase 1 exit check.

The `core-v1.0.0`, `core-v1.0.1` and `core-v2.0.0` tags are left in
place; `git push origin --delete core-v1.0.0 core-v1.0.1 core-v2.0.0`
removes them when wanted.

## Checks

`deploy.py check`, `check-memory.py`, `check-values.py` (plus a negative
run catching each of its rules), `validate.py` against live membership,
the render diff above, `py_compile` and a local agent answering 401, 409
and a signed `down`. `backup.py` was run locally for packing a volume
with a live WAL database, verification, tamper and wrong-project
refusal, retention and the schedule. Not run: anything against Docker on
the VPS (dumps, archive pulls, restore), `graphite-up` and
`bootstrap.sh`.

`README.md` is stale throughout on the chart, `core.yaml`/`values.yaml`,
`tags/`, `people/`, the Main workflow, `/etc/graphite-agent.env` and the
backup sidecar.
@AmrMsCLL
AmrMsCLL merged commit 0f8f6e4 into main Nov 5, 2026
9 checks passed
@AmrMsCLL
AmrMsCLL deleted the feat/secret-store branch November 5, 2026 01:41
@AmrMsCLL AmrMsCLL mentioned this pull request Nov 7, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ci size/xl Over 600 changed lines, excluding lockfiles

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AmrMsCLL