Repository navigation
feat: alert on the hub's error rate and failing readiness - #30
Merged
Merged
Conversation
AmrMsCLL
added a commit
to Project-Graphite/platform
that referenced
this pull request
Nov 5, 2026
…ery rollout reported to the hub (#49) The platform half of the hub cutover, which is phase 2 step 7 of the hub server plan. It pairs with Project-Graphite/graphite-hub#29. ## Changes - **`deploy/graphite-hub/`**: the hub as a compose project. The app extends `shared/app.yaml`; Redis extends `shared/redis-password.yaml` with `noeviction`, a 96 MB maxmemory and a 128M limit. The app's `/data` volume holds SQLite and is in the nightly backup through `graphite.backup.volumes`. Health is checked on `/health/ready`. Secrets are `${NAME:?}`, and SMTP and alerts are optional. - **`values`** starts with `ENABLED=false` and `PROXIED=true`. Deploy treats it as down, so merging this changes nothing on the VPS and the apex stays on Pages. Sizing follows the plan: app 1024M limit, 512M reserve, 1.5 CPUs. - **`projects/graphite-hub.yml`**: `deploy.kind` goes from `pages` to `compose`. The hub repository has no `.graphite.yml`, so the sync leaves this file alone. - **`host/graphite-report`** runs from an exit trap in `graphite-up`, so every rollout is reported to the hub: agent deploys and console redeploys, successes and failures. It posts to the hub container's edge address, signed with `DEPLOY_REPORT_SECRET` from `/etc/graphite/agent.env` using the agent's scheme (HMAC over `stamp.body`), plus the host's status code and response time. It reports nothing for edge and backup, which have no host, or while the hub isn't running. A failed report never changes the rollout's exit code. - **`host/bootstrap.sh`** installs `graphite-report`. - **deploy/console/compose.yaml** passes `DEPLOY_REPORT_SECRET` (optional) to the console, which uses it to read the hub's `/internal/metrics` for the Alerts page (Project-Graphite/console#30). ## Checked - `check-values.py`: 0 problems. `check-memory.py`: reserved 1320M of 3000M. `deploy.py check`: 0 problems. `validate.py`: 13 projects, 0 problems. - `graphite-up` passes `sh -n`. The report script parses the console's plain, single-quoted and double-quoted env values. Its signature matches the hub's `sign()` exactly. ## Not in this PR - `host/` files reach the VPS only through `bootstrap.sh`. After merge, copy `graphite-up` and `graphite-report` to `/srv/graphite/bin/` by hand (see the hub PR's cutover steps). - `README.md` still describes the chart and Pages for the hub. That's phase 8 documentation.
AmrMsCLL
added a commit
to Project-Graphite/graphite-hub
that referenced
this pull request
Nov 5, 2026
…gn-in, live stream (#29) Phase 2 of the hub rebuild (`.claude/.Plans/as2026-11-04-graphite-hub-server.md`), steps 1 to 6: the hub becomes one Node 24 server, React Router framework mode on Express, with SQLite for records and Redis for the bus, the stream, sessions and rate limits. Every current public route keeps its markup. Step 7, the deploy folder and the cutover, is a separate platform PR and owner work, listed below. ## Commits 1. **build**: React Router framework mode, Express, one tsconfig with typegen, Dockerfile (non-root, `/data`), local `compose.yaml` with `DEV_LOGIN` and Redis. 2. **storage**: `node:sqlite` with ordered SQL migrations; the five current profiles are seeded from `registry.json`. Redis client; the bus writes the activity row first, then the 7-day `events` stream (its id is the row's `seq`), then publishes per topic. The snapshot is rebuilt when the bus says something changed. 3. **sign-in**: `?graphite` shows the control. GitHub OAuth (`read:org`), organisation membership required, owner from `role: admin`. Sessions in Redis for 7 days with the GitHub token sealed; membership re-checked at most once a minute. Sign-out checks CSRF and Origin. `CF-Connecting-IP` is trusted only from Cloudflare's ranges, refreshed daily. CSP with a per-request nonce, HSTS and the rest. 4. **errors**: fingerprinted by normalised message and top frame, upserted into `errors`, and the owner is emailed on a new or reopened fingerprint, at most once a day. 5. **registry and GitHub**: `platform/projects/*.yml` validated with platform's own schema through Ajv 2020-12. Org members, repository stats, and the webhook: HMAC check, idempotent by delivery id, 202 at once, processed afterwards. A `.graphite.yml` push dispatches the platform sync. Hourly full sync as the fallback. 6. **changelog**: every project gets 1.0.0 once, on import day; after that, GitHub Releases (webhook plus hourly fallback), with generated notes classified by `rules.mjs`. `releases/` (YAML history) is removed, and `scripts/sync.mjs` only cuts tags, reading listed projects from platform. 7. **deploys and uptime**: `/internal/deploys` uses the agent's signing scheme (HMAC over `stamp.body`, 5-minute skew, replay set) and refuses anything that came through Traefik. Uptime pings every 30 s. `toPublicEvent` allow-lists fields per kind. `/internal/metrics` reports error rate, p95 and live connections. 8. **live**: `GET /live?topics=` as SSE, with replay after `Last-Event-ID`, 25 s heartbeat, 12-topic limit and 100 ms coalescing. Visitors get only public projections. The client hook revalidates the loaders. Also the nightly lifecycle job (uptime roll-ups, retention). 9. **routes**: all 13 current URL forms plus `sitemap.xml`, `llms.txt`, the old `_redirects` as 301s, and a real 404 status. 10. **ci**: one `pipeline.yml` on the shared pipeline (image `ghcr.io/project-graphite/graphite-hub/site:sha-…`), plus types, format and bundle size. ## Verified - `npm run lint`, `format:check`, `typecheck`: clean. `npm test`: 50 pass (the new `server/*.test.ts` specs and the existing `scripts/rules.test.mjs`). - The real platform schema validates all 13 real `projects/*.yml`. The listed set gives the same 9 projects, paths and URLs as the current site. - Ran against a local Redis with real GitHub data. Every route answered 200, unknown handles and slugs 404, redirects 301. Signed deploy reports reached an open live stream without the host. Webhook signature, redelivery, CSRF, origin and open-redirect checks behaved. Production build: all 5 inline scripts carry the CSP nonce, assets are immutable, and Chromium hydrated with no console errors. A published `releases` event made an open tab revalidate. ## Bundle budget 150 KB of compressed JavaScript per page (owner decision). Pages load 116–120 KB today, about 107 KB of it React 19 and React Router. ## Cutover steps The PRs, in merge order: Project-Graphite/actions#11 → Project-Graphite/platform#49 → Project-Graphite/console#30 → this PR → Project-Graphite/platform#50. **A. Before merging anything** 1. Create an OAuth App: homepage `https://project-graphite.com`, callback `https://project-graphite.com/auth/callback`. 2. Give the platform GitHub App **Organisation → Members: read** and accept the new permission on the installation. Without it the hub cannot list members. 3. Generate four values: `SESSION_SECRET`, `GITHUB_WEBHOOK_SECRET`, `DEPLOY_REPORT_SECRET` and `REDIS_PASSWORD`, each with `node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"`. 4. Enter them in the console's Secrets page: - `/graphite-hub/`: `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, `GITHUB_WEBHOOK_SECRET`, `SESSION_SECRET`, `DEPLOY_REPORT_SECRET`, `REDIS_PASSWORD`, `PLATFORM_APP_CLIENT_ID`, `PLATFORM_APP_PRIVATE_KEY`; optionally `SMTP_URL`, `ALERTS_EMAIL`, `ALERTS_FROM`. - `/agent/`: `DEPLOY_REPORT_SECRET` (the same value). 5. Add `DEPLOY_REPORT_SECRET=<the same value>` to `/etc/graphite/console.env` on the VPS over SSH. The Secrets page refuses `/console/`; the console's own secrets are placed by hand. **B. Merge, in order** 6. actions#11. The changelog workflow stops writing YAML into the hub. 7. platform#49. The hub's deploy folder lands disabled; the console redeploys with the new variable. 8. On the VPS, install the host files (they only travel through `bootstrap.sh`): ```sh cd /srv/graphite/bin sudo curl -fsSL -o graphite-up https://raw.githubusercontent.com/project-graphite/platform/main/host/graphite-up sudo curl -fsSL -o graphite-report https://raw.githubusercontent.com/project-graphite/platform/main/host/graphite-report sudo chmod 755 graphite-up graphite-report && sudo chown deploy:deploy graphite-up graphite-report ``` 9. console#30. The console gains the two hub alert rules. 10. This PR. The pipeline pushes `ghcr.io/project-graphite/graphite-hub/site:sha-…` and records it in platform's `values`; Deploy still holds the hub disabled. Then make the new `graphite-hub/site` package **public** in the organisation's packages, matching `console/app` and `strata/app`, so the VPS can pull it. **C. Cutover** 11. platform#50 retargets to `main` once #49's branch is deleted. Mark it ready and merge it. Deploy starts the hub, waits for readiness, routes the apex and points DNS at the VPS through Cloudflare. 12. Check: - `https://project-graphite.com/health/ready` answers `ready`. - The response headers carry `Content-Security-Policy` and `X-Request-Id`. - `/?graphite` shows *sign in*, and signing in works. - The console shows `graphite-hub-app-1` and `graphite-hub-redis-1` healthy, with no hub alerts. 13. Create the organisation webhook: payload `https://project-graphite.com/internal/github`, content type JSON, secret `GITHUB_WEBHOOK_SECRET`. Events: push, release, star, fork, member, membership, organization, repository, meta. The ping under Recent Deliveries should answer 202. 14. Once the site has run cleanly for a day, delete the `graphite-hub` Pages project in Cloudflare. **Rollback before step 14:** in Cloudflare DNS, set the apex back to a proxied CNAME to `graphite-hub.pages.dev`. **After step 14:** set `IMAGE_TAG` in `deploy/graphite-hub/values` back to the last good `sha-`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The console's side of the hub's observability (hub plan, phase 2 exit: "the console's Alerts page shows the hub's error rate").
/internal/metrics./health/ready.http://graphite-hub:3000over the edge network. It signs the metrics request withDEPLOY_REPORT_SECRET, using the agent's HMAC scheme overstamp.; the hub refuses anything that came through Traefik.DEPLOY_REPORT_SECRETin the console env, the hub checks are off.Checked:
test/hub.spec.tsandtest/alerts.spec.ts(14 pass),tscfor server and web,oxlint.Needs Project-Graphite/platform#49, which passes
DEPLOY_REPORT_SECRETto the console container, and the value added by hand to/etc/graphite/console.env(the Secrets page refuses/console/).