Skip to content

feat: the hub as a server: React Router on Express, SQLite, Redis, sign-in, live stream - #29

Merged
AmrMsCLL merged 12 commits into
mainfrom
feat/hub-server
Nov 5, 2026
Merged

AmrMsCLL merged 12 commits into
mainfrom
feat/hub-server

Conversation

@AmrMsCLL

@AmrMsCLL AmrMsCLL commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Phase 2 of the hub rebuild (.claude/.Plans/as2026-11-04-graphite-hub-server.md), steps 1 to 6: the hub becomes one Node 24 server, React Router framework mode on Express, with SQLite for records and Redis for the bus, the stream, sessions and rate limits. Every current public route keeps its markup. Step 7, the deploy folder and the cutover, is a separate platform PR and owner work, listed below.

Commits

  1. build: React Router framework mode, Express, one tsconfig with typegen, Dockerfile (non-root, /data), local compose.yaml with DEV_LOGIN and Redis.
  2. storage: node:sqlite with ordered SQL migrations; the five current profiles are seeded from registry.json. Redis client; the bus writes the activity row first, then the 7-day events stream (its id is the row's seq), then publishes per topic. The snapshot is rebuilt when the bus says something changed.
  3. sign-in: ?graphite shows the control. GitHub OAuth (read:org), organisation membership required, owner from role: admin. Sessions in Redis for 7 days with the GitHub token sealed; membership re-checked at most once a minute. Sign-out checks CSRF and Origin. CF-Connecting-IP is trusted only from Cloudflare's ranges, refreshed daily. CSP with a per-request nonce, HSTS and the rest.
  4. errors: fingerprinted by normalised message and top frame, upserted into errors, and the owner is emailed on a new or reopened fingerprint, at most once a day.
  5. registry and GitHub: platform/projects/*.yml validated with platform's own schema through Ajv 2020-12. Org members, repository stats, and the webhook: HMAC check, idempotent by delivery id, 202 at once, processed afterwards. A .graphite.yml push dispatches the platform sync. Hourly full sync as the fallback.
  6. changelog: every project gets 1.0.0 once, on import day; after that, GitHub Releases (webhook plus hourly fallback), with generated notes classified by rules.mjs. releases/ (YAML history) is removed, and scripts/sync.mjs only cuts tags, reading listed projects from platform.
  7. deploys and uptime: /internal/deploys uses the agent's signing scheme (HMAC over stamp.body, 5-minute skew, replay set) and refuses anything that came through Traefik. Uptime pings every 30 s. toPublicEvent allow-lists fields per kind. /internal/metrics reports error rate, p95 and live connections.
  8. live: GET /live?topics= as SSE, with replay after Last-Event-ID, 25 s heartbeat, 12-topic limit and 100 ms coalescing. Visitors get only public projections. The client hook revalidates the loaders. Also the nightly lifecycle job (uptime roll-ups, retention).
  9. routes: all 13 current URL forms plus sitemap.xml, llms.txt, the old _redirects as 301s, and a real 404 status.
  10. ci: one pipeline.yml on the shared pipeline (image ghcr.io/project-graphite/graphite-hub/site:sha-…), plus types, format and bundle size.

Verified

  • npm run lint, format:check, typecheck: clean. npm test: 50 pass (the new server/*.test.ts specs and the existing scripts/rules.test.mjs).
  • The real platform schema validates all 13 real projects/*.yml. The listed set gives the same 9 projects, paths and URLs as the current site.
  • Ran against a local Redis with real GitHub data. Every route answered 200, unknown handles and slugs 404, redirects 301. Signed deploy reports reached an open live stream without the host. Webhook signature, redelivery, CSRF, origin and open-redirect checks behaved. Production build: all 5 inline scripts carry the CSP nonce, assets are immutable, and Chromium hydrated with no console errors. A published releases event made an open tab revalidate.

Bundle budget

150 KB of compressed JavaScript per page (owner decision). Pages load 116–120 KB today, about 107 KB of it React 19 and React Router.

Cutover steps

The PRs, in merge order: Project-Graphite/actions#11 → Project-Graphite/platform#49 → Project-Graphite/console#30 → this PR → Project-Graphite/platform#50.

A. Before merging anything

  1. Create an OAuth App: homepage https://project-graphite.com, callback https://project-graphite.com/auth/callback.

  2. Give the platform GitHub App Organisation → Members: read and accept the new permission on the installation. Without it the hub cannot list members.

  3. Generate four values: SESSION_SECRET, GITHUB_WEBHOOK_SECRET, DEPLOY_REPORT_SECRET and REDIS_PASSWORD, each with node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))".

  4. Enter them in the console's Secrets page:

    • /graphite-hub/: GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, GITHUB_WEBHOOK_SECRET, SESSION_SECRET, DEPLOY_REPORT_SECRET, REDIS_PASSWORD, PLATFORM_APP_CLIENT_ID, PLATFORM_APP_PRIVATE_KEY; optionally SMTP_URL, ALERTS_EMAIL, ALERTS_FROM.
    • /agent/: DEPLOY_REPORT_SECRET (the same value).
  5. Add DEPLOY_REPORT_SECRET=<the same value> to /etc/graphite/console.env on the VPS over SSH. The Secrets page refuses /console/; the console's own secrets are placed by hand.

B. Merge, in order
6. actions#11. The changelog workflow stops writing YAML into the hub.
7. platform#49. The hub's deploy folder lands disabled; the console redeploys with the new variable.
8. On the VPS, install the host files (they only travel through bootstrap.sh):

cd /srv/graphite/bin
sudo curl -fsSL -o graphite-up https://raw.githubusercontent.com/project-graphite/platform/main/host/graphite-up
sudo curl -fsSL -o graphite-report https://raw.githubusercontent.com/project-graphite/platform/main/host/graphite-report
sudo chmod 755 graphite-up graphite-report && sudo chown deploy:deploy graphite-up graphite-report
  1. console#30. The console gains the two hub alert rules.
  2. This PR. The pipeline pushes ghcr.io/project-graphite/graphite-hub/site:sha-… and records it in platform's values; Deploy still holds the hub disabled. Then make the new graphite-hub/site package public in the organisation's packages, matching console/app and strata/app, so the VPS can pull it.

C. Cutover
11. platform#50 retargets to main once #49's branch is deleted. Mark it ready and merge it. Deploy starts the hub, waits for readiness, routes the apex and points DNS at the VPS through Cloudflare.
12. Check:
- https://project-graphite.com/health/ready answers ready.
- The response headers carry Content-Security-Policy and X-Request-Id.
- /?graphite shows sign in, and signing in works.
- The console shows graphite-hub-app-1 and graphite-hub-redis-1 healthy, with no hub alerts.
13. Create the organisation webhook: payload https://project-graphite.com/internal/github, content type JSON, secret GITHUB_WEBHOOK_SECRET. Events: push, release, star, fork, member, membership, organization, repository, meta. The ping under Recent Deliveries should answer 202.
14. Once the site has run cleanly for a day, delete the graphite-hub Pages project in Cloudflare.

Rollback before step 14: in Cloudflare DNS, set the apex back to a proxied CNAME to graphite-hub.pages.dev. After step 14: set IMAGE_TAG in deploy/graphite-hub/values back to the last good sha-.

@github-actions github-actions Bot added the size/xl Over 600 changed lines, excluding lockfiles label Nov 5, 2026
@AmrMsCLL
AmrMsCLL marked this pull request as ready for review November 5, 2026 13:26
@AmrMsCLL
AmrMsCLL requested a review from a team as a code owner November 5, 2026 13:26
AmrMsCLL added a commit to Project-Graphite/actions that referenced this pull request Nov 5, 2026
…rding them in the hub (#11)

The changelog workflow stops writing YAML into graphite-hub. The hub now
reads GitHub Releases directly (Project-Graphite/graphite-hub#29).

- Removed the `node scripts/compile.mjs` and **Record releases** steps.
Tagging the next version and publishing its GitHub Release are
unchanged.
- Added `platform` to the App token's repositories. The hub's new
`scripts/sync.mjs` reads the listed projects from
`platform/projects/*.yml` instead of the hub's `registry.json`.

**Merge order:** merge this before graphite-hub#29. Until #29 merges,
the hub's current `sync.mjs` still reads `registry.json` and works
without the removed steps. If #29 merges first, every project's
changelog run fails at `compile.mjs`, which no longer exists, after its
tags and releases have already been published.

**Stale docs:** `README.md` lines 61 and 172–179 still say the workflow
records entries under `releases/` in graphite-hub. That's phase 8
documentation and needs permission.
AmrMsCLL added a commit to Project-Graphite/platform that referenced this pull request Nov 5, 2026
…ery rollout reported to the hub (#49)

The platform half of the hub cutover, which is phase 2 step 7 of the hub
server plan. It pairs with Project-Graphite/graphite-hub#29.

## Changes

- **`deploy/graphite-hub/`**: the hub as a compose project. The app
extends `shared/app.yaml`; Redis extends `shared/redis-password.yaml`
with `noeviction`, a 96 MB maxmemory and a 128M limit. The app's `/data`
volume holds SQLite and is in the nightly backup through
`graphite.backup.volumes`. Health is checked on `/health/ready`. Secrets
are `${NAME:?}`, and SMTP and alerts are optional.
- **`values`** starts with `ENABLED=false` and `PROXIED=true`. Deploy
treats it as down, so merging this changes nothing on the VPS and the
apex stays on Pages. Sizing follows the plan: app 1024M limit, 512M
reserve, 1.5 CPUs.
- **`projects/graphite-hub.yml`**: `deploy.kind` goes from `pages` to
`compose`. The hub repository has no `.graphite.yml`, so the sync leaves
this file alone.
- **`host/graphite-report`** runs from an exit trap in `graphite-up`, so
every rollout is reported to the hub: agent deploys and console
redeploys, successes and failures. It posts to the hub container's edge
address, signed with `DEPLOY_REPORT_SECRET` from
`/etc/graphite/agent.env` using the agent's scheme (HMAC over
`stamp.body`), plus the host's status code and response time. It reports
nothing for edge and backup, which have no host, or while the hub isn't
running. A failed report never changes the rollout's exit code.
- **`host/bootstrap.sh`** installs `graphite-report`.
- **deploy/console/compose.yaml** passes `DEPLOY_REPORT_SECRET`
(optional) to the console, which uses it to read the hub's
`/internal/metrics` for the Alerts page (Project-Graphite/console#30).

## Checked

- `check-values.py`: 0 problems. `check-memory.py`: reserved 1320M of
3000M. `deploy.py check`: 0 problems. `validate.py`: 13 projects, 0
problems.
- `graphite-up` passes `sh -n`. The report script parses the console's
plain, single-quoted and double-quoted env values. Its signature matches
the hub's `sign()` exactly.

## Not in this PR

- `host/` files reach the VPS only through `bootstrap.sh`. After merge,
copy `graphite-up` and `graphite-report` to `/srv/graphite/bin/` by hand
(see the hub PR's cutover steps).
- `README.md` still describes the chart and Pages for the hub. That's
phase 8 documentation.
@AmrMsCLL
AmrMsCLL merged commit 84803f5 into main Nov 5, 2026
10 checks passed
@AmrMsCLL
AmrMsCLL deleted the feat/hub-server branch November 5, 2026 14:09
AmrMsCLL added a commit to Project-Graphite/platform that referenced this pull request Nov 5, 2026
**The cutover itself. Merge last, by hand, after the steps in
Project-Graphite/graphite-hub#29.**

Removes `ENABLED=false` from `deploy/graphite-hub/values`. On merge,
Deploy:

1. asks the agent to bring up `graphite-hub` (app and Redis). The agent
refuses and waits if any required secret is missing from
`/etc/graphite/graphite-hub.env`;
2. waits for `/health/ready`, then publishes the Traefik route for
`project-graphite.com`;
3. points the apex at the VPS through the Cloudflare proxy
(`PROXIED=true`), replacing the CNAME to `graphite-hub.pages.dev`.

The rollout is then reported to the hub by `graphite-report`, so the
hub's own first deploy is its first activity row.

**Rollback while the Pages project still exists:** in Cloudflare DNS,
set `project-graphite.com` back to a proxied CNAME to
`graphite-hub.pages.dev`. Pages still holds its last deployment.

Stacked on #49; it retargets to `main` when #49 merges and its branch is
deleted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xl Over 600 changed lines, excluding lockfiles

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant