Add Connector Gateway KEK rotation guide

[tgrunnagle]

Description

Adds Key rotation (connector-gateway/rotate-encryption-key.mdx) to the Connector Gateway sidebar's Operate group. It's an operator-facing how-to adapted from the internal enterprise/connector-gateway/docs/kek-operations.md, and it covers:

• How key versions and the active version work (brief orientation)
• Rotating with two rolling restarts: add the new version with kek.activeVersion pinned to the old one and a bumped kek.keyringGeneration, then activate it
• Rolling back by moving activeVersion back
• Retiring an old version: the 30-day wait, removal, watching kek_unsealable_reads_total, and the effects of dynamic client registration (everyone reconnects, old client is orphaned and logged as replaced_client_id)
• Keeping the Secret append-only when using helm rollback, Argo CD, or Terraform
• Startup canary behavior, plus troubleshooting for canary mismatch, the retirement guard, and integrity failures

It also links to the new page from the KEK backup note in Configure the Connector Gateway.

I left out internal-only material: ADR links, Go type and function names, the illustrative Terraform/ESO pipeline, and the first-rotation precondition for pre-Active() binaries. I checked these facts against source: the chart values and checksum inputs (helm/values.yaml, templates/deployment.yaml, templates/kek-secret.yaml), the per-version canary and its log/error strings (kek_canary.go), the retirement guard message (runtime/composition.go), the metric name and that it goes through the global MeterProvider so both OTLP and Prometheus export it, and the 30-day upstream token retention (toolhive v0.50.0 DefaultRefreshTokenTTL). I also ran the jq/openssl keyring commands locally to confirm they produce a valid map that keeps the version 1 bytes unchanged.

One thing for reviewers: the internal doc says the canary verifies only the active version. The current code keeps a per-version canary and checks every version, so the page describes that behavior.

Type of change

• New documentation

Related issues/PRs

Source: enterprise/connector-gateway/docs/kek-operations.md in the enterprise platform repo.

Submitter checklist

Content and formatting

• I have reviewed the content for technical accuracy
• I have reviewed the content for spelling, grammar, and style

Navigation

• New pages include a frontmatter section with title and description at a minimum
• Sidebar navigation (sidebars.ts) updated for added, deleted, reordered, or renamed files
• Redirects added to vercel.json for moved, renamed, or deleted pages (i.e., if the URL slug changed) - N/A, no moved pages

Reviewer checklist

Content

• I have reviewed the content for technical accuracy
• I have reviewed the content for spelling, grammar, and style

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs-website	Ready	Preview	Oct 6, 2026 4:06pm UTC

[github-actions]

Visual regression failed

Commit ed83bcd6fca8 changed 1 visual snapshot across 1 screen.

Screen	Route	Viewport	Difference
nav page - Connector Gateway	/connector-gateway	Desktop	Light: 762 pixels (ratio 0.01)

Open the failed visual job · Download the full Playwright report

If the change is intentional, a repository collaborator can comment /update-snapshots to regenerate the baselines.

[danbarr]

The guide has a clear purpose, explains why rotation needs two rollouts, and provides useful recovery guidance. Please address these three operator-facing issues before merging.

This is an AI-assisted editorial and source review of the current PR head (ed83bcd6); the rotation procedure has not been executed against a cluster.

1. Correct the retirement timing guarantee (docs/connector-gateway/rotate-encryption-key.mdx, lines 182–186). “Waiting at least 30 days after the activation rollout leaves only credentials with no expiry” overstates what the wait guarantees. ToolHive retains a token until its access-token expiry plus 30 days. A token written before activation could expire after activation and therefore remain encrypted under the old key beyond that window. Expiring dynamic client registrations can also remain until their own expiry. Describe 30 days as a minimum that reduces reconnects, rather than proof that all expiring credentials have cleared. Explain that the necessary retention period depends on the remaining token and client-registration lifetimes, and that removal can still require reconnects.

2. Make the supported Secret configuration explicit (lines 23–28 and the rotation procedure). The prerequisites offer kek.value and kek.generate alternatives, but the subsequent commands and YAML consistently assume existingSecret: connector-gateway-kek. Copying that YAML into either alternative configuration introduces a second key source; the chart requires exactly one. Inline mode also requires updating the Helm value, rather than applying the external-Secret command. Keep the worked procedure scoped to existingSecret, or provide explicit substitutions for each alternative, including which source settings to retain or remove. A short prerequisite aside is insufficient for these different update paths.

3. Separate the canary warning from startup failures (lines 290–293). The message verified but could not advance to the current version appears under “Replicas refuse to start.” In the implementation, ErrKEKCanaryAdvanceFailed produces a warning and startup continues because verification already succeeded. Give this warning its own troubleshooting entry. State that verification succeeded, startup continues, and operators should investigate Redis write connectivity or permissions.

One nonblocking clarification: at lines 235–236, distinguish rolling back activeVersion from rolling back key material. Changing the active pin is the supported rollback described earlier; reverting the Secret’s contents is what can discard required versions.