From 023dcc87c57d76f855cf17cd40c37f6a8cf59f50 Mon Sep 17 00:00:00 2001 From: Trey Date: Tue, 6 Oct 2026 09:02:45 -0700 Subject: [PATCH 1/3] Add Connector Gateway KEK rotation guide Operator-facing guide for rotating, rolling back, and retiring the gateway's key-encryption key, adapted from the internal KEK operations reference. Co-Authored-By: Claude Opus 5.5 --- .../rotate-encryption-key.mdx | 316 ++++++++++++++++++ .../configure-connector-gateway.mdx | 4 +- sidebars.ts | 1 + 3 files changed, 320 insertions(+), 1 deletion(-) create mode 100644 docs/connector-gateway/rotate-encryption-key.mdx diff --git a/docs/connector-gateway/rotate-encryption-key.mdx b/docs/connector-gateway/rotate-encryption-key.mdx new file mode 100644 index 00000000..1e50bff8 --- /dev/null +++ b/docs/connector-gateway/rotate-encryption-key.mdx @@ -0,0 +1,316 @@ +--- +title: Rotate the Connector Gateway encryption key +sidebar_label: Key rotation +description: + Rotate the key-encryption key that seals stored connector tokens without + forcing users to reconnect, then retire the old key version safely. +--- + +The Connector Gateway seals the upstream OAuth tokens and dynamic client +registrations it stores in Redis with a key-encryption key (KEK). To rotate the +KEK, you add a new key version to the KEK Secret, roll the gateway so every +replica can read it, and then switch new writes to it. Old versions stay in the +Secret, so credentials sealed under them remain readable and users don't have to +reconnect their connectors. + +This page uses the `stacklok-enterprise` release name, the `stacklok-system` +namespace, and the `connector-gateway-kek` Secret from +[Configure the Connector Gateway](../platform/enterprise-platform/configure-connector-gateway.mdx). +Adjust the names if yours differ. + +## Prerequisites + +- **The KEK in a Secret you manage**, referenced by + `connector-gateway.kek.existingSecret`. If you set `kek.value` instead, put + the same version maps shown below, base64-encoded, into `kek.value`. If you + set `kek.generate: true`, the chart preserves whatever the + `stacklok-enterprise-connector-gateway-kek` Secret holds across upgrades, so + edit that Secret in place. +- **`kubectl`, `helm`, `openssl`, and `jq`**, with access to the release + namespace. +- **A secure place to keep key material.** The commands below write keys to + local files. Store them according to your secrets-handling policy and delete + the local copies when you're done. + +## How key versions work + +The KEK Secret's `kek` entry holds either a single 32-byte key, which the +gateway treats as version 1, or a JSON _version map_ that maps positive integer +versions to base64-encoded 32-byte keys: + +```json +{ "1": "", "2": "" } +``` + +The gateway reads every version in the map and can open credentials sealed under +any of them. It seals new credentials under the _active version_, which you pin +with `connector-gateway.kek.activeVersion`. When the pin is unset, the active +version is the highest version in the map. + +Each replica reads the Secret once at startup, so a change to the Secret or to +`activeVersion` reaches a replica only when it restarts. Rotation therefore +takes two rolling restarts: the first makes the new version readable on every +replica, and the second makes it active. If you made the new version active in a +single rollout, an updated replica could seal a credential that a replica still +running the old keyring can't open. + +## Rotate the key + +### Add the new version + +1. Export the current key material: + + ```bash + kubectl get secret connector-gateway-kek \ + --namespace stacklok-system \ + --output jsonpath='{.data.kek}' | base64 -d > kek-current + ``` + +2. Build the current version map. If `wc -c < kek-current` prints `32`, the + Secret holds a single key; wrap it as version 1 without changing its bytes: + + ```bash + jq -n --arg k "$(base64 < kek-current | tr -d '\n')" '{"1": $k}' > keyring.json + ``` + + Otherwise, the Secret already holds a version map: + + ```bash + cp kek-current keyring.json + ``` + +3. Add a new key one version higher than the current highest. For example, to + add version 2: + + ```bash + jq --arg k "$(openssl rand -base64 32)" '. + {"2": $k}' keyring.json > keyring-new.json + ``` + +4. Replace the Secret's contents with the new map: + + ```bash + kubectl create secret generic connector-gateway-kek \ + --namespace stacklok-system \ + --from-file=kek=./keyring-new.json \ + --dry-run=client --output yaml | kubectl apply -f - + ``` + + If External Secrets Operator or another tool syncs this Secret from a secrets + manager, write the new map to the source instead, and confirm the sync has + updated the Secret in the cluster before you continue. + +5. In your values file, pin the active version to the _current_ version, and set + `keyringGeneration` to a new value: + + ```yaml title="values.yaml" + connector-gateway: + kek: + existingSecret: 'connector-gateway-kek' + # highlight-start + activeVersion: '1' + keyringGeneration: '2' + # highlight-end + ``` + + The pin keeps new writes on version 1 while replicas restart. Without it, + each restarted replica would seal under version 2 immediately. The chart + restarts the gateway only when a KEK-related value changes, and editing the + Secret changes none of them, so change `keyringGeneration` on every rotation + to force the rollout. Any new string works; using the new version number + keeps it easy to track. + +6. Upgrade the release and wait for the rollout to finish: + + ```bash + helm upgrade stacklok-enterprise \ + oci://oci.stacklok.com/stacklok-enterprise//stacklok-enterprise-platform \ + --version \ + --namespace stacklok-system \ + --values values.yaml + + kubectl rollout status deployment/stacklok-enterprise-connector-gateway \ + --namespace stacklok-system + ``` + + Continue only when the rollout completes and no pod from the previous + ReplicaSet is still running. On this rollout, the first replica to start logs + a `no KEK canary found` warning for version 2, which is expected. See + [Startup checks](#startup-checks). + +### Activate the new version + +Set `activeVersion` to the new version, then upgrade the release again and wait +for the rollout: + +```yaml title="values.yaml" +connector-gateway: + kek: + existingSecret: 'connector-gateway-kek' + # highlight-next-line + activeVersion: '2' + keyringGeneration: '2' +``` + +Changing `activeVersion` restarts the gateway on its own. Once the rollout +completes, new credentials are sealed under version 2, and credentials sealed +under version 1 stay readable. Keep `activeVersion` pinned explicitly from here +on, rather than unsetting it. + +Keep version 1 in the Secret until you [retire it](#retire-an-old-version). +Credentials move to version 2 gradually, as users sign in and the gateway +refreshes their tokens. + +## Roll back a rotation + +To make an earlier version active again, set `activeVersion` back to it and +upgrade the release. Because every version is still in the Secret, credentials +sealed under either version stay readable and no one has to reconnect. This +works only while the version you return to is still in the Secret, so don't +retire a version you might need to roll back to. + +## Retire an old version + +Removing a version from the Secret makes every credential still sealed under it +unreadable. The gateway treats those credentials as missing and recovers them: +users reconnect the affected connectors, and the gateway re-registers clients +for providers that use dynamic client registration. Waiting before you remove +the version keeps that wave small. + +1. **Confirm the version is inactive.** `activeVersion` names a newer version, + and that rollout has completed on every replica. + +2. **Wait for stored tokens to turn over.** Every token refresh reseals the + credential under the active version, and a stored upstream token expires 30 + days after its access token does. Waiting at least 30 days after the + activation rollout leaves only credentials with no expiry, such as clients + registered with a non-expiring client secret, under the old version. + +3. **Remove the version.** Export the current version map as in + [Add the new version](#add-the-new-version), delete the old entry, and apply + the result. For example, to remove version 1: + + ```bash + kubectl get secret connector-gateway-kek \ + --namespace stacklok-system \ + --output jsonpath='{.data.kek}' | base64 -d > keyring.json + jq 'del(."1")' keyring.json > keyring-retired.json + kubectl create secret generic connector-gateway-kek \ + --namespace stacklok-system \ + --from-file=kek=./keyring-retired.json \ + --dry-run=client --output yaml | kubectl apply -f - + ``` + +4. **Roll the gateway.** Change `keyringGeneration` to a new value, upgrade the + release, and wait for the rollout to complete. Replicas that haven't + restarted yet keep reading the removed version until they do. + +5. **Watch the recovery.** The `kek_unsealable_reads_total` counter increments + each time the gateway reads a credential sealed under a version that's no + longer in the Secret, labeled with that `version`. Its rate shows the + remaining reconnect wave draining. The counter tracks read events, so it + stays at zero while the version is still present and can't tell you in + advance how many credentials still use it. Export it with the gateway's + [metrics](./telemetry.mdx). + +When a removed version still seals a dynamic client registration, the gateway +registers a new client with that provider on startup. This has two effects to +plan for: + +- **Every user of that connector reconnects**, not only those whose tokens were + sealed under the removed version, because refresh tokens are bound to the + client they were issued to. Schedule the removal for a window where that's + acceptable. +- **The old client stays registered at the provider.** The gateway logs a + warning with the `replaced_client_id` field so you can find and delete it. + +Replicas coordinate through Redis, so the gateway registers one new client per +provider regardless of the replica count. + +## Keep the KEK Secret append-only + +Treat the KEK Secret as append-only: add higher versions, never change the bytes +of an existing version, and remove a version only through the retirement steps +above. Several routine operations can break this rule without warning: + +- `helm rollback` to a revision whose values carried an older `kek.value` or + `activeVersion`. +- An Argo CD sync to an earlier Git revision of the Secret or of the + `ExternalSecret` that populates it. +- A Terraform apply of an earlier configuration, or any change that destroys and + recreates the secret resource, such as `terraform taint`. + +If a rollback drops a version that still seals credentials, the gateway starts +normally and those users reconnect, as in a retirement without the waiting +period. If it drops the active version, every replica refuses to start. If you +manage the key with Terraform, set `lifecycle.prevent_destroy` on the secret +resource, and prevent in-place edits to the key value with +`lifecycle.ignore_changes` or an equivalent write-once rule. + +## Startup checks + +The gateway verifies its keyring against a _canary_ in Redis before it serves +traffic. For each key version, the first replica to start with that version +seals a known value under it and stores the result. Every later startup opens +those values to confirm each version still holds the same key. + +A replica logs `no KEK canary found; establishing a new baseline` at warning +level when it seeds a canary. That's expected on a first install and on the +first rollout after you add a version. On an install that has already stored +credentials, an unexpected occurrence means the canary was deleted or evicted +from Redis, so the gateway can no longer detect a wrong key for that version. +Check your Redis eviction policy. + +## Next steps + +- [Collect Connector Gateway telemetry](./telemetry.mdx) to export + `kek_unsealable_reads_total` before you retire a version. +- [Forward Connector Gateway audit logs](./forward-audit-logs.mdx) to keep a + record of every tool call. + +## Related information + +- [Configure the Connector Gateway](../platform/enterprise-platform/configure-connector-gateway.mdx) - + creating the KEK Secret and the other gateway secrets +- [Connector authentication](./connector-authentication.mdx) - how users connect + to connectors that use OAuth + +## Troubleshooting + +
+Replicas refuse to start with a `kek canary` error + +The error wraps +`kek canary: cannot unwrap under the KEK version it is sealed under`. A version +in the mounted Secret holds different bytes than the version that sealed its +canary: the Secret was regenerated, overwritten, or restored from the wrong +backup. Restore the original bytes for that version. Don't delete the canary to +get past the check, because the gateway would then accept the wrong key and fail +to open every credential sealed under that version. + +If the error instead reads +`kek canary: verified but could not advance to the current version`, the key is +correct and the gateway couldn't write to Redis. Check Redis connectivity and +permissions. + +
+ +
+Replicas refuse to start with a `kek retirement guard` error + +`kek.activeVersion` names a version the mounted Secret doesn't hold, usually +because the active version was removed or a rollback reverted the Secret. Either +restore that version to the Secret, or set `activeVersion` to a version the +Secret holds, then upgrade the release. + +
+ +
+Reads fail with a credential integrity error + +The credential's version is present in the Secret, but its key doesn't open the +credential, which means that version's bytes changed. The gateway keeps the +credential and doesn't prompt the user to reconnect. Restore the original bytes +for that version to make every affected credential readable again. Removing the +version would discard those credentials instead. + +
diff --git a/docs/platform/enterprise-platform/configure-connector-gateway.mdx b/docs/platform/enterprise-platform/configure-connector-gateway.mdx index c8ca5c2e..6ec1266c 100644 --- a/docs/platform/enterprise-platform/configure-connector-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-connector-gateway.mdx @@ -58,7 +58,9 @@ kubectl create secret generic connector-gateway-kek \ ``` Keep a backup of the key. Losing or replacing it makes every credential the -gateway has stored unreadable. +gateway has stored unreadable. To change the key later, add a new version rather +than replacing it, as described in +[Rotate the Connector Gateway encryption key](../../connector-gateway/rotate-encryption-key.mdx). **Authorization server keys.** The signing key keeps issued tokens valid across restarts and replicas, and the HMAC secret keeps authorization codes and refresh diff --git a/sidebars.ts b/sidebars.ts index 189c2493..129105d9 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -493,6 +493,7 @@ const connectorGatewaySidebar: SidebarsConfig[string] = [ 'connector-gateway/tool-usage', 'connector-gateway/telemetry', 'connector-gateway/forward-audit-logs', + 'connector-gateway/rotate-encryption-key', ]; const aiGatewaySidebar: SidebarsConfig[string] = [ From ed83bcd6fca8ffe6401230b01ce3727f99e50bd7 Mon Sep 17 00:00:00 2001 From: Trey Date: Tue, 6 Oct 2026 09:04:49 -0700 Subject: [PATCH 2/3] List KEK metric on the telemetry page Co-Authored-By: Claude Opus 5.5 --- docs/connector-gateway/telemetry.mdx | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/connector-gateway/telemetry.mdx b/docs/connector-gateway/telemetry.mdx index b4c22f18..14dc0da2 100644 --- a/docs/connector-gateway/telemetry.mdx +++ b/docs/connector-gateway/telemetry.mdx @@ -93,9 +93,11 @@ gateway automatically. The chart creates no Service port or ServiceMonitor for metrics; if you use the Prometheus Operator, create a PodMonitor that selects the gateway pods and targets the `metrics` port. -The endpoint exposes Go runtime and process metrics for the gateway. For -per-tool call counts, use traces, [audit events](./forward-audit-logs.mdx), or -the [Tool Usage](./tool-usage.mdx) screen. +The endpoint exposes Go runtime and process metrics for the gateway, plus +`kek_unsealable_reads_total`, which tracks reads of credentials sealed under a +[retired encryption key version](./rotate-encryption-key.mdx#retire-an-old-version). +For per-tool call counts, use traces, [audit events](./forward-audit-logs.mdx), +or the [Tool Usage](./tool-usage.mdx) screen. ## Next steps From 949c3034a152829525a7aad4423d244b7313cd4f Mon Sep 17 00:00:00 2001 From: Trey Date: Thu, 8 Oct 2026 07:29:24 -0700 Subject: [PATCH 3/3] Address KEK rotation guide review feedback Scope the procedure to existingSecret with a migration path, qualify the retirement wait, split the non-fatal canary warning into its own troubleshooting entry, and clarify which rollbacks are safe. Co-Authored-By: Claude Opus 5.5 --- .../rotate-encryption-key.mdx | 75 +++++++++++++++---- 1 file changed, 59 insertions(+), 16 deletions(-) diff --git a/docs/connector-gateway/rotate-encryption-key.mdx b/docs/connector-gateway/rotate-encryption-key.mdx index 1e50bff8..ee90e764 100644 --- a/docs/connector-gateway/rotate-encryption-key.mdx +++ b/docs/connector-gateway/rotate-encryption-key.mdx @@ -21,17 +21,48 @@ Adjust the names if yours differ. ## Prerequisites - **The KEK in a Secret you manage**, referenced by - `connector-gateway.kek.existingSecret`. If you set `kek.value` instead, put - the same version maps shown below, base64-encoded, into `kek.value`. If you - set `kek.generate: true`, the chart preserves whatever the - `stacklok-enterprise-connector-gateway-kek` Secret holds across upgrades, so - edit that Secret in place. + `connector-gateway.kek.existingSecret`. The procedures on this page edit that + Secret directly. If the chart manages the key instead, through `kek.value` or + `kek.generate`, first + [move it into your own Secret](#move-a-chart-managed-key-into-your-own-secret). - **`kubectl`, `helm`, `openssl`, and `jq`**, with access to the release namespace. - **A secure place to keep key material.** The commands below write keys to local files. Store them according to your secrets-handling policy and delete the local copies when you're done. +### Move a chart-managed key into your own Secret + +The chart accepts exactly one key source, so switching to `existingSecret` means +copying the key into a new Secret and removing the old setting in the same +upgrade. The key bytes don't change, so stored credentials stay readable. + +1. Copy the key from the chart-managed Secret into a new Secret: + + ```bash + kubectl get secret stacklok-enterprise-connector-gateway-kek \ + --namespace stacklok-system \ + --output jsonpath='{.data.kek}' | base64 -d > kek-current + kubectl create secret generic connector-gateway-kek \ + --namespace stacklok-system \ + --from-file=kek=./kek-current + ``` + +2. In your values file, remove `kek.value`, `kek.generate`, and + `kek.allowGenerateOnUpgrade`, and reference the new Secret: + + ```yaml title="values.yaml" + connector-gateway: + kek: + existingSecret: 'connector-gateway-kek' + ``` + +3. Upgrade the release and wait for the rollout to complete, using the commands + in [Add the new version](#add-the-new-version). + +The chart leaves `stacklok-enterprise-connector-gateway-kek` in place. Once the +rollout completes, delete it so the namespace holds a single copy of the key. + ## How key versions work The KEK Secret's `kek` entry holds either a single 32-byte key, which the @@ -179,11 +210,16 @@ the version keeps that wave small. 1. **Confirm the version is inactive.** `activeVersion` names a newer version, and that rollout has completed on every replica. -2. **Wait for stored tokens to turn over.** Every token refresh reseals the - credential under the active version, and a stored upstream token expires 30 - days after its access token does. Waiting at least 30 days after the - activation rollout leaves only credentials with no expiry, such as clients - registered with a non-expiring client secret, under the old version. +2. **Wait for stored credentials to turn over.** Every token refresh reseals the + credential under the active version. A stored upstream token that isn't + refreshed expires 30 days after its access token does, so a token whose + access token outlives the activation stays under the old version for longer + than 30 days. A dynamic client registration stays until its client secret + expires, and one with a non-expiring secret never does. Wait at least 30 days + after the activation rollout, and longer if your providers issue long-lived + access tokens or client secrets. Waiting reduces the reconnect wave but + doesn't guarantee every credential has moved, so expect some reconnects after + removal. 3. **Remove the version.** Export the current version map as in [Add the new version](#add-the-new-version), delete the old entry, and apply @@ -232,8 +268,10 @@ Treat the KEK Secret as append-only: add higher versions, never change the bytes of an existing version, and remove a version only through the retirement steps above. Several routine operations can break this rule without warning: -- `helm rollback` to a revision whose values carried an older `kek.value` or - `activeVersion`. +- `helm rollback` to a revision from before you moved to `existingSecret`, whose + `kek.value` carried older key material. Rolling back `activeVersion` alone is + safe, as described in [Roll back a rotation](#roll-back-a-rotation); reverting + the key material is what drops versions. - An Argo CD sync to an earlier Git revision of the Secret or of the `ExternalSecret` that populates it. - A Terraform apply of an earlier configuration, or any change that destroys and @@ -287,10 +325,15 @@ backup. Restore the original bytes for that version. Don't delete the canary to get past the check, because the gateway would then accept the wrong key and fail to open every credential sealed under that version. -If the error instead reads -`kek canary: verified but could not advance to the current version`, the key is -correct and the gateway couldn't write to Redis. Check Redis connectivity and -permissions. + + +
+Replicas log a `kek canary: verified but failed to advance` warning + +The replica verified every key version and started normally, but it couldn't +write an updated canary to Redis after a rotation. The next startup retries the +write. Check that the gateway can write to Redis: connectivity, the Redis user's +permissions, and memory limits.