Skip to content

feat: support expiring Redis credentials, including AWS ElastiCache IAM auth - #1224

Open
vtremblay wants to merge 2 commits into
envoyproxy:mainfrom
vtremblay:redis-dynamic-credentials
Open

feat: support expiring Redis credentials, including AWS ElastiCache IAM auth#1224
vtremblay wants to merge 2 commits into
envoyproxy:mainfrom
vtremblay:redis-dynamic-credentials

Conversation

@vtremblay

Copy link
Copy Markdown

Problem

REDIS_AUTH is resolved once, at construction: createDialer copies it into radix.Dialer.AuthUser/AuthPass, two plain strings that radix replays on every connection the pool ever opens. A credential whose lifetime is shorter than the process therefore cannot be used. Once it lapses, no new connection can authenticate, and the pool cannot recover without a restart.

AWS ElastiCache IAM authentication works exactly this way: the password is a SigV4 query-presigned request, valid for 15 minutes. #799 asked for it. The same limitation blocks Vault- and CSI-provisioned passwords and scheduled rotation.

Solution

Two commits, in order:

  1. A credential hook, no new dependencies. A dialer wrapper asks a CredentialProvider for credentials on every dial, then dials through the fully-configured base radix.Dialer, so TLS, timeouts and radix's own AUTH handshake keep working. Ships with a file-backed provider.
  2. An ElastiCache IAM provider that signs the token locally from whatever the AWS SDK resolves, so Pod Identity, IRSA and task roles work with nothing configured in ratelimit.
Variable Purpose
REDIS_AUTH_FILE File holding password or username:password, re-read on every dial
REDIS_AWS_IAM_AUTH Authenticate with ElastiCache IAM tokens
REDIS_AWS_IAM_CACHE_NAME Cache name, or replication group ID, the token is signed for
REDIS_AWS_IAM_USER_ID ElastiCache user id to connect as
REDIS_AWS_IAM_SERVERLESS Sign tokens for a serverless cache
REDIS_AWS_IAM_REGION Signing region, defaulting to what the AWS SDK resolves

Each has a REDIS_PERSECOND_ twin except the region, which is shared like the TLS config already is. Everything defaults to off, two credential sources for one pool fail fast at startup, and REDIS_AUTH and Sentinel auth are untouched.

IAM authentication also requires in-transit encryption on the cache, so REDIS_TLS normally has to be on. That is documented, not enforced: REDIS_TLS only says whether ratelimit originates TLS, and a proxy that terminates it meets AWS's requirement with the flag off.

On the dependency

aws-sdk-go-v2 adds 2 direct and 12 indirect modules. That is a real cost, but not a new class of dependency here. github.com/DataDog/datadog-go/v5 is already a direct, unconditional dependency selected at runtime by USE_DOG_STATSD, next to the Prometheus and OTel exporters, and src/ has no build tags today. This follows that precedent: compiled in, selected at runtime, does nothing unless you turn it on.

The alternative was to ship the hook alone and let operators run a sidecar that presigns the token into the credential file. That keeps go.mod free of AWS, but moves SigV4 presigning and the credential chain into an artifact this project does not ship or test. Restructuring as a build tag or separate module is fine if maintainers prefer. Commit 1 works without commit 2.

Design notes

  • The order matters when wrapping the dialer. CustomConn supersedes every other radix.Dialer field, so the credential wrapper goes in before wrapDialerCloseOnReadOnly. The reverse order silently produces unauthenticated connections, and now panics at startup instead.
  • The 12-hour forced disconnect needs no new machinery. The pool discards the closed connection and dials a replacement carrying a fresh token, usually on its own periodic PING. Proactive re-auth was rejected: radix exposes no hook for touching an idle pooled connection, and an AUTH at an arbitrary moment would race the EVAL and pipelined commands in flight.
  • One token is shared by the whole pool for most of its validity, because ElastiCache throttles authentication requests server-side. Reuse also stops at the expiry of the credentials that signed the token, since a presigned request expires with them. AWS documents that for presigned requests generally, in the S3 presigned-URL guide; no ElastiCache page covers the interaction. The cap is stricter than AWS's own ElastiCache sample, which memoizes a flat 900 seconds with no credential check.

Assumptions with no documentation behind them

  • X-Amz-Expires=900 is what every AWS implementation hardcodes, and 15 minutes is the documented validity. Whether a larger value would be rejected or silently capped is not stated anywhere, so this does not try.
  • The token is presented as AUTH <user-id> <token>, the form AWS documents. The exact HELLO 3 AUTH <user> <token> spelling is standard RESP3 rather than something AWS documents.
  • The payload hash is SHA-256 of the empty string, matching what the signers in AWS's own examples compute for a bodyless GET, and what aws-sdk-go-v2/feature/rds/auth passes explicitly. No doc states it.
  • EKS does not publish a credential lifetime for Pod Identity, so the reuse deadline is read off aws.Credentials.Expires on every mint rather than assuming a constant.

Testing

New unit tests in src/redis/credentials_test.go and src/redis/credentials_aws_test.go, each watched failing before the code that satisfies it. Three were checked by mutation: reversing the wrap order produces NOAUTH, removing the token mutex produces data races and duplicate signing, and disabling the credential-expiry cap lets a stale token be reused.

They also cover per-dial resolution against a miniredis whose password rotates between dials, the token's fields one by one, reuse and expiry, concurrent sharing under -race, and the settings-to-provider cases.

make tests_unit and make docker_tests pass, the last against real Redis, Sentinel and Cluster instances. Formatting checked with gofmt, prettier and doctoc.

REDIS_AUTH is read once, at startup, and radix replays it on every
connection the pool ever opens. A credential whose lifetime is shorter
than the process therefore cannot be used: once it expires, no new
connection can authenticate and the pool cannot recover without a
restart. That rules out short-lived credentials from a secret manager,
scheduled password rotation, and cloud providers whose Redis
authentication tokens are valid for minutes.

Add a CredentialProvider the dialer consults on every dial, and a
file-backed implementation selected with REDIS_AUTH_FILE /
REDIS_PERSECOND_AUTH_FILE: whatever writes the file owns the
credential's lifetime, and a new value takes effect on the next
connection.

The provider wrapper has to be installed before the READONLY one,
because radix uses CustomConn in place of every other Dialer field and
would ignore the credentials set on it. Wrapping a dialer that already
has CustomConn set now fails loudly rather than silently producing
unauthenticated connections.

Signed-off-by: Vincent Tremblay <vincent@vtremblay.dev>
ElastiCache IAM authentication uses a SigV4 query-presigned "connect"
request as the password. It is valid for 15 minutes, so it cannot be
carried by REDIS_AUTH, and nothing in ratelimit could mint it.

Add a CredentialProvider that signs the token locally from the
credentials the AWS SDK resolves, enabled with REDIS_AWS_IAM_AUTH. EKS
Pod Identity, IRSA, ECS task roles and instance profiles all work with
no credential configured in ratelimit itself.

One token is shared by the whole pool for most of its validity, because
ElastiCache throttles authentication requests, and reuse stops early
when the AWS credentials that signed it are about to expire. The
12-hour disconnect ElastiCache applies to IAM-authenticated connections
needs no special handling: the pool discards the connection and
reconnects with a fresh token.

Refs envoyproxy#799

Signed-off-by: Vincent Tremblay <vincent@vtremblay.dev>
@vtremblay
vtremblay marked this pull request as ready for review August 28, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant