feat: support expiring Redis credentials, including AWS ElastiCache IAM auth - #1224
Open
vtremblay wants to merge 2 commits into
Open
feat: support expiring Redis credentials, including AWS ElastiCache IAM auth#1224vtremblay wants to merge 2 commits into
vtremblay wants to merge 2 commits into
Conversation
REDIS_AUTH is read once, at startup, and radix replays it on every connection the pool ever opens. A credential whose lifetime is shorter than the process therefore cannot be used: once it expires, no new connection can authenticate and the pool cannot recover without a restart. That rules out short-lived credentials from a secret manager, scheduled password rotation, and cloud providers whose Redis authentication tokens are valid for minutes. Add a CredentialProvider the dialer consults on every dial, and a file-backed implementation selected with REDIS_AUTH_FILE / REDIS_PERSECOND_AUTH_FILE: whatever writes the file owns the credential's lifetime, and a new value takes effect on the next connection. The provider wrapper has to be installed before the READONLY one, because radix uses CustomConn in place of every other Dialer field and would ignore the credentials set on it. Wrapping a dialer that already has CustomConn set now fails loudly rather than silently producing unauthenticated connections. Signed-off-by: Vincent Tremblay <vincent@vtremblay.dev>
ElastiCache IAM authentication uses a SigV4 query-presigned "connect" request as the password. It is valid for 15 minutes, so it cannot be carried by REDIS_AUTH, and nothing in ratelimit could mint it. Add a CredentialProvider that signs the token locally from the credentials the AWS SDK resolves, enabled with REDIS_AWS_IAM_AUTH. EKS Pod Identity, IRSA, ECS task roles and instance profiles all work with no credential configured in ratelimit itself. One token is shared by the whole pool for most of its validity, because ElastiCache throttles authentication requests, and reuse stops early when the AWS credentials that signed it are about to expire. The 12-hour disconnect ElastiCache applies to IAM-authenticated connections needs no special handling: the pool discards the connection and reconnects with a fresh token. Refs envoyproxy#799 Signed-off-by: Vincent Tremblay <vincent@vtremblay.dev>
vtremblay
marked this pull request as ready for review
August 28, 2026 23:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
REDIS_AUTHis resolved once, at construction:createDialercopies it intoradix.Dialer.AuthUser/AuthPass, two plain strings that radix replays on every connection the pool ever opens. A credential whose lifetime is shorter than the process therefore cannot be used. Once it lapses, no new connection can authenticate, and the pool cannot recover without a restart.AWS ElastiCache IAM authentication works exactly this way: the password is a SigV4 query-presigned request, valid for 15 minutes. #799 asked for it. The same limitation blocks Vault- and CSI-provisioned passwords and scheduled rotation.
Solution
Two commits, in order:
CredentialProviderfor credentials on every dial, then dials through the fully-configured baseradix.Dialer, so TLS, timeouts and radix's ownAUTHhandshake keep working. Ships with a file-backed provider.REDIS_AUTH_FILEpasswordorusername:password, re-read on every dialREDIS_AWS_IAM_AUTHREDIS_AWS_IAM_CACHE_NAMEREDIS_AWS_IAM_USER_IDREDIS_AWS_IAM_SERVERLESSREDIS_AWS_IAM_REGIONEach has a
REDIS_PERSECOND_twin except the region, which is shared like the TLS config already is. Everything defaults to off, two credential sources for one pool fail fast at startup, andREDIS_AUTHand Sentinel auth are untouched.IAM authentication also requires in-transit encryption on the cache, so
REDIS_TLSnormally has to be on. That is documented, not enforced:REDIS_TLSonly says whether ratelimit originates TLS, and a proxy that terminates it meets AWS's requirement with the flag off.On the dependency
aws-sdk-go-v2adds 2 direct and 12 indirect modules. That is a real cost, but not a new class of dependency here.github.com/DataDog/datadog-go/v5is already a direct, unconditional dependency selected at runtime byUSE_DOG_STATSD, next to the Prometheus and OTel exporters, andsrc/has no build tags today. This follows that precedent: compiled in, selected at runtime, does nothing unless you turn it on.The alternative was to ship the hook alone and let operators run a sidecar that presigns the token into the credential file. That keeps
go.modfree of AWS, but moves SigV4 presigning and the credential chain into an artifact this project does not ship or test. Restructuring as a build tag or separate module is fine if maintainers prefer. Commit 1 works without commit 2.Design notes
CustomConnsupersedes every otherradix.Dialerfield, so the credential wrapper goes in beforewrapDialerCloseOnReadOnly. The reverse order silently produces unauthenticated connections, and now panics at startup instead.PING. Proactive re-auth was rejected: radix exposes no hook for touching an idle pooled connection, and anAUTHat an arbitrary moment would race theEVALand pipelined commands in flight.Assumptions with no documentation behind them
X-Amz-Expires=900is what every AWS implementation hardcodes, and 15 minutes is the documented validity. Whether a larger value would be rejected or silently capped is not stated anywhere, so this does not try.AUTH <user-id> <token>, the form AWS documents. The exactHELLO 3 AUTH <user> <token>spelling is standard RESP3 rather than something AWS documents.aws-sdk-go-v2/feature/rds/authpasses explicitly. No doc states it.aws.Credentials.Expireson every mint rather than assuming a constant.Testing
New unit tests in
src/redis/credentials_test.goandsrc/redis/credentials_aws_test.go, each watched failing before the code that satisfies it. Three were checked by mutation: reversing the wrap order producesNOAUTH, removing the token mutex produces data races and duplicate signing, and disabling the credential-expiry cap lets a stale token be reused.They also cover per-dial resolution against a
minirediswhose password rotates between dials, the token's fields one by one, reuse and expiry, concurrent sharing under-race, and the settings-to-provider cases.make tests_unitandmake docker_testspass, the last against real Redis, Sentinel and Cluster instances. Formatting checked with gofmt, prettier and doctoc.