fix(connectors): honor Retry-After on every retried status - #4192
Draft
ryankert01 wants to merge 1 commit into
Draft
ryankert01 wants to merge 1 commit into
ryankert01 wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4192 +/- ##
=============================================
- Coverage 87.42% 68.63% -18.80%
Complexity 1575 1575
=============================================
Files 1280 1278 -2
Lines 222704 183591 -39113
Branches 186067 146954 -39113
=============================================
- Hits 194706 125999 -68707
- Misses 23287 52814 +29527
- Partials 4711 4778 +67
🚀 New features to boost your workflow:
|
ryankert01
force-pushed
the
fix/connectors-retry-after
branch
from
September 15, 2026 09:32
92e136d to
cdf188f
Compare
`HttpRetryMiddleware` read `Retry-After` only on 429, but it retries any 5xx as well, and RFC 9110 allows the header on any 5xx. InfluxDB OSS documents it on 503 in particular, so the connector this middleware was written for asked us to come back at a known time and we guessed instead. The header now replaces the computed backoff on every status the middleware already retries, bounded by the same `max_delay` as the backoff itself. The bound is the operator's ceiling rather than a constant of our own, because the sleep runs inside the sink `consume` FFI call, which no shutdown signal cancels: a value the remote server picks must not decide how long a connector parks a runtime worker thread. Operators who want to honor longer windows raise `retry_max_delay`. A zero value now counts as absent. Honoring it spent the whole attempt budget inside one round trip, which is the opposite of what a server asking for an immediate retry wants. This bounds 429 as well, which had no ceiling at all before.
ryankert01
force-pushed
the
fix/connectors-retry-after
branch
from
September 15, 2026 18:59
cdf188f to
2e5e329
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR address?
Closes #4168
Rationale
HttpRetryMiddlewarereadRetry-Afteronly on 429, but it retries any 5xx too.What changed?
The middleware now reads
Retry-Afteron every status it already retries, so a503 is no longer retried with the server's own timing thrown away. InfluxDB OSS
v2 documents the header on 503 and does not list 429, so the connector this
middleware was written for hit that directly; InfluxDB Cloud uses 429, which is
what the original code covered.
The honored value is bounded by the same
max_delayas the computed backoff.That bound is the operator's configured ceiling rather than a constant of our
own, because the sleep runs inside the sink
consumeFFI call, which noshutdown signal cancels. A value the remote server picks must not decide how
long a connector parks a runtime worker thread. Operators who want to honor
longer windows raise
retry_max_delay.Net effect in both directions: 429 previously honored the header with no
ceiling, and is now bounded. 5xx previously ignored it and is now
server-informed under the same ceiling as before.
Two smaller fixes in the same code: a
Retry-After: 0counted as a real delayand spent the whole attempt budget inside one round trip, so zero now counts as
absent; and the header lookup uses
reqwest::header::RETRY_AFTERrather than astring literal, matching the rest of the repo.
Follow-up:
parse_retry_afteraccepts only the integer-seconds form, so the HTTP-date form that
RFC 9110 §10.2.3
also allows falls back to the computed backoff. Adding it needs an
httpdateorchronodependency, so it stays a separate PR. Tracked in #4168.Local Execution
fmt,clippy -D warningson the SDK and the three connectors usingbuild_retry_client, 174 SDK tests,cargo docunderRUSTDOCFLAGS=-D warnings.the 503 test, and the ceiling test on a 429. The two pure tests over
bounded_retry_aftercover a function this PR adds, so they pin its edgesrather than catch the old bug.
prekran and passes, exceptmarkdownlint,license-headers,trailing-whitespace,trailing-newlineandbinary-artifacts, which needbash >= 4.2; ran those directly instead.
AI Usage
then set aside, because those clients block one caller thread while this sleep
parks a shared runtime worker.