feat(server): RETRY-spec conformance in FDv1 streaming and polling (SDK-2789) - #200
feat(server): RETRY-spec conformance in FDv1 streaming and polling (SDK-2789)#200tanderson-ld wants to merge 2 commits into
Conversation
f87cd37 to
a754251
Compare
…polling data sources (SDK-2789) Guided by the server-sdk-guide.md in sdk-scratchpad; analogous to the Go server SDK's reference implementation. The behavioral change: HTTP responses that today cause a data source to permanently stop (notably 401, 403, other 4xx) and TLS/certificate validation failures are no longer terminal. Streaming enters an extended backoff regime (5 min -> 1 hour, doubling); polling continues at its configured cadence with extended-regime waits between failing polls. Recovery from either regime uses a healthy-operation reset (60 s of continuous connectivity for streaming; two consecutive successful polls for polling). Scope: FDv1 streaming and polling data sources under `lib/sdk/server/src/main/java/com/launchdarkly/sdk/server/`. FDv2 is out of scope for this epic and is deferred to a future one; nothing in `datasourcev2/` or the DataSystem-related code paths is touched. The classifier this depends on (`FailureClass` + `HttpErrors.classify*`) lives in `launchdarkly-java-sdk-internal` and ships in its own PR. Highlights: - PollingStrategy: new state-machine encapsulation with onFailure(class) / onSuccess() / nextWait() methods. State: n (formula input), initialDelay, maxDelay, priorPollWasSuccessful. Wait floor: max(pollInterval, T - J). Two-consecutive-successes returns from extended to normal regime. - PollingProcessor: rewired to a self-driven loop using strategy.nextWait(). Removed the State.OFF permanent-stop path entirely; state stays INITIALIZING/INTERRUPTED with a lastError. - StreamProcessor: consumes okhttp-eventsource's new multi-strategy retry API (see launchdarkly/okhttp-eventsource#110). On UNEXPECTED classification, activates the extended-regime RetryDelayStrategy on the underlying EventSource; the library's built-in healthy-op reset returns to normal-regime timing after 60 s of continuous connectivity. - Constructor plumbing: PollingProcessor and StreamProcessor take extendedInitialReconnectDelay, extendedStreamMaxRetryDelay, retryResetInterval, and extendedInitialDelay as constructor parameters; package-private defaults threaded through ComponentsImpl. - DataSourceStatusProvider Javadocs: State.INITIALIZING, State.OFF, State.INTERRUPTED, and getStateSince OFF-case updated to reflect the new semantics (no HTTP-error -> OFF transition). - LDClient constructor Javadoc: describes an SDK-key rejection as ongoing background retry rather than an "unsuccessful initialization" that reads as terminal. - Contract test service: declares retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling capabilities. Tests: - Unit tests: full test suite green. New coverage for the strategy state machine (PollingStrategyTest) and extended-regime timing observation in StreamProcessorTest. Existing 401/403 tests rewritten to assert extended-regime retry rather than permanent stop. - Contract tests via sdk-test-harness PR #404 (RETRY-conformance tests): 7/7 parallel shards pass end-to-end at production timing (5-minute extended-initial-delay), ~12 min wall clock. CI: intentionally red on this PR until launchdarkly/okhttp-eventsource#110 releases okhttp-eventsource 5.0.0 and #204 releases launchdarkly-java-sdk-internal 1.11.0. The multi-strategy retry API this SDK relies on is only in that eventsource PR's branch, and the classifier helpers are only in that internal-artifact PR's branch. Once both are released, bump both versions in lib/sdk/server/build.gradle.
9394a65 to
6d0c138
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6d0c138. Configure here.
|
CI Failures will be resolved once okhttp-eventsource and internal release. But waiting on initial review in this PR to make sure all the APIs on okhttp-eventsource and internal aren't going to need changes. |
Aligns with StreamProcessor.close() and FDv2DataSource.close(), both of which call updateStatus(State.OFF, null) after closing their upstream I/O. Also aligns with the PR's updated Javadoc for State.OFF, which now describes it as the time the data source "stopped operation" (i.e., close), rather than the pre-PR meaning of "encountered an unrecoverable error".
…rrors (SDK-2789) (#204) ## Summary Introduces a shared network-failure classifier in `launchdarkly-java-sdk-internal` that categorizes failures as either **NORMAL** (typically transient) or **UNEXPECTED** (indicative of a longer-lived condition — e.g., invalid SDK key, TLS misconfiguration). Downstream SDKs can use the classification to select between normal-regime and extended-regime backoff. Enables the server SDK's RETRY-spec conformance work in [SDK-2789](https://launchdarkly.atlassian.net/browse/SDK-2789) (see PR #200), which needs these helpers to select its retry regime. ## What changed - **`FailureClass` enum** — `NORMAL` and `UNEXPECTED`, with a package-private cause-chain scan for TLS / certificate exceptions. - **`HttpErrors.classifyHTTPFailure(int)`** — Returns `NORMAL` for 400 / 408 / 429, 5xx, and any other status the SDK treats as a failure; returns `UNEXPECTED` for other 4xx (401 / 403 / etc.). - **`HttpErrors.classifyTransportFailure(Throwable)`** — Returns `UNEXPECTED` if TLS or certificate validation appears anywhere in the exception chain; `NORMAL` otherwise. - **`HttpErrors.classifyAndLogHTTPFailure` / `classifyAndLogTransportFailure`** — Classify, log at the appropriate level (Error for `UNEXPECTED`, Warn for `NORMAL`), and return the classification. - **`HttpErrors.isHttpErrorRecoverable` and `checkIfErrorIsRecoverableAndLog` are now `@Deprecated`.** The boolean "give up permanently" contract doesn't fit callers that keep retrying regardless of classification; the Javadoc points migrations at the new helpers. Existing callers keep working — deprecation is source-compatible. ## Testing - New `HttpErrorsClassificationTest` covers the full HTTP status matrix (400/408/429/5xx = NORMAL, 401/403/404/418/451 = UNEXPECTED, non-4xx/non-5xx = NORMAL) and the transport-exception matrix (ordinary I/O = NORMAL, SSL/certificate = UNEXPECTED, TLS as a cause of a wrapper exception = UNEXPECTED). - Existing `lib/shared/internal` tests continue to pass; the deprecated methods still exercise their original behavior. ## Test plan for reviewers - [ ] Confirm the 4xx / 5xx split (400 / 408 / 429 = NORMAL, other 4xx = UNEXPECTED) matches expectations. - [ ] Confirm transport-level TLS / certificate failures at any depth in the cause chain are surfaced as `UNEXPECTED`. - [ ] Sanity-check the deprecation Javadocs — they name the migration target concretely and give a one-line reason (existing boolean contract vs. new tri-state usage). ## Downstream Once this PR merges and release-please publishes `launchdarkly-java-sdk-internal 1.11.0`, launchdarky/java-core#200 will bump its `build.gradle` dep to 1.11.0 and consume the classifier from the server SDK's data sources. [SDK-2789]: https://launchdarkly.atlassian.net/browse/SDK-2789?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Adds shared **internal** network failure classification so data sources can pick **normal** vs **extended** backoff instead of treating some errors as “stop retrying forever.” > > Introduces a `FailureClass` enum (`NORMAL` vs `UNEXPECTED`) and `HttpErrors` helpers to classify HTTP status codes (e.g. 400/408/429 and 5xx → `NORMAL`; most other 4xx including 401/403 → `UNEXPECTED`) and transport exceptions (TLS/certificate causes anywhere in the chain → `UNEXPECTED`). New `classifyAndLogHttpFailure` / `classifyAndLogTransportFailure` classify, log at **Error** vs **Warn**, and return the class for the caller. > > `isHttpErrorRecoverable` and `checkIfErrorIsRecoverableAndLog` are **deprecated** (behavior unchanged); Javadoc points callers at the new APIs. Unit tests cover the HTTP and transport classification matrices. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f67fd5e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
jsonbailey
left a comment
There was a problem hiding this comment.
Reviewed this alongside #204 and okhttp-eventsource#110 to check the API assumptions against the source rather than take the description's word for them. Broadly this holds up well — I specifically tried to break the two-consecutive-successes reset, the activateRetryDelayStrategy timing, and the polling loop lifecycle, and all three survived. The #110 integration is correct on every point I checked, including the one I most expected to be wrong: activation is a pointer swap that does not reset progression, so repeated 401s really do double 5→10→20→40→60 min. Using an explicit inExtended flag instead of comparing initialDelay == normalInterval avoids a real bug, and nInExtendedDoublesEvenWhenPollIntervalEqualsExtendedInitial is a good regression test for exactly that case.
I have a longer list of medium/low observations I'll hold for now. Two things I think block merge:
1. Won't compile against #204 HEAD. classifyAndLogHTTPFailure was renamed to classifyAndLogHttpFailure in #204's f67fd5e ("address review feedback on classifier helpers"). This branch predates that rename, and the local mavenLocal() snapshot is what hides it. Two call sites, inline below. Calling it out separately because it is not the unreleased-dependency-pin issue the description covers — it survives releasing both artifacts.
2. Worth one test before deciding anything else: the TLS classification may be far broader than intended. FailureClass.hasTlsOrCertificateCause (in #204) returns UNEXPECTED for any SSLException or GeneralSecurityException anywhere in the cause chain, and StreamIOException sets the underlying IOException as its cause, so the chain is reachable from StreamProcessor.handleError. Since all LD traffic is HTTPS, every transport error against the real endpoint arrives through the TLS layer — so SSLHandshakeException: Remote host terminated the handshake from a load balancer or proxy dropping mid-handshake reads as UNEXPECTED and pushes streaming into 5 min–1 hr backoff for a transient blip. The compounding part: a flapping stream never accumulates the 60 s of continuous connectivity the healthy-op reset needs, so it cannot climb back out.
I could not settle whether JSSE actually throws SSLException for these cases (no JVM available where I was reviewing), and about 20 lines answers it: open an SSLServerSocket, complete the handshake, then setSoLinger(true, 0) + close(), and read from the client — observe the exception type. Worth running before deciding: if the classification is as broad as it reads, this is the largest production-impact item in the change; if it isn't, the test costs nothing. Narrowing to SSLHandshakeException / SSLPeerUnverifiedException / CertificateException, or excluding causes that are themselves SocketException / SocketTimeoutException, would settle it either way.
On your test plan: the PollingStrategy transition semantics and the handleError ordering both check out — I traced the reset across intervening-failure and mixed NORMAL/UNEXPECTED orderings and found no off-by-one, and inExtended is cleared so a later UNEXPECTED re-fires the transition. No objection to the comment style.
| task.cancel(true); | ||
| task = null; | ||
| } | ||
| FailureClass failureClass = HttpErrors.classifyAndLogHTTPFailure( |
There was a problem hiding this comment.
classifyAndLogHTTPFailure → classifyAndLogHttpFailure.
#204 renamed this in f67fd5e; the only such method at #204 HEAD is classifyAndLogHttpFailure (HttpErrors.java:156), and #204's own deprecation Javadoc at HttpErrors.java:76 uses the lowercase form too. git log -S in #204 shows 52c05c2 added it all-caps and f67fd5e renamed it, so this branch was written against the earlier revision.
This will fail compileJava once #204 merges and internal 1.11.0 publishes — separate from the unreleased-pin issue, since it survives releasing both artifacts.
| if (e instanceof StreamHttpErrorException) { | ||
| int status = ((StreamHttpErrorException)e).getCode(); | ||
| ErrorInfo errorInfo = ErrorInfo.fromHttpError(status); | ||
| failureClass = HttpErrors.classifyAndLogHTTPFailure(logger, status, ERROR_CONTEXT_MESSAGE, WILL_RETRY_MESSAGE); |
There was a problem hiding this comment.
Second call site for the classifyAndLogHTTPFailure → classifyAndLogHttpFailure rename — see the note on PollingProcessor.java:126.

Summary
Implements RETRY-spec conformance in the Java server SDK's FDv1 streaming and polling data sources (SDK-2789). This PR is scoped to the server SDK only; the classifier helpers it consumes ship separately in #204.
Behavioral change. HTTP responses that today cause an FDv1 data source to permanently stop (notably 401 / 403 / other 4xx) and TLS / certificate validation failures are no longer terminal. Streaming enters an extended-regime backoff (5 min initial → 1 hr max, doubling); polling continues at its configured cadence but engages the extended-regime wait after an UNEXPECTED failure. Either regime returns to normal after 60 s of continuous healthy operation (streaming) or two consecutive successful polls (polling).
Scope. FDv1 streaming and polling under
lib/sdk/server/. FDv2, event delivery, and other network callers are unchanged.What changed
PollingStrategy— New state-machine encapsulation withonFailure(class)/onSuccess()/nextWait(). State:n(formula input),initialDelay,maxDelay,priorPollWasSuccessful. Wait floor ismax(pollInterval, T − J); two consecutive successes reset from extended → normal.PollingProcessor— Rewired to a self-driven loop that consultsstrategy.nextWait()between attempts. TheState.OFFpermanent-stop path is removed; the state staysINITIALIZINGorINTERRUPTEDwith alastError.StreamProcessor— Consumes okhttp-eventsource's new multi-strategy retry API from launchdarkly/okhttp-eventsource#110. OnUNEXPECTEDclassification it callsactivateRetryDelayStrategyon the underlyingEventSourceto switch into the extended-regimeRetryDelayStrategy. The library's built-in healthy-op reset returns the SDK to normal-regime timing after 60 s of continuous connectivity.DataSourceStatusProviderdocs —State.INITIALIZING,State.OFF,State.INTERRUPTED, andgetStateSinceOFF-case Javadocs updated to reflect the new semantics (no HTTP-error → OFF transition). Aligned with the Go server SDK's parallel doc adjustments.LDClientconstructor Javadoc — Wording tightened so a "wrong SDK key" scenario is described as ongoing retry in the background, not as an "unsuccessful initialization" that reads as terminal.retry-conformance-fdv1-streamingandretry-conformance-fdv1-pollingcapabilities.The classifier this SDK depends on —
FailureClassandHttpErrors.classify*helpers — ships in #204.Testing
PollingStrategyTest(strategy state machine), plus extended-regime timing-observation tests inStreamProcessorTest. Existing 401 / 403 tests were rewritten to assert extended-regime retry rather than permanent stop.Test plan for reviewers
PollingStrategytransition semantics: normal → extended fires exactly once perUNEXPECTEDfailure; two consecutive successful polls fully reset (n = 0, delay bounds back to normal,inExtendedcleared so a subsequentUNEXPECTEDre-triggers the transition).StreamProcessor.handleErrorordering: classifier → regime switch →updateStatus(INTERRUPTED, …), unconditionally returning true so the eventsource keeps retrying.DataSourceStatusProviderJavadoc changes for accuracy vs. the current state machine.Dependencies (why CI is red)
Two unreleased upstream artifacts:
launchdarkly-java-sdk-internal ≥ 1.11.0— Must be released before this PR can build against Maven Central. Ships via feat(internal): add FailureClass enum and classifier helpers to HttpErrors (SDK-2789) #204 + its release-please chore.okhttp-eventsource ≥ 5.0.0— Must be released before this PR can build against Maven Central. Ships via launchdarkly/okhttp-eventsource#110.Once both are released, bump their versions in
lib/sdk/server/build.gradleand CI will go green. Locally, the branch builds againstmavenLocal()snapshots of both.Note
Overview
Aligns FDv1 streaming and polling with RETRY conformance: failures that used to shut the data source down permanently (e.g. 401/403 and other UNEXPECTED classifications) now stay in INITIALIZING or INTERRUPTED and keep retrying in the background.
Polling moves from fixed-rate scheduling to a
PollingStrategyloop that picks the next wait after each attempt—normal cadence at the configured interval, extended exponential backoff (5 min initial, 1 hr cap, jitter) after UNEXPECTED failures, reset after two consecutive successful polls.PollingProcessorno longer cancels polling or setsState.OFFon HTTP errors.Streaming wires okhttp-eventsource normal vs extended
RetryDelayStrategy(30 s vs 5 min→1 hr), activates extended backoff on UNEXPECTED viaactivateRetryDelayStrategy, and always reconnects instead of stopping on “unrecoverable” HTTP statuses. Healthy connectivity for 60 s returns normal backoff via the eventsource reset threshold.Public
DataSourceStatusProviderandLDClientdocs now describeOFFas shutdown-only (not invalid SDK key) and ongoing background retry. Contract-test service addsretry-conformance-fdv1-streamingandretry-conformance-fdv1-pollingcapabilities; unit/e2e tests assert extended retry instead of permanent failure.Reviewed by Cursor Bugbot for commit d3b56a6. Bugbot is set up for automated code reviews on this repo. Configure here.