Skip to content

feat(server): RETRY-spec conformance in FDv1 streaming and polling (SDK-2789) - #200

Open
tanderson-ld wants to merge 2 commits into
mainfrom
ta/SDK-2789/retry-conformance-work
Open

feat(server): RETRY-spec conformance in FDv1 streaming and polling (SDK-2789)#200
tanderson-ld wants to merge 2 commits into
mainfrom
ta/SDK-2789/retry-conformance-work

Conversation

@tanderson-ld

@tanderson-ld tanderson-ld commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements RETRY-spec conformance in the Java server SDK's FDv1 streaming and polling data sources (SDK-2789). This PR is scoped to the server SDK only; the classifier helpers it consumes ship separately in #204.

Behavioral change. HTTP responses that today cause an FDv1 data source to permanently stop (notably 401 / 403 / other 4xx) and TLS / certificate validation failures are no longer terminal. Streaming enters an extended-regime backoff (5 min initial → 1 hr max, doubling); polling continues at its configured cadence but engages the extended-regime wait after an UNEXPECTED failure. Either regime returns to normal after 60 s of continuous healthy operation (streaming) or two consecutive successful polls (polling).

Scope. FDv1 streaming and polling under lib/sdk/server/. FDv2, event delivery, and other network callers are unchanged.

What changed

  • PollingStrategy — New state-machine encapsulation with onFailure(class) / onSuccess() / nextWait(). State: n (formula input), initialDelay, maxDelay, priorPollWasSuccessful. Wait floor is max(pollInterval, T − J); two consecutive successes reset from extended → normal.
  • PollingProcessor — Rewired to a self-driven loop that consults strategy.nextWait() between attempts. The State.OFF permanent-stop path is removed; the state stays INITIALIZING or INTERRUPTED with a lastError.
  • StreamProcessor — Consumes okhttp-eventsource's new multi-strategy retry API from launchdarkly/okhttp-eventsource#110. On UNEXPECTED classification it calls activateRetryDelayStrategy on the underlying EventSource to switch into the extended-regime RetryDelayStrategy. The library's built-in healthy-op reset returns the SDK to normal-regime timing after 60 s of continuous connectivity.
  • DataSourceStatusProvider docsState.INITIALIZING, State.OFF, State.INTERRUPTED, and getStateSince OFF-case Javadocs updated to reflect the new semantics (no HTTP-error → OFF transition). Aligned with the Go server SDK's parallel doc adjustments.
  • LDClient constructor Javadoc — Wording tightened so a "wrong SDK key" scenario is described as ongoing retry in the background, not as an "unsuccessful initialization" that reads as terminal.
  • Contract-test service — Declares retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling capabilities.

The classifier this SDK depends on — FailureClass and HttpErrors.classify* helpers — ships in #204.

Testing

  • Unit tests — Full suite green. New coverage: PollingStrategyTest (strategy state machine), plus extended-regime timing-observation tests in StreamProcessorTest. Existing 401 / 403 tests were rewritten to assert extended-regime retry rather than permanent stop.
  • Contract tests via sdk-test-harness#404 — All 7 RETRY-conformance test cases pass end-to-end at production timing (5-minute extended-initial delay). Total wall clock ~12 min via parallel shards.

Test plan for reviewers

  • Verify PollingStrategy transition semantics: normal → extended fires exactly once per UNEXPECTED failure; two consecutive successful polls fully reset (n = 0, delay bounds back to normal, inExtended cleared so a subsequent UNEXPECTED re-triggers the transition).
  • Verify StreamProcessor.handleError ordering: classifier → regime switch → updateStatus(INTERRUPTED, …), unconditionally returning true so the eventsource keeps retrying.
  • Review the DataSourceStatusProvider Javadoc changes for accuracy vs. the current state machine.
  • Code comments deliberately describe current behavior only — no spec section refs, no historical framing ("previously", "no longer"), no cross-SDK references. Confirm you'd expect a reader to find that acceptable.

Dependencies (why CI is red)

Two unreleased upstream artifacts:

Once both are released, bump their versions in lib/sdk/server/build.gradle and CI will go green. Locally, the branch builds against mavenLocal() snapshots of both.


Note

Overview
Aligns FDv1 streaming and polling with RETRY conformance: failures that used to shut the data source down permanently (e.g. 401/403 and other UNEXPECTED classifications) now stay in INITIALIZING or INTERRUPTED and keep retrying in the background.

Polling moves from fixed-rate scheduling to a PollingStrategy loop that picks the next wait after each attempt—normal cadence at the configured interval, extended exponential backoff (5 min initial, 1 hr cap, jitter) after UNEXPECTED failures, reset after two consecutive successful polls. PollingProcessor no longer cancels polling or sets State.OFF on HTTP errors.

Streaming wires okhttp-eventsource normal vs extended RetryDelayStrategy (30 s vs 5 min→1 hr), activates extended backoff on UNEXPECTED via activateRetryDelayStrategy, and always reconnects instead of stopping on “unrecoverable” HTTP statuses. Healthy connectivity for 60 s returns normal backoff via the eventsource reset threshold.

Public DataSourceStatusProvider and LDClient docs now describe OFF as shutdown-only (not invalid SDK key) and ongoing background retry. Contract-test service adds retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling capabilities; unit/e2e tests assert extended retry instead of permanent failure.

Reviewed by Cursor Bugbot for commit d3b56a6. Bugbot is set up for automated code reviews on this repo. Configure here.

@tanderson-ld
tanderson-ld force-pushed the ta/SDK-2789/retry-conformance-work branch from f87cd37 to a754251 Compare August 21, 2026 20:32
@tanderson-ld
tanderson-ld marked this pull request as ready for review August 21, 2026 20:36
@tanderson-ld
tanderson-ld requested a review from a team as a code owner August 21, 2026 20:36
@tanderson-ld tanderson-ld changed the title feat(server): RETRY-spec conformance in FDv1 streaming and polling (SDK-2789) feat(server): RETRY-spec conformance in FDv1 streaming and polling Aug 21, 2026
…polling data sources (SDK-2789)

Guided by the server-sdk-guide.md in sdk-scratchpad; analogous to the
Go server SDK's reference implementation.

The behavioral change: HTTP responses that today cause a data source to
permanently stop (notably 401, 403, other 4xx) and TLS/certificate
validation failures are no longer terminal. Streaming enters an extended
backoff regime (5 min -> 1 hour, doubling); polling continues at its
configured cadence with extended-regime waits between failing polls.
Recovery from either regime uses a healthy-operation reset (60 s of
continuous connectivity for streaming; two consecutive successful polls
for polling).

Scope: FDv1 streaming and polling data sources under
`lib/sdk/server/src/main/java/com/launchdarkly/sdk/server/`. FDv2 is out
of scope for this epic and is deferred to a future one; nothing in
`datasourcev2/` or the DataSystem-related code paths is touched. The
classifier this depends on (`FailureClass` + `HttpErrors.classify*`)
lives in `launchdarkly-java-sdk-internal` and ships in its own PR.

Highlights:
- PollingStrategy: new state-machine encapsulation with
  onFailure(class) / onSuccess() / nextWait() methods. State: n
  (formula input), initialDelay, maxDelay, priorPollWasSuccessful.
  Wait floor: max(pollInterval, T - J). Two-consecutive-successes
  returns from extended to normal regime.
- PollingProcessor: rewired to a self-driven loop using
  strategy.nextWait(). Removed the State.OFF permanent-stop path
  entirely; state stays INITIALIZING/INTERRUPTED with a lastError.
- StreamProcessor: consumes okhttp-eventsource's new multi-strategy
  retry API (see launchdarkly/okhttp-eventsource#110). On UNEXPECTED
  classification, activates the extended-regime RetryDelayStrategy on
  the underlying EventSource; the library's built-in healthy-op reset
  returns to normal-regime timing after 60 s of continuous
  connectivity.
- Constructor plumbing: PollingProcessor and StreamProcessor take
  extendedInitialReconnectDelay, extendedStreamMaxRetryDelay,
  retryResetInterval, and extendedInitialDelay as constructor
  parameters; package-private defaults threaded through
  ComponentsImpl.
- DataSourceStatusProvider Javadocs: State.INITIALIZING, State.OFF,
  State.INTERRUPTED, and getStateSince OFF-case updated to reflect the
  new semantics (no HTTP-error -> OFF transition).
- LDClient constructor Javadoc: describes an SDK-key rejection as
  ongoing background retry rather than an "unsuccessful initialization"
  that reads as terminal.
- Contract test service: declares retry-conformance-fdv1-streaming
  and retry-conformance-fdv1-polling capabilities.

Tests:
- Unit tests: full test suite green. New coverage for the strategy
  state machine (PollingStrategyTest) and extended-regime timing
  observation in StreamProcessorTest. Existing 401/403 tests rewritten
  to assert extended-regime retry rather than permanent stop.
- Contract tests via sdk-test-harness PR #404 (RETRY-conformance
  tests): 7/7 parallel shards pass end-to-end at production timing
  (5-minute extended-initial-delay), ~12 min wall clock.

CI: intentionally red on this PR until
launchdarkly/okhttp-eventsource#110 releases okhttp-eventsource 5.0.0
and #204 releases launchdarkly-java-sdk-internal
1.11.0. The multi-strategy retry API this SDK relies on is only in
that eventsource PR's branch, and the classifier helpers are only in
that internal-artifact PR's branch. Once both are released, bump both
versions in lib/sdk/server/build.gradle.
@tanderson-ld
tanderson-ld force-pushed the ta/SDK-2789/retry-conformance-work branch from 9394a65 to 6d0c138 Compare August 24, 2026 13:26
@tanderson-ld tanderson-ld changed the title feat(server): RETRY-spec conformance in FDv1 streaming and polling feat(server): RETRY-spec conformance in FDv1 streaming and polling (SDK-2789) Aug 24, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6d0c138. Configure here.

@tanderson-ld

Copy link
Copy Markdown
Contributor Author

CI Failures will be resolved once okhttp-eventsource and internal release. But waiting on initial review in this PR to make sure all the APIs on okhttp-eventsource and internal aren't going to need changes.

Aligns with StreamProcessor.close() and FDv2DataSource.close(), both of
which call updateStatus(State.OFF, null) after closing their upstream
I/O. Also aligns with the PR's updated Javadoc for State.OFF, which now
describes it as the time the data source "stopped operation" (i.e.,
close), rather than the pre-PR meaning of "encountered an unrecoverable
error".
tanderson-ld added a commit that referenced this pull request Aug 25, 2026
…rrors (SDK-2789) (#204)

## Summary

Introduces a shared network-failure classifier in
`launchdarkly-java-sdk-internal` that categorizes failures as either
**NORMAL** (typically transient) or **UNEXPECTED** (indicative of a
longer-lived condition — e.g., invalid SDK key, TLS misconfiguration).
Downstream SDKs can use the classification to select between
normal-regime and extended-regime backoff.

Enables the server SDK's RETRY-spec conformance work in
[SDK-2789](https://launchdarkly.atlassian.net/browse/SDK-2789) (see PR
#200), which needs these helpers to select its
retry regime.

## What changed

- **`FailureClass` enum** — `NORMAL` and `UNEXPECTED`, with a
package-private cause-chain scan for TLS / certificate exceptions.
- **`HttpErrors.classifyHTTPFailure(int)`** — Returns `NORMAL` for 400 /
408 / 429, 5xx, and any other status the SDK treats as a failure;
returns `UNEXPECTED` for other 4xx (401 / 403 / etc.).
- **`HttpErrors.classifyTransportFailure(Throwable)`** — Returns
`UNEXPECTED` if TLS or certificate validation appears anywhere in the
exception chain; `NORMAL` otherwise.
- **`HttpErrors.classifyAndLogHTTPFailure` /
`classifyAndLogTransportFailure`** — Classify, log at the appropriate
level (Error for `UNEXPECTED`, Warn for `NORMAL`), and return the
classification.
- **`HttpErrors.isHttpErrorRecoverable` and
`checkIfErrorIsRecoverableAndLog` are now `@Deprecated`.** The boolean
"give up permanently" contract doesn't fit callers that keep retrying
regardless of classification; the Javadoc points migrations at the new
helpers. Existing callers keep working — deprecation is
source-compatible.

## Testing

- New `HttpErrorsClassificationTest` covers the full HTTP status matrix
(400/408/429/5xx = NORMAL, 401/403/404/418/451 = UNEXPECTED,
non-4xx/non-5xx = NORMAL) and the transport-exception matrix (ordinary
I/O = NORMAL, SSL/certificate = UNEXPECTED, TLS as a cause of a wrapper
exception = UNEXPECTED).
- Existing `lib/shared/internal` tests continue to pass; the deprecated
methods still exercise their original behavior.

## Test plan for reviewers

- [ ] Confirm the 4xx / 5xx split (400 / 408 / 429 = NORMAL, other 4xx =
UNEXPECTED) matches expectations.
- [ ] Confirm transport-level TLS / certificate failures at any depth in
the cause chain are surfaced as `UNEXPECTED`.
- [ ] Sanity-check the deprecation Javadocs — they name the migration
target concretely and give a one-line reason (existing boolean contract
vs. new tri-state usage).

## Downstream

Once this PR merges and release-please publishes
`launchdarkly-java-sdk-internal 1.11.0`, launchdarky/java-core#200 will
bump its `build.gradle` dep to 1.11.0 and consume the classifier from
the server SDK's data sources.

[SDK-2789]:
https://launchdarkly.atlassian.net/browse/SDK-2789?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Adds shared **internal** network failure classification so data
sources can pick **normal** vs **extended** backoff instead of treating
some errors as “stop retrying forever.”
> 
> Introduces a `FailureClass` enum (`NORMAL` vs `UNEXPECTED`) and
`HttpErrors` helpers to classify HTTP status codes (e.g. 400/408/429 and
5xx → `NORMAL`; most other 4xx including 401/403 → `UNEXPECTED`) and
transport exceptions (TLS/certificate causes anywhere in the chain →
`UNEXPECTED`). New `classifyAndLogHttpFailure` /
`classifyAndLogTransportFailure` classify, log at **Error** vs **Warn**,
and return the class for the caller.
> 
> `isHttpErrorRecoverable` and `checkIfErrorIsRecoverableAndLog` are
**deprecated** (behavior unchanged); Javadoc points callers at the new
APIs. Unit tests cover the HTTP and transport classification matrices.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
f67fd5e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

@jsonbailey jsonbailey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this alongside #204 and okhttp-eventsource#110 to check the API assumptions against the source rather than take the description's word for them. Broadly this holds up well — I specifically tried to break the two-consecutive-successes reset, the activateRetryDelayStrategy timing, and the polling loop lifecycle, and all three survived. The #110 integration is correct on every point I checked, including the one I most expected to be wrong: activation is a pointer swap that does not reset progression, so repeated 401s really do double 5→10→20→40→60 min. Using an explicit inExtended flag instead of comparing initialDelay == normalInterval avoids a real bug, and nInExtendedDoublesEvenWhenPollIntervalEqualsExtendedInitial is a good regression test for exactly that case.

I have a longer list of medium/low observations I'll hold for now. Two things I think block merge:

1. Won't compile against #204 HEAD. classifyAndLogHTTPFailure was renamed to classifyAndLogHttpFailure in #204's f67fd5e ("address review feedback on classifier helpers"). This branch predates that rename, and the local mavenLocal() snapshot is what hides it. Two call sites, inline below. Calling it out separately because it is not the unreleased-dependency-pin issue the description covers — it survives releasing both artifacts.

2. Worth one test before deciding anything else: the TLS classification may be far broader than intended. FailureClass.hasTlsOrCertificateCause (in #204) returns UNEXPECTED for any SSLException or GeneralSecurityException anywhere in the cause chain, and StreamIOException sets the underlying IOException as its cause, so the chain is reachable from StreamProcessor.handleError. Since all LD traffic is HTTPS, every transport error against the real endpoint arrives through the TLS layer — so SSLHandshakeException: Remote host terminated the handshake from a load balancer or proxy dropping mid-handshake reads as UNEXPECTED and pushes streaming into 5 min–1 hr backoff for a transient blip. The compounding part: a flapping stream never accumulates the 60 s of continuous connectivity the healthy-op reset needs, so it cannot climb back out.

I could not settle whether JSSE actually throws SSLException for these cases (no JVM available where I was reviewing), and about 20 lines answers it: open an SSLServerSocket, complete the handshake, then setSoLinger(true, 0) + close(), and read from the client — observe the exception type. Worth running before deciding: if the classification is as broad as it reads, this is the largest production-impact item in the change; if it isn't, the test costs nothing. Narrowing to SSLHandshakeException / SSLPeerUnverifiedException / CertificateException, or excluding causes that are themselves SocketException / SocketTimeoutException, would settle it either way.

On your test plan: the PollingStrategy transition semantics and the handleError ordering both check out — I traced the reset across intervening-failure and mixed NORMAL/UNEXPECTED orderings and found no off-by-one, and inExtended is cleared so a later UNEXPECTED re-fires the transition. No objection to the comment style.

task.cancel(true);
task = null;
}
FailureClass failureClass = HttpErrors.classifyAndLogHTTPFailure(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

classifyAndLogHTTPFailureclassifyAndLogHttpFailure.

#204 renamed this in f67fd5e; the only such method at #204 HEAD is classifyAndLogHttpFailure (HttpErrors.java:156), and #204's own deprecation Javadoc at HttpErrors.java:76 uses the lowercase form too. git log -S in #204 shows 52c05c2 added it all-caps and f67fd5e renamed it, so this branch was written against the earlier revision.

This will fail compileJava once #204 merges and internal 1.11.0 publishes — separate from the unreleased-pin issue, since it survives releasing both artifacts.

if (e instanceof StreamHttpErrorException) {
int status = ((StreamHttpErrorException)e).getCode();
ErrorInfo errorInfo = ErrorInfo.fromHttpError(status);
failureClass = HttpErrors.classifyAndLogHTTPFailure(logger, status, ERROR_CONTEXT_MESSAGE, WILL_RETRY_MESSAGE);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second call site for the classifyAndLogHTTPFailureclassifyAndLogHttpFailure rename — see the note on PollingProcessor.java:126.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants