state: close the config feed when the listener loses its connection - #127
Merged
Merged
Conversation
PgListener::recv swallows a lost connection: try_recv returns Ok(None) after an eager reconnect and recv loops past it, so the subscribe task never ended and the server never reached its re-subscribe path. A version published during the gap stayed unapplied until the next publish. Drive the loop with try_recv so the channel closes on the first lost connection, as the subscribe doc states, and the caller's catch-up reload runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
configstore::subscribedrove its forwarding task withPgListener::recv(). In sqlx-postgres 0.9,recv()loops overtry_recv(), which returnsOk(None)when the connection is lost (after reconnecting eagerly by default). The lost connection was swallowed: the task never ended, the version channel never closed, and the server's feed loop never reached its re-subscribe path with the catch-up reload. A config version published while the listener was disconnected (Postgres restart, terminated backend, network drop) was stored but not applied on that instance until the next publish.Fix
Drive the loop with
try_recv()and stop onOk(None)as well as on errors. The channel now closes on the first lost connection, matching thesubscribedoc; the server logs the drop, re-subscribes after its retry delay (LISTEN first), then reloads the latest version to catch up.A new
GW_TEST_PG_URL-gated regression test,subscribe_closes_when_the_listener_connection_drops, subscribes under its ownapplication_name, terminates exactly that listener backend withpg_terminate_backend, and requires the version channel to close within 10s.Evidence
Notes
Env-gated suites that did not run against a backend (they early-return when their variable is unset, so the green runs above do not cover them):
GW_TEST_PG_URL: the Postgres tests incrates/state/src/store.rsandcrates/state/src/keystore.rs(filtered out of the PG-backed run),crates/handler/src/lib.rs,crates/server/tests/e2e.rs.GW_TEST_REDIS_URL: the Redis tests incrates/state/src/lib.rs,crates/state/src/governance.rs,crates/state/src/avail.rs,crates/state/src/health.rs.No hot-path impact: the change is confined to the config feed task.