Skip to content

state: close the config feed when the listener loses its connection - #127

Merged
CMGS merged 2 commits into
mainfrom
fix/config-feed-reconnect
Sep 23, 2026
Merged

CMGS merged 2 commits into
mainfrom
fix/config-feed-reconnect

Conversation

@CMGS

@CMGS CMGS commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

configstore::subscribe drove its forwarding task with PgListener::recv(). In sqlx-postgres 0.9, recv() loops over try_recv(), which returns Ok(None) when the connection is lost (after reconnecting eagerly by default). The lost connection was swallowed: the task never ended, the version channel never closed, and the server's feed loop never reached its re-subscribe path with the catch-up reload. A config version published while the listener was disconnected (Postgres restart, terminated backend, network drop) was stored but not applied on that instance until the next publish.

Fix

Drive the loop with try_recv() and stop on Ok(None) as well as on errors. The channel now closes on the first lost connection, matching the subscribe doc; the server logs the drop, re-subscribes after its retry delay (LISTEN first), then reloads the latest version to catch up.

A new GW_TEST_PG_URL-gated regression test, subscribe_closes_when_the_listener_connection_drops, subscribes under its own application_name, terminates exactly that listener backend with pg_terminate_backend, and requires the version channel to close within 10s.

Evidence

$ cargo fmt --check
(no output, rc=0)

$ cargo clippy --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.25s

$ cargo test    # workspace, before the regression test was added
32 "test result: ok" lines, 0 FAILED/panicked lines, 684 passed, 4 ignored, rc=0

$ cargo test -p gw-state    # GW_TEST_PG_URL unset, with the regression test
test configstore::tests::publish_notifies_and_load_returns_latest ... ok
test configstore::tests::subscribe_closes_when_the_listener_connection_drops ... ok
test result: ok. 95 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.36s

$ GW_TEST_PG_URL=postgres://... cargo test -p gw-state configstore    # PostgreSQL 16
test configstore::tests::subscribe_closes_when_the_listener_connection_drops ... ok
test configstore::tests::publish_notifies_and_load_returns_latest ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 93 filtered out; finished in 0.25s

# same PG-backed run with the old recv() loop restored
test configstore::tests::publish_notifies_and_load_returns_latest ... ok
test configstore::tests::subscribe_closes_when_the_listener_connection_drops ... FAILED
feed must close on a dropped connection
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 93 filtered out; finished in 10.07s

# earlier throwaway probe (same drop, standalone), main vs this branch
main:   test feed_closes_when_listener_backend_is_terminated ... FAILED
        feed must close, got Err(Elapsed(()))
        test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.08s
branch: test feed_closes_when_listener_backend_is_terminated ... ok
        test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.63s

Notes

Env-gated suites that did not run against a backend (they early-return when their variable is unset, so the green runs above do not cover them):

  • GW_TEST_PG_URL: the Postgres tests in crates/state/src/store.rs and crates/state/src/keystore.rs (filtered out of the PG-backed run), crates/handler/src/lib.rs, crates/server/tests/e2e.rs.
  • GW_TEST_REDIS_URL: the Redis tests in crates/state/src/lib.rs, crates/state/src/governance.rs, crates/state/src/avail.rs, crates/state/src/health.rs.

No hot-path impact: the change is confined to the config feed task.

PgListener::recv swallows a lost connection: try_recv returns Ok(None)
after an eager reconnect and recv loops past it, so the subscribe task
never ended and the server never reached its re-subscribe path. A version
published during the gap stayed unapplied until the next publish. Drive
the loop with try_recv so the channel closes on the first lost
connection, as the subscribe doc states, and the caller's catch-up
reload runs.
@CMGS
CMGS merged commit 3892799 into main Sep 23, 2026
2 checks passed
@CMGS
CMGS deleted the fix/config-feed-reconnect branch September 23, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant