fix(hub): surface fanout slow-listener disconnects - #164
Open
CoderMungan wants to merge 1 commit into
Open
Conversation
CoderMungan
force-pushed
the
fix/hub-fanout-drop-observability
branch
from
August 27, 2026 08:57
1a16152 to
a3543b0
Compare
The fanout broadcaster disconnects a listener whose buffer is full rather than dropping its entries, but the only record of it was f.dropped -- a counter incremented at fanout.go and read nowhere else in the codebase. Operators had no way to know listeners were being kicked. Make the counter observable on two surfaces: - broadcast() warns on stderr at the moment of disconnect, through a new HubFanOutSlowListener format in internal/config/warn, so log aggregators can rate the event as it happens. - StatusResponse gains DroppedListeners, populated from a new droppedCount() accessor. ctx hub status prints "Dropped listeners: N" only when N > 0, so a healthy hub's output is unchanged. The counter moves to sync/atomic as the acceptance list asks: broadcast increments with atomic.AddUint64 (its return value is what the warning reports) and droppedCount reads with atomic.LoadUint64, so the Status RPC handler never contends with an in-flight broadcast. Every release target is 64-bit, so the struct-field alignment caveat for the raw atomic.*Uint64 helpers does not apply here. Pin the contract with two tests. TestFanOut_DisconnectsSlowListener asserts the channel closes, the subscriber is removed from f.subs, and the counter increments; verified by mutation -- removing the delete/close block fails it while the original three pass. TestFanOut_DroppedCountRaceWithBroadcast reads the counter from four goroutines while broadcast disconnects listeners, so -race fails if the counter stops being atomic; also verified by mutation. The rendered status line gets its own pin. desc.Text returns "" for an unknown key, so a renamed text key would blank the line silently; TestClusterStatus_DroppedListeners asserts the rendered count and TestClusterStatus_NoDroppedListeners asserts the omission at zero. Verified by mutation: renaming the key fails the first. Leaves fanOutBuffer alone -- tuning it before the counter is observable would be tuning blind. Closes ActiveMemory#94 Signed-off-by: CoderMungan <codermungan@gmail.com>
CoderMungan
force-pushed
the
fix/hub-fanout-drop-observability
branch
from
August 27, 2026 10:16
a3543b0 to
472b684
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #94.
Covers items #1 (make the drop-counter observable) and #2 (regression
test) from the issue. Item #3 (configurable
fanOutBuffer) is deliberatelyleft out — as the issue puts it, tuning the constant before the counter is
observable would be tuning blind.
#1 —
f.droppedis no longer deadThe counter is now readable on two surfaces, both options the issue listed:
Warning at the moment of disconnect.
broadcast()callslogWarn.Warnwhen it cuts a slow listener loose, using a new
HubFanOutSlowListenerformat constant in
internal/config/warn(same pattern as the existingHubReplicate*family — no literal in the hub package). The message carriesthe cumulative count so log aggregators can rate it:
Cumulative count on the Status RPC.
StatusResponsegainsDroppedListeners uint64(json:"dropped_listeners"), populated byhubStatusfrom a newdroppedCount()accessor.ctx hub statusprintsonly when the count is non-zero, so a healthy hub's output is byte-for-byte
what it was before and no existing test or doc example changes.
On
sync/atomicPer the third acceptance item,
broadcastincrements withatomic.AddUint64(its return value is what the warning reports) anddroppedCountreads withatomic.LoadUint64, so the Status RPC handlerreads the counter without contending with an in-flight broadcast.
Every release target in
hack/build-all.shis 64-bit (darwin, linux andwindows on amd64/arm64), so the struct-field alignment caveat that applies to
the raw
atomic.*Uint64helpers on 32-bit platforms doesn't bite here.#2 — regression test
TestFanOut_DisconnectsSlowListeneroverflows the buffer against a listenerthat never drains, then asserts all three halves of the contract: the channel
is closed, the subscriber is gone from
f.subs, and the counter incremented.TestFanOut_DroppedCountStartsAtZeropins the healthy path so the countercan't start drifting upward.
TestFanOut_DroppedCountRaceWithBroadcastcovers the concurrency theacceptance item is really about: four goroutines call
droppedCount()— theStatus RPC handler's read path — while
broadcastdisconnects listeners.I verified both by mutation rather than trusting them:
delete(f.subs, ch); close(ch)block failsTestFanOut_DisconnectsSlowListener(count = 1, want 0 after disconnectand
disconnected channel never closed) while the original three testsstill pass — the exact silent regression the issue describes;
f.dropped++/return f.droppedmakesgo test -racereportWARNING: DATA RACEand failTestFanOut_DroppedCountRaceWithBroadcast.The tests redirect
logWarn.SetSink(io.Discard)so the new warning doesn'tpollute test output.
The rendered line is pinned too
desc.Textreturns""for an unknown key, so a renamed text key would blankthe new
ctx hub statusline silently — the same shape of bug this issue isabout.
internal/write/hubgainsTestClusterStatus_DroppedListeners(asserts the rendered count) and
TestClusterStatus_NoDroppedListeners(asserts the omission at zero, and that the existing stats line survives).
Renaming
DescKeyWriteHubDroppedListenersfails the first one.Docs
docs/cli/hub.md— notes the conditionalDropped listeners:line.docs/operations/hub-failure-modes.md— new Slow Listener Disconnectedentry under Network, explaining that the disconnect is the loss-prevention
mechanism (the client reconnects with its last-seen sequence and the hub
replays), what the warning and counter mean, and when a climbing count is
worth acting on.
internal/hub/doc.go— the Concurrency section said "slow subscribers aredropped", which read as entry loss; corrected to describe the disconnect
and point at the counter.
Verification
Ran the CI commands verbatim:
CGO_ENABLED=0 go build ./...CGO_ENABLED=0 go test ./...go test -race ./internal/hub/CGO_ENABLED=0 go vet ./...golangci-lint runhack/lint-docstrings.shNothing here touches the shell, PowerShell, OpenCode-plugin, or VS Code
extension surfaces.