Skip to content

fix(hub): surface fanout slow-listener disconnects - #164

Open
CoderMungan wants to merge 1 commit into
ActiveMemory:mainfrom
CoderMungan:fix/hub-fanout-drop-observability
Open

fix(hub): surface fanout slow-listener disconnects#164
CoderMungan wants to merge 1 commit into
ActiveMemory:mainfrom
CoderMungan:fix/hub-fanout-drop-observability

Conversation

@CoderMungan

@CoderMungan CoderMungan commented Aug 27, 2026

Copy link
Copy Markdown
Member

Closes #94.

Covers items #1 (make the drop-counter observable) and #2 (regression
test)
from the issue. Item #3 (configurable fanOutBuffer) is deliberately
left out — as the issue puts it, tuning the constant before the counter is
observable would be tuning blind.

#1f.dropped is no longer dead

The counter is now readable on two surfaces, both options the issue listed:

Warning at the moment of disconnect. broadcast() calls logWarn.Warn
when it cuts a slow listener loose, using a new HubFanOutSlowListener
format constant in internal/config/warn (same pattern as the existing
HubReplicate* family — no literal in the hub package). The message carries
the cumulative count so log aggregators can rate it:

ctx: hub fanout: disconnected slow listener (buffer full); cumulative disconnects: 3

Cumulative count on the Status RPC. StatusResponse gains
DroppedListeners uint64 (json:"dropped_listeners"), populated by
hubStatus from a new droppedCount() accessor. ctx hub status prints

Dropped listeners: 3 (slow subscribers disconnected)

only when the count is non-zero, so a healthy hub's output is byte-for-byte
what it was before and no existing test or doc example changes.

On sync/atomic

Per the third acceptance item, broadcast increments with
atomic.AddUint64 (its return value is what the warning reports) and
droppedCount reads with atomic.LoadUint64, so the Status RPC handler
reads the counter without contending with an in-flight broadcast.

Every release target in hack/build-all.sh is 64-bit (darwin, linux and
windows on amd64/arm64), so the struct-field alignment caveat that applies to
the raw atomic.*Uint64 helpers on 32-bit platforms doesn't bite here.

#2 — regression test

TestFanOut_DisconnectsSlowListener overflows the buffer against a listener
that never drains, then asserts all three halves of the contract: the channel
is closed, the subscriber is gone from f.subs, and the counter incremented.
TestFanOut_DroppedCountStartsAtZero pins the healthy path so the counter
can't start drifting upward.

TestFanOut_DroppedCountRaceWithBroadcast covers the concurrency the
acceptance item is really about: four goroutines call droppedCount() — the
Status RPC handler's read path — while broadcast disconnects listeners.

I verified both by mutation rather than trusting them:

  • deleting the delete(f.subs, ch); close(ch) block fails
    TestFanOut_DisconnectsSlowListener (count = 1, want 0 after disconnect
    and disconnected channel never closed) while the original three tests
    still pass — the exact silent regression the issue describes;
  • reverting the counter to a plain f.dropped++ / return f.dropped makes
    go test -race report WARNING: DATA RACE and fail
    TestFanOut_DroppedCountRaceWithBroadcast.

The tests redirect logWarn.SetSink(io.Discard) so the new warning doesn't
pollute test output.

The rendered line is pinned too

desc.Text returns "" for an unknown key, so a renamed text key would blank
the new ctx hub status line silently — the same shape of bug this issue is
about. internal/write/hub gains TestClusterStatus_DroppedListeners
(asserts the rendered count) and TestClusterStatus_NoDroppedListeners
(asserts the omission at zero, and that the existing stats line survives).
Renaming DescKeyWriteHubDroppedListeners fails the first one.

Docs

  • docs/cli/hub.md — notes the conditional Dropped listeners: line.
  • docs/operations/hub-failure-modes.md — new Slow Listener Disconnected
    entry under Network, explaining that the disconnect is the loss-prevention
    mechanism (the client reconnects with its last-seen sequence and the hub
    replays), what the warning and counter mean, and when a climbing count is
    worth acting on.
  • internal/hub/doc.go — the Concurrency section said "slow subscribers are
    dropped", which read as entry loss; corrected to describe the disconnect
    and point at the counter.

Verification

Ran the CI commands verbatim:

Check Result
CGO_ENABLED=0 go build ./... clean
CGO_ENABLED=0 go test ./... 181 packages, 0 failures
go test -race ./internal/hub/ clean
CGO_ENABLED=0 go vet ./... clean
golangci-lint run 0 issues
hack/lint-docstrings.sh clean

Nothing here touches the shell, PowerShell, OpenCode-plugin, or VS Code
extension surfaces.

@CoderMungan
CoderMungan force-pushed the fix/hub-fanout-drop-observability branch from 1a16152 to a3543b0 Compare August 27, 2026 08:57
The fanout broadcaster disconnects a listener whose buffer is full
rather than dropping its entries, but the only record of it was
f.dropped -- a counter incremented at fanout.go and read nowhere
else in the codebase. Operators had no way to know listeners were
being kicked.

Make the counter observable on two surfaces:

- broadcast() warns on stderr at the moment of disconnect, through
  a new HubFanOutSlowListener format in internal/config/warn, so
  log aggregators can rate the event as it happens.
- StatusResponse gains DroppedListeners, populated from a new
  droppedCount() accessor. ctx hub status prints "Dropped
  listeners: N" only when N > 0, so a healthy hub's output is
  unchanged.

The counter moves to sync/atomic as the acceptance list asks:
broadcast increments with atomic.AddUint64 (its return value is
what the warning reports) and droppedCount reads with
atomic.LoadUint64, so the Status RPC handler never contends with an
in-flight broadcast. Every release target is 64-bit, so the
struct-field alignment caveat for the raw atomic.*Uint64 helpers
does not apply here.

Pin the contract with two tests. TestFanOut_DisconnectsSlowListener
asserts the channel closes, the subscriber is removed from f.subs,
and the counter increments; verified by mutation -- removing the
delete/close block fails it while the original three pass.
TestFanOut_DroppedCountRaceWithBroadcast reads the counter from
four goroutines while broadcast disconnects listeners, so -race
fails if the counter stops being atomic; also verified by mutation.

The rendered status line gets its own pin. desc.Text returns "" for
an unknown key, so a renamed text key would blank the line silently;
TestClusterStatus_DroppedListeners asserts the rendered count and
TestClusterStatus_NoDroppedListeners asserts the omission at zero.
Verified by mutation: renaming the key fails the first.

Leaves fanOutBuffer alone -- tuning it before the counter is
observable would be tuning blind.

Closes ActiveMemory#94

Signed-off-by: CoderMungan <codermungan@gmail.com>
@CoderMungan
CoderMungan force-pushed the fix/hub-fanout-drop-observability branch from a3543b0 to 472b684 Compare August 27, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hub fanout: drop-counter is dead, disconnect path untested, buffer size unconfigurable

1 participant