Skip to content

fix(realtime): a page held open through a bus outage is live again without a reload; an outage is logged as one fact; x dev no longer goes down for the bus - #759

Merged
sebyx07 merged 2 commits into
mainfrom
fix/channel-recovers-after-bus-outage
Oct 11, 2026
Merged

sebyx07 merged 2 commits into
mainfrom
fix/channel-recovers-after-bus-outage

Conversation

@sebyx07

@sebyx07 sebyx07 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Found while proving 27.7.0 on a consumer app with a real NATS stopped and started under a running sync node and open pages.

  • A channel membership recovers. X_TRANSPORT_UNAVAILABLE is the one transient refusal: a live membership goes to joining and the node's next answer makes it live again; every other refusal stays terminal. Before, the heartbeat's error ack during an outage marked the membership failed for good, so the page read fallback until reloaded although events reached it.
  • A KV request with no live connection is refused at once (the library buffered it for 5 s while reconnecting).
  • A silent NATS is detected in 30 s, measured against a relay that stops forwarding.
  • An outage is logged as one fact: failures 1, 2, 4, 8… plus one recovered line, WARN on a publishing role and ERROR on a role that cannot do its job (sync.bus_unavailable replaces the per-sweep presence.sweep failed); cache.invalidate.partial is thinned per failing tier, the edge purge included.
  • x dev waits 3 s for the bus, logs the coded refusal once and boots; the sync node starts on the transport's next recovery. Containers keep the hard 15 s refusal.

Tests: client and sync-node outage tests, a black-hole live test against a real nats-server, thinning counts and levels; full cli, realtime, cache, core, render suites and scripts green; both reference apps' unit and budgets steps green.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Realtime channels now remain reconnecting rather than entering a terminal error when the messaging service is temporarily unavailable, and can recover without a page reload.
    • Requests made while disconnected are rejected immediately. Silent connection failures are detected, and cache invalidation outage logs are summarized at milestones.
    • In development, services can start without the messaging service; sync remains unready until the connection returns.
  • Documentation
    • Updated outage guidance explains recovery behavior, readiness, logging, and transient channel states.

sebyx07 and others added 2 commits October 10, 2026 20:19
… without a reload; an outage is logged as one fact, thinned, at the role's level; x dev serves without the bus; a silent NATS is detected in 30 s, measured

A page open through a NATS outage kept its channel membership `failed` until a
reload: the sync node answers the page's 10 s beat with an `ack` carrying
X_TRANSPORT_UNAVAILABLE (the presence write under the beat cannot land), the
client took every refusal as a decision, and nothing cleared it.

- client-channels.ts: X_TRANSPORT_UNAVAILABLE is the one transient refusal.
  The membership reads `joining`, the beat keeps re-asking, the node's first
  answer makes it `live`. Every other refusal stays terminal.
- nats-lib-client.ts: a KV request made with no live connection is refused at
  once, as a publish already was (it held a socket's frame lane for 5 s).
- core outage-log.ts: failures 1, 2, 4, 8, … and one closing line. Used by the
  sync node (`sync.bus_unavailable`, error, one count per node, the monitor
  thinned with it), the transport (`nats transport error`: warn on a publisher,
  error on sync/replicator, `nats transport recovered`), the cache
  (`cache.invalidate.partial` per failing tier) and the boot subscribe.
- runtime-bus.ts / role-sync-bus-wait.ts: `x dev` waits 3 s for the bus, logs
  the coded refusal once and boots; the sync node starts when the bus answers.
  `transport` is `degraded` there. A container keeps the 15 s refusal.
- nats-black-hole.live.test.ts: a relay that stops forwarding, against a real
  server — both a publisher and a sync node give up the connection 30.0 s in;
  the seated page reads `joining` throughout and is live on the same socket.

serve-graph pins re-measured: migrate 641, serving 945, web 1072 (five new
modules, each named on its row). Page boot +118 B; both reference apps stay
inside their route budgets (/feed 162,270 of 162,304 B).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ve-graph pins re-measured (serving 950, web 1078)
@developerz-ai

developerz-ai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Review did not complete

developerz.ai started reviewing this pull request and stopped before finishing: the pull request was merged or closed while it ran.

This is a failure of the review run, not a verdict on the changes — nothing here says the diff is good or bad. The run is recorded on this task's audit trail.

⏱ 5m 09s wall clock · MiniMax-M3 via minimax · 1 model call(s) · 18,153 output token(s) · slowest call 58s

🤖 developerz.ai — automated review, running on your box. This run did not complete.

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: developerz-ai/ultimate/.coderabbit.yml
  • Review profile: ASSERTIVE
  • Plan: Essentials
  • Run ID: c6dd9ab2-b674-44c0-979e-e79bffe17905

📥 Commits

Reviewing files that changed from the base of the PR and between 38150f5 and c8deea0.


📒 Files selected for processing (40)
  • CHANGELOG.md
  • docs/ops/01-kubernetes.md
  • packages/cache/src/invalidate.ts
  • packages/cache/src/partial-log.test.ts
  • packages/cache/src/partial-log.ts
  • packages/cli/src/dev-boot.ts
  • packages/cli/src/role-sync-bus-wait.test.ts
  • packages/cli/src/role-sync-bus-wait.ts
  • packages/cli/src/role-sync.ts
  • packages/cli/src/runtime-bus.test.ts
  • packages/cli/src/runtime-bus.ts
  • packages/cli/src/runtime-cache-bus-loss.test.ts
  • packages/cli/src/runtime-cache.ts
  • packages/cli/src/runtime-services.ts
  • packages/cli/src/serve-graph.test.ts
  • packages/core/src/index.ts
  • packages/core/src/outage-log.test.ts
  • packages/core/src/outage-log.ts
  • packages/realtime/README.md
  • packages/realtime/src/client-bus-outage.test.ts
  • packages/realtime/src/client-channels.ts
  • packages/realtime/src/detach.ts
  • packages/realtime/src/nats-black-hole.live.test.ts
  • packages/realtime/src/nats-lib-client.ts
  • packages/realtime/src/nats-relay-fixture.ts
  • packages/realtime/src/nats-reports.ts
  • packages/realtime/src/nats-transport-background.test.ts
  • packages/realtime/src/nats-transport.live.test.ts
  • packages/realtime/src/nats-transport.ts
  • packages/realtime/src/presence.ts
  • packages/realtime/src/socket-engine.test.ts
  • packages/realtime/src/sync-bus-handlers.ts
  • packages/realtime/src/sync-bus-outage.test.ts
  • packages/realtime/src/sync-bus-outage.ts
  • packages/realtime/src/sync-frames.ts
  • packages/realtime/src/sync-node.ts
  • packages/realtime/src/sync-outage-rig-fixture.ts
  • packages/realtime/src/transport-env.ts
  • wiki/Realtime.md
  • wiki/Troubleshooting.md

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.



📝 Walkthrough

Walkthrough

The changes add milestone-based outage logging across core, cache, and realtime code. They also change how realtime channels handle NATS outages and allow selected x dev roles to start while NATS is unavailable.

Changes

Bus outage handling

Layer / File(s) Summary
Outage milestones and cache invalidation logs
packages/core/src/outage-log.ts, packages/core/src/index.ts, packages/core/src/outage-log.test.ts, packages/cache/src/partial-log.ts, packages/cache/src/invalidate.ts, packages/cache/src/partial-log.test.ts, packages/cli/src/runtime-cache.ts, packages/cli/src/runtime-cache-bus-loss.test.ts, CHANGELOG.md, wiki/Realtime.md
Core outage logs report at failure milestones and on recovery. Cache invalidation tracks successful tiers and logs partial failures at milestones; cache subscription logs also report recovery.
NATS outage reporting and detection
packages/realtime/src/nats-reports.ts, packages/realtime/src/nats-transport.ts, packages/realtime/src/nats-lib-client.ts, packages/realtime/src/nats-relay-fixture.ts, packages/realtime/src/nats-black-hole.live.test.ts, packages/realtime/src/nats-transport.*.test.ts, packages/realtime/src/transport-env.ts, docs/ops/01-kubernetes.md, wiki/Realtime.md, CHANGELOG.md
NATS reporting uses configured outage levels and reports recovery. Disconnected requests fail immediately. Live tests exercise silent connections, outage detection, and reconnection.
Sync outage and channel recovery
packages/realtime/src/client-channels.ts, packages/realtime/src/sync-bus-outage.ts, packages/realtime/src/sync-bus-handlers.ts, packages/realtime/src/sync-frames.ts, packages/realtime/src/sync-node.ts, packages/realtime/src/detach.ts, packages/realtime/src/presence.ts, packages/realtime/src/*outage*.test.ts, packages/realtime/src/socket-engine.test.ts, packages/realtime/src/sync-outage-rig-fixture.ts, packages/realtime/README.md, docs/ops/01-kubernetes.md, wiki/Realtime.md, wiki/Troubleshooting.md, CHANGELOG.md
A transport-unavailable refusal leaves channel membership joining without an error. Sync outage tracking logs milestones and recovery; later answers can restore channel membership. Other coded refusals remain terminal.
Development startup without an initial bus
packages/cli/src/runtime-bus.ts, packages/cli/src/runtime-services.ts, packages/cli/src/dev-boot.ts, packages/cli/src/role-sync-bus-wait.ts, packages/cli/src/role-sync.ts, packages/cli/src/*bus*.test.ts, packages/cli/src/serve-graph.test.ts, CHANGELOG.md, wiki/Realtime.md
x dev waits three seconds for NATS for bus-using roles, then continues after X_TRANSPORT_UNAVAILABLE. Sync stays unready until it starts; container startup remains non-tolerant.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant DevBoot
  participant StartServices
  participant SelectBus
  participant NatsTransport
  participant SyncNode
  DevBoot->>StartServices: start with dev enabled
  StartServices->>SelectBus: select bus for role
  SelectBus->>NatsTransport: wait up to three seconds for connection
  NatsTransport-->>SelectBus: return X_TRANSPORT_UNAVAILABLE
  SelectBus-->>StartServices: continue with tolerant bus
  StartServices->>SyncNode: prepare sync role
  SyncNode-->>StartServices: remain unready while bus is absent
  NatsTransport-->>SyncNode: reconnect and retry startup
  SyncNode-->>StartServices: start successfully
Loading

Merge Risk: ⚪ Minimal · up to c8dee

The change is mergeable after normal checks; no actionable issue is established by the supplied evidence.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 62.07% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 35 files. (5 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title accurately summarizes the main changes: realtime recovery after a bus outage, consolidated outage logging, and tolerant x dev startup. It is longer than preferred but remains specific and re…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 62.07% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 35 files. (5 skipped: 5 unsupported.)


  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR


🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@sebyx07
sebyx07 merged commit e57c7eb into main Oct 11, 2026
17 checks passed
@sebyx07
sebyx07 deleted the fix/channel-recovers-after-bus-outage branch October 11, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant