Skip to content

fix(reverse-attach): answer runtime pings during long tool calls (liveness) - #69

Merged
chaodu-agent merged 3 commits into
mainfrom
fix/reverse-attach-liveness
Nov 5, 2026
Merged

chaodu-agent merged 3 commits into
mainfrom
fix/reverse-attach-liveness

Conversation

@chaodu-agent

@chaodu-agent chaodu-agent commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #68. Runtime side: openabdev/openab-pty#86.

Problem

The openab-pty runtime pings the Mac every 20 s and closes the tools attach after more than three intervals with no inbound frame. ReverseAttachClient.dialOnce handled frames one at a time (receive → answer → send → receive). While a tool ran, no receive() was outstanding, and URLSessionWebSocketTask answered no pings. Any tool call longer than about 60–80 s evicted the Mac, and the agent saw attached=false, grant_pending=true.

Changes

  1. Root cause. ws.receive() is now always outstanding. Each inbound frame is answered in its own task, and replies go out through one writer task per socket, so sends never interleave. Calls also run concurrently now. When a socket ends for any reason (a stop code, a retryable close, a send failure, or a revoke), the calls still running on it are cancelled, and cancel() also reaches calls directly during a redial backoff. exec (and osascript, which runs through it) kills its process group when its task is cancelled, and a cancelled call no longer waits for a child backgrounded with cmd & that still holds its pipes (that child is not killed, because the leader's pid may already be recycled). Long work started with exec_start is not tied to a socket and keeps running.
  2. Keepalive. While attached, the client pings the peer every 15 s (Config.keepaliveInterval). The runtime counts any inbound frame as alive.
  3. Close code 4011. openab-pty#86 closes a silent Mac with 4011 (liveness) instead of a plain 1000. disposition(closeCode: 4011) returns .redial for both kinds. 4011 was picked because 4001–4010 are taken.
  4. exec limit on openab-pty attaches. An exec with timeout_secs over 100 s is refused with a tool error pointing to exec_start + exec_poll, because the runtime fails any call still running after 120 s (REQUEST_TIMEOUT). Switchboard attaches and the local HTTP endpoint keep the 600 s cap. osascript gets the same 100 s limit on openab-pty attaches, and both tools' timeout_secs descriptions mention it.
  5. Instructions. Every grant profile, owner included, now gets one line saying to keep calls under ~60 s, to use exec_start + exec_poll for long work, and never to sleep inside exec. The desktop and owner profiles also get a line saying calls run concurrently, so mouse, key and osascript steps should be issued one at a time.
  6. Backoff. openab-pty attaches now reset the redial backoff after a connection stayed up stableAfter (60 s), as switchboard attaches already did. A drop after a long attach redials in about 1 s instead of 30 s.

Coordination with openab-pty

  • openab-pty#86 adds close code 4011 for the liveness close. This PR treats it as redial, and an older runtime that still sends 1000 is also redialled, so the two PRs can merge in either order.
  • exec's 100 s limit is tied to the runtime's 120 s REQUEST_TIMEOUT. If that changes, ReverseAttachClient.maxAttachedExecTimeout should follow.

Test results

Run on an M4 Mac mini (macOS 15.7) from a fresh rsync of the branch:

  • swift build -c release: OK, no warnings.
  • swift test: 150 tests, 0 failures (head after review round 2).
  • New tests:
    • ReverseAttachLivenessTests.testSlowToolCallSurvivesTheRuntimePingBudget: a fake runtime pings every 0.2 s and closes 1000 after more than 3 missed intervals, while a tool takes 2.5 s and the client's own keepalive is off. The attach survives, ≥5 pings are answered during the call, and a fast echo sent after the slow call is answered first. Passed.
    • ReverseAttachLivenessTests.testClientSendsItsOwnKeepalivePings: passed.
    • ReverseAttachEndToEndTests.testRuntimeAttachResetsBackoffAfterAStableConnection: passed.
    • ReverseAttachPolicyTests.testLivenessCloseRedials, testLongExecIsRefusedOnARuntimeAttachWithAPointerToExecStart, testLongExecIsNotRefusedOnASwitchboardAttach, testEveryGrantProfileIsToldToKeepCallsShort: passed.
    • ReverseAttachLivenessTests.testCallsRunningWhenTheSocketClosesAreCancelled: a fake runtime starts a 3 s cancellation-aware call, closes 1000 after 0.3 s, and the client redials. The call is cancelled, never completes, and only the new initialize is answered on the second connection. Passed.
    • ReverseAttachLivenessTests.testRevokingTheGrantCancelsCallsInFlight: cancel() during a slow call cancels it. Passed.
    • ExecToolTests.testCancellingTheCallKillsTheProcessGroup: cancelling exec of sleep 3; touch <marker> returns at once with exit 137, and the marker is never written. Passed.
    • ExecToolTests.testCancelledCallDoesNotWaitForABackgroundedChildHoldingThePipes: cancelling exec of sleep 4 & exit 0 returns within 2.5 s instead of waiting for the child. Passed.
  • Reproduction: the slow-tool test run against the unfixed main client fails three ways: the runtime evicts the Mac, 0 pings are answered, and replies arrive serialized ([2, 3]). This confirms the inference in Reverse attach drops during long tool calls: receive loop is serial, so runtime pings go unanswered #68 that URLSession answers pings only while a receive() is outstanding.

Not deployed anywhere; that is a separate step.

chaodu-agent and others added 2 commits November 4, 2026 13:59
The openab-pty runtime pings every 20 s and drops a Mac that sends no
frame for more than three intervals. dialOnce handled one frame at a
time, so no receive() was outstanding while a tool ran and URLSession
answered no pings: any call over ~60-80 s evicted the Mac (#68).

- Keep ws.receive() always outstanding. Each inbound frame is answered
  in its own task; replies go out through one writer task per socket,
  so calls also run concurrently. Our own cancellation still reaches
  in-flight calls.
- Ping the peer every 15 s while attached (keepaliveInterval).
- Treat close code 4011 (runtime liveness close, openab-pty#86) as
  redial, with a unit test.
- Reset the redial backoff after a stable connection on openab-pty
  attaches too, as switchboard already did.
- On openab-pty attaches, refuse exec timeout_secs over 100 s (the
  runtime fails calls after 120 s) with a tool error pointing to
  exec_start + exec_poll.
- Tell every grant profile, owner included, to keep calls under ~60 s
  and never sleep inside exec.

Tested on macmini (rsync to ~/src/oab-instance-mcp-pr):
- swift build -c release: OK, 0 warnings
- swift test: 146 tests, 0 failures. New:
  ReverseAttachLivenessTests.testSlowToolCallSurvivesTheRuntimePingBudget
  (2.5 s tool vs a 0.8 s ping budget, keepalive off) passed,
  testClientSendsItsOwnKeepalivePings passed,
  testRuntimeAttachResetsBackoffAfterAStableConnection passed,
  testLivenessCloseRedials, testLongExecIs*, testEveryGrantProfile* passed.
- The slow-tool test on unfixed origin/main code fails: evicted,
  0 pongs answered, replies serialized [2, 3]. That reproduces the
  inferred URLSession behaviour.

Fixes #68
…rs cancellation

Review round 1 on #69:

- Every dialOnce exit (stop code, retryable close, send failure, our own
  cancel) now cancels the calls still running on that socket, so none
  outlives it or reaches the next connection. cancel() also cancels
  tracked calls directly, which covers a revoke during redial backoff.
- ExecTool.run kills its process group from a task cancellation
  handler (osascript goes through it too). The kill is guarded by a
  waitid(WNOWAIT) so it can never hit a reaped, recycled pid.
- osascript gets the same 100 s limit on openab-pty attaches as exec.
- exec / osascript timeout_secs descriptions mention the attach limit
  and exec_start + exec_poll.
- Comment in dialOnce on why there is no per-socket concurrency cap and
  that pipelined calls may finish out of order.

Tested on macmini (rsync to ~/src/oab-instance-mcp-pr):
- swift build -c release: OK, no warnings
- swift test: 149 tests, 0 failures. New:
  ReverseAttachLivenessTests.testCallsRunningWhenTheSocketClosesAreCancelled,
  ReverseAttachLivenessTests.testRevokingTheGrantCancelsCallsInFlight,
  ExecToolTests.testCancellingTheCallKillsTheProcessGroup passed;
  testSlowToolCallSurvivesTheRuntimePingBudget still passes.
@chaodu-agent

chaodu-agent commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Review round 1: findings → fixes

All fixed in 8b47951.

# Severity Finding Fix
1 HIGH In-flight calls outlive socket closure and grant revocation dialOnce's defer now calls cancelInflight() on every exit (stop codes, retryable closes, send failures, our own cancel), so no call carries into the next connection. cancel() also cancels tracked calls directly, which covers a revoke during redial backoff. ExecTool.run kills its process group from withTaskCancellationHandler; osascript runs through it, so it is covered too. The kill waits with waitid(WNOWAIT) before reaping, so it cannot hit a recycled pid. New tests: testCallsRunningWhenTheSocketClosesAreCancelled, testRevokingTheGrantCancelsCallsInFlight, ExecToolTests.testCancellingTheCallKillsTheProcessGroup. The PR body sentence and the cancelInflight doc comment now describe this behaviour.
2 LOW osascript up to 300 s on openab-pty attaches refuseLongExec covers osascript too (100 s limit), with a test.
3 LOW exec schema says max 600 timeout_secs descriptions for exec and osascript mention the 100 s attach limit and exec_start + exec_poll.
4 LOW Unbounded per-socket concurrency; UI-call order Comment in dialOnce: no Mac-side cap, the runtime bounds calls in flight (64), and pipelined calls may finish out of order, so ordered UI steps must wait for each reply.

Behaviour note: a synchronous exec still running when the attach drops is now killed. The runtime has already failed that call for the caller. Work meant to outlive a call belongs in exec_start, whose jobs are not tied to a socket.

On an M4 Mac mini (macOS 15.7): swift build -c release OK with no warnings; swift test 149 tests, 0 failures.

…ll agents to serialize UI steps

- ExecTool: the pipe-drain wait is now cancellable, so a cancelled exec returns
  even when a child backgrounded with `cmd &` still holds stdout/stderr after
  the shell leader exited (review O5). That child is not killed: once the
  leader is reaped its pid may be recycled.
- desktop and owner instructions now say calls run concurrently and to issue
  mouse/key/osascript steps one at a time (review O4).

Tested on an M4 Mac mini (macOS 15.7) from a fresh rsync: swift build -c release
OK; swift test 150 tests, 0 failures (new:
testCancelledCallDoesNotWaitForABackgroundedChildHoldingThePipes).
@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Review round 2 fixes (HEAD 71b3e8e):

Reviewer Severity ID Resolution
opus MEDIUM O1 PR body edited: the build host is now described as "an M4 Mac mini (macOS 15.7)"; test counts updated. No code change.
opus LOW O2 Not fixed on the branch: rewriting history needs a force push. Please squash-merge with an edited message so the old "Tested on …" lines in fad111d and 8b47951 do not reach main. 71b3e8e does not repeat them.
opus LOW O4 71b3e8e: desktop and owner instructions now say calls run concurrently and to issue mouse/key/osascript steps one at a time (AttachManager.uiOrderNotice, covered by testEveryGrantProfileIsToldToKeepCallsShort).
opus LOW O5 71b3e8e: the pipe-drain wait in ExecTool.run is cancellable, so a cancelled call returns even when a cmd & child still holds stdout/stderr. That child is not killed, because the leader's pid may already be recycled. New test testCancelledCallDoesNotWaitForABackgroundedChildHoldingThePipes.
opus LOW O6 Out of scope for this PR (Linux port untouched). Filed #70.

Sol raised no findings.

Build: swift build -c release OK; swift test 150 tests, 0 failures.

@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Review round 3: findings → fixes

No code change; HEAD stays at 71b3e8e.

Reviewer Severity ID Resolution
opus MEDIUM O7 (G1) Edited the "Review round 1: findings → fixes" comment so its test line no longer names a build host. Rescanned the body, issue comments, review comments and reviews on #69 and openab-pty#87: no host names, tailnet addresses or logins remain.
opus / sol LOW O2 / S1 (G2) Not fixed in-branch: rewriting fad111d and 8b47951 needs a force push. Squash-merge with an edited message so those lines do not reach main.
opus / sol LOW O2 / S2 (G3) Same for openab-pty#87 (873a849, fccc69e, 26932e1): squash-merge with an edited message.

@chaodu-agent
chaodu-agent merged commit 607a153 into main Nov 5, 2026
8 checks passed
chaodu-agent added a commit that referenced this pull request Nov 5, 2026
Co-authored-by: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reverse attach drops during long tool calls: receive loop is serial, so runtime pings go unanswered

1 participant