Skip to content

fix(tools): close a ping-silent Mac with 4011 and report last_detach_reason - #87

Merged
chaodu-agent merged 5 commits into
mainfrom
fix/liveness-close-code
Nov 5, 2026
Merged

chaodu-agent merged 5 commits into
mainfrom
fix/liveness-close-code

Conversation

@chaodu-agent

@chaodu-agent chaodu-agent commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #86. Client side: openabdev/openab-computer#68.

Problem

The tools attach pings the Mac every 20 s and, after more than three silent intervals, closed it with plain WS 1000. That reads like a normal close in both logs. instance_status then reported only attached=false, grant_pending=true, so the PTY agent took it as a lost lease and asked the human to lend the Mac again, while the Mac was about to redial on its own. The Mac went silent because instance-mcp's reverse-attach client handles one frame at a time and does not answer pings during a long tool call (fixed in openab-computer#68).

Change

  • Close code 4011 LIVENESS_TIMEOUT. The ping-budget path now closes with 4011 instead of 1000, and the log line carries the code. Codes 4001–4010 are all taken.
  • last_detach_reason in instance_status. The hub records the last detach per session: when the current socket goes away (liveness timeout, Mac closed or dropped) and when a grant is revoked or expires. While nothing is attached, instance_status returns last_detach_reason (liveness_timeout, closed, revoked, ttl_expired, session_ended, or null), last_detach_close_code, last_detach_secs_ago and a hint. With a valid grant the hint says the Mac redials on its own within about 30 s, so the agent should wait and check again. With no grant it says to ask the human. A tool call made while detached with a valid grant returns the same "wait for the redial" text instead of "ask the human to lend one". The record is forgotten with the session.
  • ServerConfig.tools_ping_interval (default PING_INTERVAL, 20 s) lets the e2e suite see a liveness close in under a second. Production keeps 20 s.
  • CLIENT-CONTRACT.md. 4011 is added to the §5 and §9.2 close-code tables, and the retry rule changes to "give up on any 4xxx except 4006 and 4011". §9.2 item 3 now states the liveness rule: keep a receive outstanding during a call, missed pings count even while a request is in flight, and redial on 4011. §9.3 documents the new status fields.

Decision: missed pings still count while a request is in flight

I did not suspend the ping budget while a request younger than REQUEST_TIMEOUT (120 s) is in flight. Reasons:

  • A ping is the runtime's only way to tell a busy Mac from a half-open TCP path, such as a Wi-Fi or tailnet path change that produces no error. Suspending it for in-flight requests would keep a dead socket attached for up to 120 s more, and the waiting caller would get -32003 at 120 s anyway.
  • It would not have saved the observed drops. They came from exec calls lasting up to 600 s, so suspension would only move the eviction from about 80 s to about 200 s, and the call would still fail with -32003 at 120 s.
  • A client that keeps a receive outstanding answers pings during any call, because the WS layer does it. openab-computer#68 makes instance-mcp do this, so the budget costs a correct client nothing. An older client now gets a code that names the cause, plus a status hint, instead of a silent 1000.

Coordination with openab-computer

openab-computer#68 (PR A) treats exactly 4011 as redial in ReverseAttachClient's close-code disposition for the openab-pty attach. Today's client already redials on any code it does not list as terminal (4001, 4002, 4004, 4010), so this PR also works with the currently deployed client. The two can merge in either order.

Test results

Run on a macOS build host (runtime/, same commands as ci.yml):

  • cargo check --locked --all-targets: ok
  • cargo clippy --locked --all-targets -- -D warnings: ok
  • cargo test --locked: 186 + 29 passed, 0 failed
  • cargo test --locked -- --ignored --test-threads=1: 2 + 16 + 9 passed, 0 failed
  • cargo fmt --check: ok
  • CI on fccc69e: all checks green (runtime ubuntu-24.04 and macos-14, image matrix, manifests, startup hook). The first push failed the new e2e test on CI with BrokenPipe: reading through tungstenite queued a pong into the already-closed socket. The stuck Mac now reads the raw TCP stream and finds the 4011 close frame on the wire (5/5 local reruns pass).

New tests:

  • tools_e2e::a_ping_silent_mac_is_closed_with_liveness_timeout_and_may_redial: a Mac that answers pings outlives the budget. A Mac that never reads is closed with 4011. instance_status reports liveness_timeout with grant_pending=true and the redial hint, and a tool call says to wait for the redial. The same secret then redials and serves calls. A later revoke reads as revoked with no grant.
  • tools::tests::detached_status_names_the_last_detach_and_whether_to_wait and liveness_has_its_own_close_code_and_reason.

chaodu-agent and others added 3 commits November 4, 2026 13:59
…reason

The tools attach closed a Mac that missed the ping budget with plain 1000,
which reads like a normal close on both sides, and instance_status then
showed only grant_pending=true. Agents read that as a lost lease and asked
the human to re-lend a Mac that was about to redial on its own.

- close_code::LIVENESS_TIMEOUT = 4011 (4001-4010 are taken); the ping
  budget now closes with it.
- The hub records the last detach per session (current socket going away,
  or a revoke/expiry). instance_status while detached reports
  last_detach_reason / last_detach_close_code / last_detach_secs_ago and a
  hint: wait for the redial while the grant is valid, ask the human only
  when there is no grant. A tool call while detached with a valid grant
  says the same instead of "ask the human".
- Missed pings still count while a request is in flight (decision and
  reason documented on PING_INTERVAL).
- ServerConfig.tools_ping_interval (default PING_INTERVAL) so the e2e
  suite can observe a liveness close in under a second.
- CLIENT-CONTRACT.md: 4011 in the §5 and §9 tables, §9.2 liveness rule
  (keep a receive outstanding during calls; redial on 4011), §9.3 status
  shape.

Fixes #86

Verified on macmini (rsync to ~/src/openab-pty-pr, runtime/):
- cargo check --locked --all-targets: ok
- cargo clippy --locked --all-targets -- -D warnings: ok
- cargo test --locked: 186 + 29 passed, 0 failed
- cargo test --locked -- --ignored --test-threads=1: 2 + 16 + 9 passed
  (incl. new a_ping_silent_mac_is_closed_with_liveness_timeout_and_may_redial)
- cargo fmt --check: ok
- cargo build --locked: ok
CI (ubuntu-24.04, macos-14) failed the new liveness e2e test with
BrokenPipe: reading through tungstenite queued a pong for the first
buffered ping and flushed it into a socket the runtime had already
closed, before the close frame was returned. The stuck Mac now reads the
raw TCP stream until EOF and looks for the 0x88 close frame carrying
4011, which is also closer to what a blocked receive loop does.

Verified on macmini (~/src/openab-pty-pr/runtime):
- cargo fmt --check, cargo check --locked --all-targets,
  cargo clippy --locked --all-targets -- -D warnings: ok
- cargo test --locked: 186 + 29 passed
- cargo test --locked -- --ignored --test-threads=1: 2 + 16 + 9 passed
- new test alone 5x in a row: 5/5 passed
detach_reason() also yields runtime_replaced (close_all during a drain) and
takeover (reserved); the §9 list named only five.

Verified on macmini: cargo test --locked (all suites ok, 186 unit passed) and
cargo test --locked --test tools_e2e -- --ignored --test-threads=1 (9 passed,
incl. a_ping_silent_mac_is_closed_with_liveness_timeout_and_may_redial).
@chaodu-agent

Copy link
Copy Markdown
Collaborator Author

First fix pass (no reviewer findings yet).

Verified: cargo test --locked (all suites pass) and cargo test --test tools_e2e -- --ignored --test-threads=1 (9/9 pass) on macOS arm64.

close_all drains the attached map before closing each socket, so a drain never
records a detach (review O3). Doc-only change; no build or test run.
@chaodu-agent

Copy link
Copy Markdown
Collaborator Author

Review round 2 fixes (HEAD 30dc15a):

Reviewer Severity ID Resolution
opus LOW O3 30dc15a: CLIENT-CONTRACT.md §9.3 now lists runtime_replaced and takeover as reserved and not recorded today, because a drain removes every socket from the attached map before closing it. Doc-only.
opus LOW O2 Not fixed on the branch: rewriting history needs a force push. Please squash-merge with an edited message so the "Verified on …" lines in 873a849, fccc69e and 26932e1 do not reach main.

Sol raised no findings. No code changed in this round, so the Rust suites were not re-run locally; CI runs them on the new HEAD.

The minimum-viable-lender checklist said to stop on every 4xxx except
4006, contradicting §9.2, which requires a redial on 4011
LIVENESS_TIMEOUT. The retry list is now 1000/4006/4011.

Verification: docs-only change; grep of both PR worktrees finds no
other retry list that omits 4011. No build or test run (no code changed).
@chaodu-agent

Copy link
Copy Markdown
Collaborator Author

Review round 5 findings:

Reviewer Severity ID Resolution
Sol MEDIUM S3 (gate G1) Fixed in ea70214: the §9.4 minimum-viable-lender checklist now redials on 1000/4006/4011, matching §9.2 and the close-code table. No other retry list in either PR omits 4011.
Opus / Sol LOW O2 / S2 (gate G3) Not fixed in-branch: older commit bodies (873a849, fccc69e, 26932e1) name an internal build host, and rewriting them needs a force push. Squash-merge with an edited message so they do not reach main.

@chaodu-agent
chaodu-agent merged commit ba5fb48 into main Nov 5, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tools attach: close ping-silent Mac with distinct 4011 liveness code and report last_detach_reason

1 participant