Skip to content

fix(runner): seed the pinned Codex adapter under HOME overrides and bound ACP init - #6924

Draft
mmabrouk wants to merge 1 commit into
mainfrom
fix/runner-codex-adapter-seed-and-init-timeout
Draft

mmabrouk wants to merge 1 commit into
mainfrom
fix/runner-codex-adapter-seed-and-init-timeout

Conversation

@mmabrouk

Copy link
Copy Markdown
Member

The symptom

On a self-hosted stack, a Codex session sat at "running" for 17.6 minutes with no output and no error. The runner kept heart-beating the turn, the agent service's read timed out, and the API watchdog finally settled the execution as lost after 15 minutes. The user had no error to read and nothing to retry.

The deployment followed the documented subscription scheme: the runner container runs as the operator's own uid with HOME=/tmp.

The cause

Two things, and the second is what made the first fatal.

The runner image bakes a pinned Codex ACP adapter (@agentclientprotocol/codex-acp 1.1.7, patched) under $HOME/.local/share/sandbox-agent, with ENV HOME fixed so the build and the runtime agree. The harness daemon resolves that directory from $XDG_DATA_HOME, else $HOME/.local/share. Overriding HOME moves the directory, the daemon finds no adapter there, and it cold-installs one instead: it downloads the native codex binary from GitHub "latest", npm-installs the floating latest codex-acp (1.12.0 today, unpatched and drifting), and then verifies it by running codex-acp --help with stdin set to /dev/null.

That probe never returns. It hangs on every version tested (1.1.7, 1.11.0, 1.12.0), and it exits in two seconds only when stdin is a closed pipe. The daemon puts no timeout on it.

Nothing on the runner's side had a deadline either, so initialize never answered and the stage recorded [timing] stage=create_session ms=1059706.

The baked tree was also mode 0700, so a container running as any other uid could not have read the pin even if it had looked there.

The fix

The runner seeds the pin itself. At boot, before any daemon starts, it computes the daemon's data directory the same way the daemon does and copies the baked bin/agent_processes and bin/codex into it, then rewrites the launcher's exec path to the copy. It skips when the data directory is the baked location, skips when anything is already installed there, and logs one line either way. A missing or unreadable pin is a warning, never a crash.

Both images make the pin readable by any uid, so the seed can read what it is copying.

Opening the ACP session is bounded. AGENTA_RUNNER_ACP_INIT_TIMEOUT_MS (default 120000) caps the handshake and fails the run with a typed error naming the harness ("Codex harness did not initialize within 120s"), which settles the turn on the same path a harness crash takes. A load that times out does not degrade into a create, because that would spend the budget twice on a harness that is not answering. This bounds the handshake only, never the model's own time.

Sandbox teardown is bounded too. The first end-to-end test of the timeout found that the failed run still could not answer its caller: with an ACP request in flight, the local provider's teardown does not return, so the run hung anyway. Each provider teardown call now gets 30 seconds, after which teardown moves on and logs that it did.

Verification

Built the gh image from this branch and ran it as --user 1004:1004 -e HOME=/tmp.

The boot log:

[adapter-seed] seeded the pinned Codex adapter into the daemon data dir: /home/node/.local/share/sandbox-agent/bin/agent_processes -> /tmp/.local/share/sandbox-agent (947ms)

The seeded adapter is the pin, and the launcher points at the copy:

  "version": "1.1.7",
exec '/tmp/.local/share/sandbox-agent/bin/agent_processes/codex/node_modules/.bin/codex-acp' "$@"

A daemon started by hand inside that container answered initialize in 219ms with agentInfo.version 1.1.7, where the old image hung indefinitely.

For the negative case, the seeded launcher was replaced with sleep 999 and the budget lowered to 3000ms. The run now fails instead of hanging:

{"ok":false,"error":"Codex harness did not initialize within 3s."}
[sandbox-agent] [timing] stage=create_session ms=3000 ... mode=create
[sandbox-agent] sandbox delete did not finish within 30000ms sandbox=local/...; continuing teardown without it

The run returned in 33.3 seconds, against a request that never returned at all before the teardown bound.

Runner suite: 186 files, 3149 tests, all passing. tsc --noEmit clean.

One note for anyone rebuilding the image from main: the build currently fails at the agent-tools layer with E: Version '2.100.0*' for 'gh' was not found, because that version left GitHub's apt repo. It is unrelated to this change, and the fix (#6895) has landed on release/v0.118.3 but not on main. The QA image here was built with that pin bumped locally, and the bump is not part of this branch.

Note for reviewers

The 120s default is a trade-off worth a second opinion. Claude Code is installed at runtime by design (licensing), so a genuinely cold Claude install on a slow link now has 120 seconds to finish the handshake before the run fails. The variable exists to raise it.

…ound ACP init

A self-hosted runner that follows the documented subscription scheme (run the
container as your own uid with HOME=/tmp) left a Codex session reported as
"running" for 17.6 minutes with no feedback, and the API watchdog settled it as
lost.

The image bakes a pinned codex-acp under $HOME/.local/share/sandbox-agent, and
the daemon resolves that directory from the HOME it runs with. Overriding HOME
moves the directory, so the daemon found no adapter and cold-installed one: it
fetched the floating latest codex-acp and then verified it by running
`codex-acp --help` with stdin=/dev/null, a probe that never returns on any
version we tested. Nothing on that path had a deadline. The baked tree was also
mode 0700, so a foreign uid could not have read it anyway.

Four changes:

- The runner seeds the baked adapter into the daemon's data dir at boot, before
  any daemon starts, and rewrites the launcher to exec out of the copy. It never
  seeds over an existing install and never throws.
- Both images make the baked pin readable by any uid.
- Opening the ACP session is bounded by AGENTA_RUNNER_ACP_INIT_TIMEOUT_MS
  (default 120s) and fails with a typed error naming the harness.
- Sandbox teardown is bounded at 30s per provider call. A call that never
  settled used to hang the failed run before it could answer its caller, which
  is how the timeout was found to be insufficient on its own.
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant