Conversation
…ound ACP init A self-hosted runner that follows the documented subscription scheme (run the container as your own uid with HOME=/tmp) left a Codex session reported as "running" for 17.6 minutes with no feedback, and the API watchdog settled it as lost. The image bakes a pinned codex-acp under $HOME/.local/share/sandbox-agent, and the daemon resolves that directory from the HOME it runs with. Overriding HOME moves the directory, so the daemon found no adapter and cold-installed one: it fetched the floating latest codex-acp and then verified it by running `codex-acp --help` with stdin=/dev/null, a probe that never returns on any version we tested. Nothing on that path had a deadline. The baked tree was also mode 0700, so a foreign uid could not have read it anyway. Four changes: - The runner seeds the baked adapter into the daemon's data dir at boot, before any daemon starts, and rewrites the launcher to exec out of the copy. It never seeds over an existing install and never throws. - Both images make the baked pin readable by any uid. - Opening the ACP session is bounded by AGENTA_RUNNER_ACP_INIT_TIMEOUT_MS (default 120s) and fails with a typed error naming the harness. - Sandbox teardown is bounded at 30s per provider call. A call that never settled used to hang the failed run before it could answer its caller, which is how the timeout was found to be insufficient on its own.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The symptom
On a self-hosted stack, a Codex session sat at "running" for 17.6 minutes with no output and no error. The runner kept heart-beating the turn, the agent service's read timed out, and the API watchdog finally settled the execution as
lostafter 15 minutes. The user had no error to read and nothing to retry.The deployment followed the documented subscription scheme: the runner container runs as the operator's own uid with
HOME=/tmp.The cause
Two things, and the second is what made the first fatal.
The runner image bakes a pinned Codex ACP adapter (
@agentclientprotocol/codex-acp1.1.7, patched) under$HOME/.local/share/sandbox-agent, withENV HOMEfixed so the build and the runtime agree. The harness daemon resolves that directory from$XDG_DATA_HOME, else$HOME/.local/share. OverridingHOMEmoves the directory, the daemon finds no adapter there, and it cold-installs one instead: it downloads the native codex binary from GitHub "latest", npm-installs the floating latestcodex-acp(1.12.0 today, unpatched and drifting), and then verifies it by runningcodex-acp --helpwith stdin set to/dev/null.That probe never returns. It hangs on every version tested (1.1.7, 1.11.0, 1.12.0), and it exits in two seconds only when stdin is a closed pipe. The daemon puts no timeout on it.
Nothing on the runner's side had a deadline either, so
initializenever answered and the stage recorded[timing] stage=create_session ms=1059706.The baked tree was also mode 0700, so a container running as any other uid could not have read the pin even if it had looked there.
The fix
The runner seeds the pin itself. At boot, before any daemon starts, it computes the daemon's data directory the same way the daemon does and copies the baked
bin/agent_processesandbin/codexinto it, then rewrites the launcher'sexecpath to the copy. It skips when the data directory is the baked location, skips when anything is already installed there, and logs one line either way. A missing or unreadable pin is a warning, never a crash.Both images make the pin readable by any uid, so the seed can read what it is copying.
Opening the ACP session is bounded.
AGENTA_RUNNER_ACP_INIT_TIMEOUT_MS(default 120000) caps the handshake and fails the run with a typed error naming the harness ("Codex harness did not initialize within 120s"), which settles the turn on the same path a harness crash takes. A load that times out does not degrade into a create, because that would spend the budget twice on a harness that is not answering. This bounds the handshake only, never the model's own time.Sandbox teardown is bounded too. The first end-to-end test of the timeout found that the failed run still could not answer its caller: with an ACP request in flight, the local provider's teardown does not return, so the run hung anyway. Each provider teardown call now gets 30 seconds, after which teardown moves on and logs that it did.
Verification
Built the gh image from this branch and ran it as
--user 1004:1004 -e HOME=/tmp.The boot log:
The seeded adapter is the pin, and the launcher points at the copy:
A daemon started by hand inside that container answered
initializein 219ms withagentInfo.version1.1.7, where the old image hung indefinitely.For the negative case, the seeded launcher was replaced with
sleep 999and the budget lowered to 3000ms. The run now fails instead of hanging:The run returned in 33.3 seconds, against a request that never returned at all before the teardown bound.
Runner suite: 186 files, 3149 tests, all passing.
tsc --noEmitclean.One note for anyone rebuilding the image from
main: the build currently fails at the agent-tools layer withE: Version '2.100.0*' for 'gh' was not found, because that version left GitHub's apt repo. It is unrelated to this change, and the fix (#6895) has landed onrelease/v0.118.3but not onmain. The QA image here was built with that pin bumped locally, and the bump is not part of this branch.Note for reviewers
The 120s default is a trade-off worth a second opinion. Claude Code is installed at runtime by design (licensing), so a genuinely cold Claude install on a slow link now has 120 seconds to finish the handshake before the run fails. The variable exists to raise it.