Skip to content

Wait ten minutes for a build's machine after a Chopin restart - #713

Merged
lavaman131 merged 1 commit into
mainfrom
longer-restart-grace
Oct 21, 2026
Merged

lavaman131 merged 1 commit into
mainfrom
longer-restart-grace

Conversation

@lavaman131

Copy link
Copy Markdown
Collaborator

A build survived a hosted outage of 89 seconds with one second of its 90-second post-restart window left. A slow deploy or a host reboot would have ended running builds.

Server: a longer, configurable window

The server now waits 10 minutes by default for a build's machine to come back after a restart. That matches how long the connector keeps an agent running without reaching Chopin. CHOPIN_BUILD_RESUME_MS changes the window; it accepts 1000 to 3600000 ms and is documented in docs/self-hosting.md.

One rule decides whether a build may still resume, and every place that used the old 90-second window now asks it, so they never disagree:

Place Follows the new window? Why
Sweep (ending a build nobody carried on) Yes This is the window itself. A build whose machine never returns ends as Your local agent disconnected. once it passes.
unavailable (503) answers to the agent's run credential Yes The agent should keep retrying exactly as long as the build can still resume, and no longer.
Takeover of a forgotten connection's build by the machine's new connection Yes A machine back at minute 5 must be able to take the build back.
A machine that registered again without carrying its build on Gets 1 minute, measured from its oldest live connection A connector still running the build renews it within seconds of registering. It gets a heartbeat's allowance (the same 60 s a renewal grants) instead of the whole window, then the build ends as Your local agent stopped responding.
Connection expiry (90 s without hearing from a connector) No It's unrelated to restarts. Raising it would delay noticing a vanished machine in normal operation.
Connector's two-minute "forgotten" window No After a restart, a forgotten connection registers again on the next call; that window only covers repeated forgotten refusals.

Connector: retries are visible

While Chopin is out of reach during a run, the heartbeat loop prints a line at most every 10 seconds, then reports the outage length when contact returns:

  … Lost contact with Chopin. The agent keeps working; reports wait until it is back.
  … Still trying to reach Chopin (attempt 3, 12 s, ConnectionRefused), the run keeps going.
  … Still trying to reach Chopin (attempt 9, 36 s, HTTP 502), the run keeps going.
  ✓ Back in contact with Chopin after 44 s.

The line names only the HTTP status and/or the server's error code, or the network error code. It never includes the failure's message, so no token can appear. A test checks this with a bearer token placed in an error message.

The connector change needs a CLI release (no protocol change). There's a note under "Notes for the next release" in docs/releasing-cli.md.

Tests (written red first)

  • routes.test.ts: a build carries on after a five-minute outage. The agent gets 503 unavailable every minute, then the machine's new connection takes the build back.
  • routes.test.ts: a build whose machine never comes back ends as disconnected, both after the default 10-minute window and after a configured 2-minute window.
  • routes.test.ts: a machine that comes back without its build ends it as stopped responding one minute later.
  • config-build-resume.test.ts: the default, the bounds and malformed values.
  • run.test.ts (fake clock): checks the rate-limited retry lines, the outage length and that no token reaches the terminal.
  • e2e/build-restart.e2e.ts: the connector says how long Chopin was out of reach.

A build survived a hosted outage of 89 seconds with one second of its
90-second post-restart window left, so a slow deploy or a host reboot
would have ended running builds.

The server now waits 10 minutes by default for a build's machine to come
back after a restart, as long as the connector keeps an agent running
without reaching Chopin. CHOPIN_BUILD_RESUME_MS changes it (1 second to
1 hour). One rule decides whether a build may still resume, and all three
places that use it follow the new window: the sweep keeps the build
running, the agent's calls are answered unavailable with a request to
retry, and the machine's new connection may take the build back. A
machine that registered again without carrying its build on gets a
heartbeat's allowance, one minute from its oldest live connection, before
the build ends as stopped responding, rather than the whole window. A
build whose machine never returns ends as disconnected when the window
passes. The connection expiry (90 seconds without hearing from a
connector) and the connector's two-minute forgotten window are unrelated
to restarts and stay as they were.

While Chopin is out of reach during a run, the connector now says so at
most every 10 seconds: Still trying to reach Chopin (attempt 4, 40 s,
HTTP 502), the run keeps going. It names only the HTTP status or error
code, never the failure's message, and the back-in-contact line says how
long the outage lasted. That part needs a CLI release; no protocol
change.

Assistant-model: Claude Opus 5.5 (fast)
Assistant-workflow: inline (fast subagents)
Assistant-verification: bun test passed (6704 pass, 0 fail)
Assistant-verification: bun run types passed
Assistant-verification: bun run ci passed
Co-authored-by: Alex Lavaee <alexlavaee@microsoft.com>
@lavaman131 lavaman131 self-assigned this Oct 21, 2026
@lavaman131
lavaman131 merged commit cc4c548 into main Oct 21, 2026
24 checks passed
@lavaman131
lavaman131 deleted the longer-restart-grace branch October 21, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant