Repository navigation
Wait ten minutes for a build's machine after a Chopin restart - #713
Merged
Merged
Conversation
A build survived a hosted outage of 89 seconds with one second of its 90-second post-restart window left, so a slow deploy or a host reboot would have ended running builds. The server now waits 10 minutes by default for a build's machine to come back after a restart, as long as the connector keeps an agent running without reaching Chopin. CHOPIN_BUILD_RESUME_MS changes it (1 second to 1 hour). One rule decides whether a build may still resume, and all three places that use it follow the new window: the sweep keeps the build running, the agent's calls are answered unavailable with a request to retry, and the machine's new connection may take the build back. A machine that registered again without carrying its build on gets a heartbeat's allowance, one minute from its oldest live connection, before the build ends as stopped responding, rather than the whole window. A build whose machine never returns ends as disconnected when the window passes. The connection expiry (90 seconds without hearing from a connector) and the connector's two-minute forgotten window are unrelated to restarts and stay as they were. While Chopin is out of reach during a run, the connector now says so at most every 10 seconds: Still trying to reach Chopin (attempt 4, 40 s, HTTP 502), the run keeps going. It names only the HTTP status or error code, never the failure's message, and the back-in-contact line says how long the outage lasted. That part needs a CLI release; no protocol change. Assistant-model: Claude Opus 5.5 (fast) Assistant-workflow: inline (fast subagents) Assistant-verification: bun test passed (6704 pass, 0 fail) Assistant-verification: bun run types passed Assistant-verification: bun run ci passed Co-authored-by: Alex Lavaee <alexlavaee@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A build survived a hosted outage of 89 seconds with one second of its 90-second post-restart window left. A slow deploy or a host reboot would have ended running builds.
Server: a longer, configurable window
The server now waits 10 minutes by default for a build's machine to come back after a restart. That matches how long the connector keeps an agent running without reaching Chopin.
CHOPIN_BUILD_RESUME_MSchanges the window; it accepts 1000 to 3600000 ms and is documented indocs/self-hosting.md.One rule decides whether a build may still resume, and every place that used the old 90-second window now asks it, so they never disagree:
unavailable(503) answers to the agent's run credentialConnector: retries are visible
While Chopin is out of reach during a run, the heartbeat loop prints a line at most every 10 seconds, then reports the outage length when contact returns:
The line names only the HTTP status and/or the server's error code, or the network error code. It never includes the failure's message, so no token can appear. A test checks this with a bearer token placed in an error message.
The connector change needs a CLI release (no protocol change). There's a note under "Notes for the next release" in
docs/releasing-cli.md.Tests (written red first)
routes.test.ts: a build carries on after a five-minute outage. The agent gets 503unavailableevery minute, then the machine's new connection takes the build back.routes.test.ts: a build whose machine never comes back ends as disconnected, both after the default 10-minute window and after a configured 2-minute window.routes.test.ts: a machine that comes back without its build ends it as stopped responding one minute later.config-build-resume.test.ts: the default, the bounds and malformed values.run.test.ts(fake clock): checks the rate-limited retry lines, the outage length and that no token reaches the terminal.e2e/build-restart.e2e.ts: the connector says how long Chopin was out of reach.