Repository navigation
0.29.1: stop restarting a service that never comes up - #4
Merged
Merged
Conversation
…rts tries A service whose start timed out was restarted on the next failed wake, then timed out again, forever: one stop/start cycle per ready_timeout_s. Seen with an exclusive window that ends while another process still holds the GPU. Restarts are now capped (max_restarts, default 3); then GapRunner alerts and leaves it to requests' wakes every dead_retry_s. The count resets when the service answers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A service that starts but never becomes ready was stopped and started forever, one cycle per
ready_timeout_s: the start timed out, the next request's wake failed, and GapRunner restarted it. This happens when an exclusive window ends while a process GapRunner doesn't control still holds the GPU, or when the engine can't load.Restarts are now capped by a new engine setting,
max_restarts(default 3). After that GapRunner stops runningstop_cmd/start_cmd, sends anerroralert and logsservice_restart_gave_up. Requests still try a wake everydead_retry_s, so a service started by hand is picked up. The count resets when the service answers.Tests:
tests/test_restart_cap.py(fails on main, passes here); full suite 198 tests OK.