Repository navigation
Keep deploys terminated during model load - #437
Merged
Priyanshu-u07 merged 2 commits intoOct 25, 2026
Merged
Conversation
Priyanshu-u07
approved these changes
Oct 25, 2026
Priyanshu-u07
left a comment
Collaborator
There was a problem hiding this comment.
The linker fix looks right. I ran the new tests against a Postgres with the repo's schema: they fail on main and pass here, and all 16 linker and 9 worker lifecycle tests pass. Guarding FAILED and releasing the GPU only when it lands is the right fix for the double release.
The warm path in place_and_provision has the same race (deployment_server.py). #436 didn't cover it, so I'll fix it in a separate PR. I've changed this PR to "Part of #436" so the issue stays open until then.
Note: CI ignores test_deployment_linker.py and test_deploy_endpoint_pool_first.py, so these tests never run there. I'll open an issue for that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #436
Why
If a dashboard deploy was terminated while the model was still loading, the linker could still write RUNNING (or FAILED) over TERMINATED. On the failure path it also released the GPU a second time, which can take the GPU away from another deploy that got it after terminate.
What changed
update_state(..., unless_in=ENDED_STATES)for both RUNNING and FAILED, same as Keep terminated deploys terminated #420 did for the legacy worker.ENDED_STATESmoved tomodel_deployment_reposo the worker and the linker share it.Verification
load_model, for both success and failure. It failed on main (RUNNING/FAILEDinstead ofTERMINATED) and passes now. It also checks the GPU count is not touched andunload_modelis called.