Skip to content

Keep deploys terminated during model load - #437

Merged
Priyanshu-u07 merged 2 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:fix/linker-terminate-race
Oct 25, 2026
Merged

Priyanshu-u07 merged 2 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:fix/linker-terminate-race

Conversation

@Dhirenderchoudhary

@Dhirenderchoudhary Dhirenderchoudhary commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #436

Why

If a dashboard deploy was terminated while the model was still loading, the linker could still write RUNNING (or FAILED) over TERMINATED. On the failure path it also released the GPU a second time, which can take the GPU away from another deploy that got it after terminate.

What changed

  • The linker now uses update_state(..., unless_in=ENDED_STATES) for both RUNNING and FAILED, same as Keep terminated deploys terminated #420 did for the legacy worker.
  • If RUNNING is skipped, the linker unloads the model on the worker, since terminate's own unload ran before the container existed. No endpoint is published.
  • On failure, the GPU is released only if the FAILED write landed.
  • ENDED_STATES moved to model_deployment_repo so the worker and the linker share it.

Verification

  • New test terminates the deploy inside load_model, for both success and failure. It failed on main (RUNNING / FAILED instead of TERMINATED) and passes now. It also checks the GPU count is not touched and unload_model is called.
  • Linker, worker lifecycle and model deployment tests: same failures as main (54, all existing), nothing new.

@Priyanshu-u07 Priyanshu-u07 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linker fix looks right. I ran the new tests against a Postgres with the repo's schema: they fail on main and pass here, and all 16 linker and 9 worker lifecycle tests pass. Guarding FAILED and releasing the GPU only when it lands is the right fix for the double release.

The warm path in place_and_provision has the same race (deployment_server.py). #436 didn't cover it, so I'll fix it in a separate PR. I've changed this PR to "Part of #436" so the issue stays open until then.

Note: CI ignores test_deployment_linker.py and test_deploy_endpoint_pool_first.py, so these tests never run there. I'll open an issue for that.

@Priyanshu-u07
Priyanshu-u07 merged commit 90ce983 into InferiaAI:main Oct 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants