Skip to content

Keep warm-path deploys terminated during model load - #440

Merged
Priyanshu-u07 merged 2 commits into
mainfrom
fix/warm-path-terminate-race
Oct 25, 2026
Merged

Priyanshu-u07 merged 2 commits into
mainfrom
fix/warm-path-terminate-race

Conversation

@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

Fixes #436

#437 fixed the linker (deploys that wait for a new node). This covers the warm path: deploys onto a node that is already ready, where place_and_provision loads the model directly.

What changed in place_and_provision:

  • RUNNING is written with update_state(..., unless_in=ENDED_STATES). If it's skipped because the deploy was terminated during the load, the model is unloaded on the worker and no endpoint is published.
  • Both load-failure branches write FAILED with the same guard and release the GPU only if that write lands, in the same transaction. Before, a failed load after a terminate released the GPU a second time, freeing GPU another deploy on the node holds.
  • The exception branch now records error_message, like the linker.

Testing:

  • New test test_terminate_during_warm_load_is_not_overwritten (load ok and load failed): a terminate lands inside load_model on a node where another deploy holds 2 GPUs. It fails on main (RUNNING / FAILED over TERMINATED) and passes here; the other deploy's GPUs stay allocated.
  • test_deploy_endpoint_pool_first.py is on CI's ignore list, so I ran it against a Postgres container with the repo schema. The other 22 failures in that file are the same on main (missing X-Organization-ID header since Check pool ownership and take new owners from the caller's org #417). Linker and worker lifecycle tests still pass.

Not changed: when RUNNING is skipped, the API response still reports DEPLOYING (the DB row is TERMINATED), same as the linker.

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
@Priyanshu-u07
Priyanshu-u07 merged commit 00952c6 into main Oct 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Terminate during pool-first deploy can be overwritten by RUNNING

1 participant