Skip to content

Keep terminated deploys terminated - #420

Merged
Priyanshu-u07 merged 3 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:fix/deploy-state-race
Oct 24, 2026
Merged

Priyanshu-u07 merged 3 commits into
InferiaAI:mainfrom
Dhirenderchoudhary:fix/deploy-state-race

Conversation

@Dhirenderchoudhary

Copy link
Copy Markdown
Collaborator

If you terminate a deployment while the worker is still provisioning it, the worker can set it back to RUNNING when it finishes.

update_state was a plain UPDATE ... WHERE deployment_id=$1, and the worker wrote RUNNING (and FAILED) with it at the end of each provisioning path. There's a "state changed during provisioning" check in the job path, but after it the worker still does update_endpoint, register_node and attach_runtime before writing RUNNING. The placement path writes RUNNING after strategy.deploy with no check. A terminate landing in that gap gets overwritten: the deploy shows RUNNING again, but terminate already released its GPU.

Changes:

  • update_state takes an optional unless_in and skips the write if the current state is in it. It returns whether it wrote. Callers that don't pass it behave the same as before, so start/resume from TERMINATED still works.
  • In handle_deploy_requested the worker's state writes skip when the deploy is TERMINATING, TERMINATED or STOPPED.
  • If the RUNNING write gets skipped, the worker raises, so it goes through the existing cleanup in the except block and deprovisions the node it just made, instead of leaving it running.
  • The except block used to read the state and then write FAILED. It now uses the same guarded write, so that check is atomic too.

Testing:

  • New test test_terminate_during_provisioning_is_not_overwritten: it terminates during attach_runtime. It fails on main (state ends up RUNNING) and passes here (state stays TERMINATED and the node is deprovisioned).
  • Ran the new SQL against Postgres: TERMINATED -> RUNNING with the guard is skipped. PROVISIONING -> RUNNING, NULL -> RUNNING, and TERMINATED -> PENDING without the guard all still update.
  • Ran the full suite with the CI ignore list on main and on this branch. The same tests fail on both.

Not covered: on the placement path, if terminate already ran, a model that strategy.deploy loaded on an existing node isn't unloaded here. The state now stays right, but that's a separate cleanup.

Dhirenderchoudhary and others added 3 commits October 23, 2026 00:47
SCHEDULING and DEPLOYING were still plain writes, so a terminate before placement was overwritten and later promoted to RUNNING. Use the guarded write and return when the deploy has ended. When an ended deploy deprovisions the node it just created, also mark that node's inventory row terminated so it does not stay ready.

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>

@Priyanshu-u07 Priyanshu-u07 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed two fixes to this branch:

  • The placement path's SCHEDULING and DEPLOYING writes now use the guarded write and return if the deploy has ended, so a terminate before placement is no longer overwritten. Nothing has been created at that point, so there's nothing to clean up.
  • When an ended deploy deprovisions the node it just created, its inventory row is now marked terminated instead of staying ready.
  • Tests added for both.

@Priyanshu-u07
Priyanshu-u07 merged commit 3f6345e into InferiaAI:main Oct 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants