Skip to content

Terminate during pool-first deploy can be overwritten by RUNNING #436

Closed

Description

@Priyanshu-u07

DeploymentLinker promotes a deploy from DEPLOYING to RUNNING with a plain set_state and no terminate check (deployment_linker.py). If the deploy is terminated while the model is loading, the linker can still write RUNNING after terminate has released the GPU. The window is long, since load_model waits for the image pull and model load on the worker.

The load-failure path has the same problem: it writes FAILED over TERMINATED.

This is the dashboard path (/deployment/deploy → place_and_provision). #420 fixed the same race only in the legacy deploy-requested worker. Suggest a guarded write that skips TERMINATING, TERMINATED and STOPPED, like #420's update_state(..., unless_in=...).

Found by reading the code; not reproduced yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions