[shim] Persist task state to clean up tasks after a restart - #4220
Merged
Conversation
Since #4203, tasks outlive the shim process, but a restarted shim cannot finish them properly. Restoring a task from its container recovers the container ID, the GPUs, and the ports, but not the task config, so nothing tells the shim which volumes to unmount and which host SSH keys to remove. The reason a task was terminated for is lost as well, as it is not a property of the container. Each task now has a `task.json` file in its runner dir, holding what the container cannot tell: the task config, the termination reason and message, and whether the task resources are already released. It is written atomically and flushed to the disk on every task state change, starting right after the runner dir is created, that is, before any resource is acquired -- releasing a resource that was never acquired is a no-op, while the opposite order would leak. * Restored tasks get their config back, therefore their volumes are unmounted and their host SSH keys are removed once they finish. The volumes of a container created by an earlier shim version are still recovered from the container mounts; its host SSH keys are not recoverable. * A task with a recorded termination reason is restored as terminated, so that the reason reported by the server, e.g., `terminated_by_user`, is not replaced with the exit code of the container it stopped. * GPUs are locked back only for the restored tasks whose resources are not released yet, so that a task cleaned up before the restart does not hold them forever. * Task dirs that have a state file but no container are cleaned up on start. Normally, such a dir is left behind when the shim stops running before the container is created, e.g., while pulling the image. Dirs without a state file are left intact, as there is no way to tell whether they belong to a task. * The registry credentials are not persisted: the image is already pulled by the time the state is read back. Also fixes the runner dir rename fallback in remove(), which never worked: the trash name was built from the absolute path of the dir, so the rename target was a relative path under a `.trash-` dir that does not exist. Part-of: #4182 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
un-def
added a commit
that referenced
this pull request
Aug 28, 2026
`~/.dstack/runners/<name>` is mounted into the task container as `/tmp/runner`. It used to hold runner's files only, but shim now keeps its own files there as well -- the image pull log since #2903 and the task state file since #4220 -- sharing them with the runner and the user workload, which may corrupt or delete them. Nothing sensitive is stored there today, but the approach is unsafe: nothing stops a contributor from putting a secret into a file the container can read. That dir is now the task dir, private to shim, and only its new `runner` subdir is mounted into the container: ~/.dstack/runners/<container-name>/ 0700, shim only task.json pull.log runner/ 0755, mounted as /tmp/runner * `runnerDir`/`runnersDir` are renamed to `taskDir`/`tasksDir` throughout, including the `DockerParameters` methods, to signal that the dir is managed by shim rather than by runner. The `runners` path itself is kept: an upgraded shim must find the dirs of the tasks created by the previous version. * The dir of a restored task is no longer derived from the container mounts, which now point at the `runner` subdir, but from the task ID in its state file. The tasks dir is scanned once on start, and the result is shared by the restore and the orphan sweep, which used to scan the dir a second time. * A task started by a shim version that did not write state files cannot be found by its state file, so its dir, named after the container, is looked up by name and, as before, removed along with the task. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #4203, tasks outlive the shim process, but a restarted shim cannot finish them properly. Restoring a task from its container recovers the container ID, the GPUs, and the ports, but not the task config, so nothing tells the shim which volumes to unmount and which host SSH keys to remove. The reason a task was terminated for is lost as well, as it is not a property of the container.
Each task now has a
task.jsonfile in its runner dir, holding what the container cannot tell: the task config, the termination reason and message, and whether the task resources are already released. It is written atomically and flushed to the disk on every task state change, starting right after the runner dir is created, that is, before any resource is acquired -- releasing a resource that was never acquired is a no-op, while the opposite order would leak.terminated_by_user, is not replaced with the exit code of the container it stopped.Also fixes the runner dir rename fallback in remove(), which never worked: the trash name was built from the absolute path of the dir, so the rename target was a relative path under a
.trash-dir that does not exist.Part-of: #4182