Skip to content

Release v2.0.0-beta.18 with TensorRails DMTCP v5.0.0-warmctl.2 #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kbespalov merged 1 commit into from
Nov 4, 2026
Merged

Release v2.0.0-beta.18 with TensorRails DMTCP v5.0.0-warmctl.2 #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,17 @@
# Changelog

## Unreleased

## v2.0.0-beta.18 — 2026-11-04

Snapshots carry fewer zero pages, and restores copy fewer.

- TensorRails DMTCP `v5.0.0-warmctl.2`: a checkpoint leaves out resident
memory that holds only zeros, and restart readers share the large runs
between them. Baked with warmctl and restored on one RTX 3090, Qwen3-0.6B's
process images fell from 8,582 to 6,065 MiB and time to `/health` from
4.6–5.1 s to 4.0–4.4 s, with the same answers. Pinned CUDA host memory
already left its zeros out, so Qwen3.8-Flash-Next's CPU-side embedding
table is unchanged. Bake new snapshots with this release; images restart
under either DMTCP version.
- `up`'s stage copy (a snapshot on FUSE or stored with `--compress`) skips
all-zero pages: the staged files keep holes there. More than half of a
process image is zero pages (57% of GLM-5.3-Flash's CRIU images), and
Expand Down
13 changes: 10 additions & 3 deletions docs/dmtcp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,15 @@ already varied before checkpoint. The canary tolerance remains 0.001.
The [full experiment record](https://github.com/TensorRails/dmtcp/blob/v5.0.0-warmctl.1/doc/h200-restart-performance.md)
includes all modes, cache conditions and the unsuccessful mmap experiment.

Since [`v5.0.0-warmctl.2`](https://github.com/TensorRails/dmtcp/releases/tag/v5.0.0-warmctl.2),
the checkpoint also leaves out resident memory that holds only zeros, and the
readers share large runs of an area that zeros split. On one RTX 3090,
Qwen3-0.6B's process images fell from 8,582 to 6,065 MiB and time to
`/health` from 4.6–5.1 s to 4.0–4.4 s. Pinned CUDA host memory, such as
Qwen3.8-Flash-Next's CPU-side embedding table, already left its zeros out.

The fork is released independently as
[`v5.0.0-warmctl.1`](https://github.com/TensorRails/dmtcp/releases/tag/v5.0.0-warmctl.1),
[`v5.0.0-warmctl.2`](https://github.com/TensorRails/dmtcp/releases/tag/v5.0.0-warmctl.2),
with source, Debian 12-compatible Linux amd64 and arm64 packages, checksums
and provenance. Both packages include fdkeep. Linux warmctl archives bundle
them at `dmtcp/linux-ARCH/VERSION/` beside the CLI; the runtime Dockerfile
Expand All @@ -88,7 +95,7 @@ versioned DMTCP directories and their mtimes, and keeps older versions for
existing snapshots. Use an old snapshot's original runtime image or explicit
`--dmtcp-dir` / `--dmtcp-plugin` paths. Publishing a CLI release does not
replace an existing runtime image. See the fork's
[release procedure](https://github.com/TensorRails/dmtcp/blob/v5.0.0-warmctl.1/RELEASING.md).
[release procedure](https://github.com/TensorRails/dmtcp/blob/v5.0.0-warmctl.2/RELEASING.md).

The packages require glibc >= 2.36, compatible libstdc++ and libatomic
(`libstdc++6` and `libatomic1` on Debian/Ubuntu). ARM64 passes
Expand Down Expand Up @@ -241,7 +248,7 @@ with the restart changes above and these earlier patches:
| Accept an already unlinked shm file after restart | Processes that share an area of a deleted `/dev/shm` file (CPython's named semaphores, unlinked at once) recreate one file at restart and each unlinks it again; every one but the first found it gone, and every vLLM restart printed a warning. |
| Change a socket's connection only with the fd table locked | `setsockopt`, `connect`, `bind` and `listen` changed the connection they had looked up while another thread could delete it: `add()` drops the entry of an fd closed past the close wrapper once the number comes back. A worker restarted under cuInterpose, whose Rust threads make and close some sockets with raw syscalls, segfaulted in `SocketConnection::addSetsockopt` once in about a dozen restores. |

The plugin [fdkeep](https://github.com/TensorRails/dmtcp/tree/v5.0.0-warmctl.1/plugin/fdkeep),
The plugin [fdkeep](https://github.com/TensorRails/dmtcp/tree/v5.0.0-warmctl.2/plugin/fdkeep),
now maintained and released with the fork, repairs what DMTCP loses:

- the CUDA driver's eventfds and pipes, recreated as real pipes;
Expand Down
6 changes: 3 additions & 3 deletions internal/dmtcp/SHA256SUMS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
496b54b3405ed69dce8d0c3562d644b698150c21a921b9abed3562814724d9eb dmtcp-v5.0.0-warmctl.1-linux-amd64-debian12.tar.gz
90cf76a7733fd2998a29df4e299d9337fdbd0a8c919ccd2db3219c1e263ffd21 dmtcp-v5.0.0-warmctl.1-linux-arm64-debian12.tar.gz
f639b50aa5ecb5b3115f36f67d8ed7ad2894010dcdff43ed4e5a6276f4eeb0b1 dmtcp-v5.0.0-warmctl.1-source.tar.gz
ba19c4c96d257c0c4f78f21a2e2d3bc13f4e895f132ae47c424b67f025ed088d dmtcp-v5.0.0-warmctl.2-linux-amd64-debian12.tar.gz
97cfb934c72e0378698fde7928517b2d93ee1c034c01a725ca7933b7ac8be8b2 dmtcp-v5.0.0-warmctl.2-linux-arm64-debian12.tar.gz
c8d0321d57c1538cc4fa60150715a0d2ea198bcb53b0c58bb65121362d68bad6 dmtcp-v5.0.0-warmctl.2-source.tar.gz
2 changes: 1 addition & 1 deletion internal/dmtcp/VERSION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v5.0.0-warmctl.1
v5.0.0-warmctl.2