diff --git a/CHANGELOG.md b/CHANGELOG.md index 4911ed9..fb2c499 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,17 @@ # Changelog -## Unreleased - +## v2.0.0-beta.18 — 2026-11-04 + +Snapshots carry fewer zero pages, and restores copy fewer. + +- TensorRails DMTCP `v5.0.0-warmctl.2`: a checkpoint leaves out resident + memory that holds only zeros, and restart readers share the large runs + between them. Baked with warmctl and restored on one RTX 3090, Qwen3-0.6B's + process images fell from 8,582 to 6,065 MiB and time to `/health` from + 4.6–5.1 s to 4.0–4.4 s, with the same answers. Pinned CUDA host memory + already left its zeros out, so Qwen3.8-Flash-Next's CPU-side embedding + table is unchanged. Bake new snapshots with this release; images restart + under either DMTCP version. - `up`'s stage copy (a snapshot on FUSE or stored with `--compress`) skips all-zero pages: the staged files keep holes there. More than half of a process image is zero pages (57% of GLM-5.3-Flash's CRIU images), and diff --git a/docs/dmtcp.md b/docs/dmtcp.md index 7471ef8..352e943 100644 --- a/docs/dmtcp.md +++ b/docs/dmtcp.md @@ -73,8 +73,15 @@ already varied before checkpoint. The canary tolerance remains 0.001. The [full experiment record](https://github.com/TensorRails/dmtcp/blob/v5.0.0-warmctl.1/doc/h200-restart-performance.md) includes all modes, cache conditions and the unsuccessful mmap experiment. +Since [`v5.0.0-warmctl.2`](https://github.com/TensorRails/dmtcp/releases/tag/v5.0.0-warmctl.2), +the checkpoint also leaves out resident memory that holds only zeros, and the +readers share large runs of an area that zeros split. On one RTX 3090, +Qwen3-0.6B's process images fell from 8,582 to 6,065 MiB and time to +`/health` from 4.6–5.1 s to 4.0–4.4 s. Pinned CUDA host memory, such as +Qwen3.8-Flash-Next's CPU-side embedding table, already left its zeros out. + The fork is released independently as -[`v5.0.0-warmctl.1`](https://github.com/TensorRails/dmtcp/releases/tag/v5.0.0-warmctl.1), +[`v5.0.0-warmctl.2`](https://github.com/TensorRails/dmtcp/releases/tag/v5.0.0-warmctl.2), with source, Debian 12-compatible Linux amd64 and arm64 packages, checksums and provenance. Both packages include fdkeep. Linux warmctl archives bundle them at `dmtcp/linux-ARCH/VERSION/` beside the CLI; the runtime Dockerfile @@ -88,7 +95,7 @@ versioned DMTCP directories and their mtimes, and keeps older versions for existing snapshots. Use an old snapshot's original runtime image or explicit `--dmtcp-dir` / `--dmtcp-plugin` paths. Publishing a CLI release does not replace an existing runtime image. See the fork's -[release procedure](https://github.com/TensorRails/dmtcp/blob/v5.0.0-warmctl.1/RELEASING.md). +[release procedure](https://github.com/TensorRails/dmtcp/blob/v5.0.0-warmctl.2/RELEASING.md). The packages require glibc >= 2.36, compatible libstdc++ and libatomic (`libstdc++6` and `libatomic1` on Debian/Ubuntu). ARM64 passes @@ -241,7 +248,7 @@ with the restart changes above and these earlier patches: | Accept an already unlinked shm file after restart | Processes that share an area of a deleted `/dev/shm` file (CPython's named semaphores, unlinked at once) recreate one file at restart and each unlinks it again; every one but the first found it gone, and every vLLM restart printed a warning. | | Change a socket's connection only with the fd table locked | `setsockopt`, `connect`, `bind` and `listen` changed the connection they had looked up while another thread could delete it: `add()` drops the entry of an fd closed past the close wrapper once the number comes back. A worker restarted under cuInterpose, whose Rust threads make and close some sockets with raw syscalls, segfaulted in `SocketConnection::addSetsockopt` once in about a dozen restores. | -The plugin [fdkeep](https://github.com/TensorRails/dmtcp/tree/v5.0.0-warmctl.1/plugin/fdkeep), +The plugin [fdkeep](https://github.com/TensorRails/dmtcp/tree/v5.0.0-warmctl.2/plugin/fdkeep), now maintained and released with the fork, repairs what DMTCP loses: - the CUDA driver's eventfds and pipes, recreated as real pipes; diff --git a/internal/dmtcp/SHA256SUMS b/internal/dmtcp/SHA256SUMS index 20faca5..62550a8 100644 --- a/internal/dmtcp/SHA256SUMS +++ b/internal/dmtcp/SHA256SUMS @@ -1,3 +1,3 @@ -496b54b3405ed69dce8d0c3562d644b698150c21a921b9abed3562814724d9eb dmtcp-v5.0.0-warmctl.1-linux-amd64-debian12.tar.gz -90cf76a7733fd2998a29df4e299d9337fdbd0a8c919ccd2db3219c1e263ffd21 dmtcp-v5.0.0-warmctl.1-linux-arm64-debian12.tar.gz -f639b50aa5ecb5b3115f36f67d8ed7ad2894010dcdff43ed4e5a6276f4eeb0b1 dmtcp-v5.0.0-warmctl.1-source.tar.gz +ba19c4c96d257c0c4f78f21a2e2d3bc13f4e895f132ae47c424b67f025ed088d dmtcp-v5.0.0-warmctl.2-linux-amd64-debian12.tar.gz +97cfb934c72e0378698fde7928517b2d93ee1c034c01a725ca7933b7ac8be8b2 dmtcp-v5.0.0-warmctl.2-linux-arm64-debian12.tar.gz +c8d0321d57c1538cc4fa60150715a0d2ea198bcb53b0c58bb65121362d68bad6 dmtcp-v5.0.0-warmctl.2-source.tar.gz diff --git a/internal/dmtcp/VERSION b/internal/dmtcp/VERSION index 9b13a7f..5498c83 100644 --- a/internal/dmtcp/VERSION +++ b/internal/dmtcp/VERSION @@ -1 +1 @@ -v5.0.0-warmctl.1 +v5.0.0-warmctl.2