Repository navigation
Release the warmctl fork with parallel large-memory restart - #2
Merged
Merged
Conversation
A restarted listening socket was bound with only IPV6_V6ONLY restored; SO_REUSEADDR came back after bind. Right after the checkpointed process is killed its accepted connections linger in TIME_WAIT on the port, so bind failed with EADDRINUSE and the restarted server listened nowhere. Every option that changes what bind accepts is now set before it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copies of checkpointed files are saved under the image's _files directory, but the restart looked for them at the absolute path of the checkpoint directory. Images copied to another host or directory restarted only if that directory still existed. The checkpoint directory now becomes the restart's before any plugin's restart hook, and a file connection takes its copy from there when it exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g them A checkpoint wrote every file-backed mapping in full: the text and read-only data of each library. A process that loaded torch and the CUDA libraries carries gigabytes of them, which every restart read back. An area that is a private, unwritable mapping within the file's pages, with no page copied on write or swapped out (Util::areaHasPrivatePages: PAGEMAP_SCAN, or pagemap's file bit as a fallback; a library's relocated RELRO keeps its data), is now saved as a header only, flagged DMTCP_FILE_MAPPED with the file's size and mtime. mtcp_restart maps the file again and stops if it is not the one checkpointed. The new header fields sit in the former padding, so images without them still read. DMTCP_DISABLE_FILE_MAPPED=1 saves every file area as before. vLLM's API server and engine core went from 5.4 and 6.9 GB to 1.6 and 2.8 GB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ProcessInfo::resetOnFork() nulls upid until the child's first checkpoint, and dmtcp_get_uniquepid() returned the field as it was: every connection a forked child created before then was named 0-0-0(n), n counting on from the parent's. Two children of one parent, as vLLM's engine core and its worker, gave different sockets the same name; the restart's rewirer handed the one reconnection to one of them and the other waited at Socket::Restart_Ns_Send_Queries forever. dmtcp_get_uniquepid() and upidStr() now go through getUpid(), which fills a null upid with the process's own, reset by UniquePid at the fork. A restart of `vllm serve --distributed-executor-backend mp` (API server, engine core, resource tracker, worker) now completes in 4.6 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without ENABLE_IP6_SUPPORT, registerIncoming files AF_INET6 connections under the IPv4 restore listener, but postRestart opened that listener only for processes with an AF_INET socket. A process whose TCP sockets were all IPv6 (a vLLM tensor-parallel worker: gloo and the TCPStore on [::]) published a zeroed restore address, and its peer failed in doReconnect with connect() == -1. restoreDomain() now names the listener a domain is rewired through, and both registerIncoming and the listener flags use it. registerNSData refuses to publish an unopened listener's address, and the reconnect assert reports errno and the address family.
A socket created with SOCK_NONBLOCK is recreated with the same type, so the rewirer's connect() returned EINPROGRESS and the restart asserted (vLLM's tensor-parallel workers). doReconnect now clears O_NONBLOCK for the connect and the identifier write and sets it back afterwards. A failed rebind now reports errno, the family and the port.
UCX's memory hooks, shipped in the vLLM image for NIXL, test shmat events by calling the original shmget through dlsym(RTLD_NEXT), which skips DMTCP's wrapper, and then removing the segment with the wrapped shmctl. shmctl found no virtual id and asserted, so vLLM's engine core died at startup under dmtcp_launch. An id with no mapping is the kernel's own: shmat and shmctl now use it as is, untracked, instead of asserting.
A vLLM process image compresses about 2.8x with zstd -1 (7.3 to 2.6 GB for Qwen3-8B's four processes), which a restore from network storage otherwise reads in full. With DMTCP_ZSTD=1 the checkpoint pipes through `zstd -1 -T0` the way it pipes through gzip, which it still uses when zstd is off or missing; dmtcp_restart recognizes zstd's magic and decompresses through `zstd -d`. The compressor runs as a separate program, as gzip does, so nothing allocates in a fork of a multithreaded process.
glibc expands $ORIGIN in a dlopen filename to the directory of the object that called dlopen. The dl plugin's wrapper calls the real dlopen itself, so the expansion named libdmtcp.so's directory and a library that loads its parts beside itself found nothing: NVIDIA's cuInterpose loads "$ORIGIN/libcuinterpose_core.so", and under DMTCP every cuInit it intercepted failed with CUDA_ERROR_NOT_INITIALIZED. A filename with a slash is a path, which dlopen(3) opens without a search: open it directly, with $ORIGIN expanded for the caller, instead of trying it under the caller's rpath, LD_LIBRARY_PATH and runpath first. test/dlopen-origin loads a library that dlopens its sibling through $ORIGIN; it fails at launch without this change.
glibc 2.28 and later compile fcntl into fcntl64 when _FILE_OFFSET_BITS
is 64, as CPython is built, so none of such a program's fcntl calls
reached DMTCP's wrapper: a duplicate made with F_DUPFD_CLOEXEC (CPython's
os.dup) was a descriptor DMTCP did not know. The child of
multiprocessing's spawn keeps its parent sentinel only as such a
duplicate, so its DMTCP no longer had the pipe and the parent drained its
end forever at checkpoint ("Still draining socket; perhaps remote host
is not running under DMTCP"). vLLM starts its engine core and workers
with spawn whenever its API server has initialized CUDA.
On 64-bit targets fcntl64 is fcntl: the wrapper passes the argument on
as a pointer-sized value, as glibc's own does.
test/fcntl64-dup: a child that keeps a pipe only through such a
duplicate; its checkpoint never ends without this change.
Processes that shared an area of a deleted /dev/shm file all recreate one file in refill (the first creates it with O_EXCL, the others open it) and each unlinks it again in resume, so every process but the first found it gone and warned. CPython's multiprocessing makes such areas: its named semaphores are unlinked at once and shared by the processes it starts, and every restart of a vLLM engine printed the warning. ENOENT is now expected; other errors still warn, with errno. test/sem-unlinked-shared: two processes ping-pong on two unlinked named semaphores; a restart that gave each process its own file would make the next wait time out.
setsockopt, connect, bind and listen looked up the fd's connection and then changed it, while another thread could delete it: add() drops the entry of an fd closed past the close wrapper when the number comes back, and processClose() drops a closed one. setsockopt held no WrapperLock either. A vLLM worker restarted under cuInterpose, whose Rust threads close some sockets with raw syscalls and set options on others, segfaulted in std::map inside SocketConnection::addSetsockopt. ConnectionList::withConnection runs the change with the table locked; setsockopt now holds the WrapperLock, as the others do, so the checkpoint never suspends a thread holding the table. test/setsockopt-race: one thread sets an option in a loop while another closes the descriptor with a raw syscall and creates a socket at the same number. Before: a segfault within 20 s in 3 runs of 3. After: 20 s and 44-106 million calls in each, and the checkpoint and restart pass.
kbespalov
force-pushed
the
perf/parallel-restart-release
branch
2 times, most recently
from
November 1, 2026 15:56
be904e9 to
9fc657e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large cache-hot CPU process images spend most of restart time allocating and copying 4 KiB pages. This adds opt-in huge-page advice and 2–8/16 parallel raw pread helpers, preserving the ordinary reader as fallback and killing/reaping helpers before resumption or when their parent dies. Per-region/phase profiling explains the critical path. Also fixes restart-environment lookup and private dirty bytes in a regular file’s final page; experimental private mmap remains disabled by default.
Four H200 restores with huge pages + 16 readers reached health in 20.21–23.43 s (median 20.83 s, streaming first-token median 20.92 s), versus 77.36/82.14 s on the same host/snapshot with ordinary reading. In the illustrated pair, DMTCP fell from 54.09 to 4.04 s. Full conditions, cache preparation, all experiment variants and numerical-canary limitations are recorded in doc/h200-restart-performance.md.
This PR promotes the existing shipped
warmctlbranch (12 commits beyondmain) together with the new changes to the fork’s default branch, so releases and issue closure follow one canonical history. The existing patches already underpin warmctl beta.16; their presence here does not represent new H200 optimizations.Validation: 20 H200 restores, matching GPU-weight fingerprints, valid streaming/tool calls and 640 short requests; CPU full-byte verification with dense/sparse memory, dirty-page writes and repeated checkpoints; helper cancellation; old/new partial-file regression. Added Linux CI repeats byte/fallback checks and selected process/thread regressions, then tests the packaged runtime. The numerical canary is still not qualified; donor variation exists before checkpoint.
Adds independent
v5.0.0-warmctl.Nprereleases with source and native Linux amd64/arm64 packages built on a pinned Debian 12 base, SHA-256 checksums and provenance. Both packages include the fdkeep plugin, now maintained in this fork with native raw syscalls and descriptor-restoration tests on both architectures. Packages are also tested after relocation. ARM64 is CPU-qualified only; parallel pread and the H200 GPU results are x86-64-only. Existing snapshots require their original byte-identical libraries; upgrading the runtime requires a fresh bake.Fixes #1