Repository navigation
Skip resident zero pages; share zero-split areas between restart readers - #4
Merged
Merged
Conversation
The residency scan saved every present page of private anonymous memory, so a pinned buffer larger than its contents or a cleared pool went into the image as zeros and was read back on every restart. On the H200 Qwen3.8-Flash-Next worker a quarter of the process image is such a tail. The writer now also reads each resident run 1 MiB at a time and records an all-zero MiB as a DMTCP_ZERO_PAGE child, the header absent pages already get, so restarts old and new read the image. The check stops at the first nonzero word, so ordinary data costs almost nothing. Resident zeros are not dropped with MADV_DONTNEED: the process may run on, and a page pinned for DMA must keep its frame. DMTCP_DISABLE_ZERO_SCAN=1 writes the old images. Zero runs cut a large area into runs below the 1 GiB a payload needs for the parallel readers, which would then be read by one thread. With readers, restart now records writable payloads of at least 1 MiB and seeks past them; the helpers read them together, every 1024 and at the end of the image, in equal shares of bytes. Smaller and read-only payloads are read at once: deferring thousands of small areas made the header pass seek per area and a cold restart slower. The batch array lives on the restart stack. On a CPU process like that worker (12 GiB with a 25% zero tail, 10,000 small areas; 16 readers, huge pages) the image went from 14.07 to 11.07 GiB and the memory pass from 8.66 to 6.86 s cold, 1.35 to 1.19 s warm. With zeros in every other 64 MiB stripe it went from 7.28 to 3.68 s cold, all runs still read in parallel. Images restart across both versions. Fixes #3
The writer split a resident run into MiBs counted from the run's start, which is wherever the mapping begins or merges with its neighbor. A zero buffer then lost up to a MiB at each end, and CI's layouts put the zeroed bench a MiB over its bound. MiBs now sit on address boundaries; a partial MiB at either end of a run is data, as a short run was. The bench aligns its region to 2 MiB, so every zeroed stripe is whole MiBs and the image bound is exact. ARM64 has no parallel readers, so the zeroed16 check for them runs on x86-64 only.
CI images of a 1025 MiB bench carry about 2 MiB more besides the payload than those of a 257 MiB one, so measuring zeroed16 against the 257 MiB ordinary image failed by that much, though no zero stripe was saved. Each zeroed image is now bounded by the dense image of its own size.
On one RTX 3090, warmctl beta.17 baking and restoring Qwen3-0.6B: process images 8,582 -> 6,065 MiB, the vLLM worker's memory pass 1.05 -> 0.74 s, /health 4.6-5.1 -> 4.0-4.4 s, same greedy answers. Pinned CUDA host memory is a shared mapping of /dev/zero, which the content scan already split into zero runs, so the comments no longer name a CUDA host buffer as what the resident scan finds.
Closed
5 tasks
This was referenced Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The checkpoint writer saved every resident page of private anonymous memory, zeros included, and restart read them back. This records resident all-zero MiBs as
DMTCP_ZERO_PAGEchildren, the header absent pages already get, and keeps the runs between them on the parallel readers. In a vLLM worker restored with cuda-checkpoint it removes 2.4 of 5.6 GiB (below).Correction to #3: pinned CUDA host memory (torch
pin_memory) is a shared mapping of/dev/zero, which the existing content scan already split into zero runs. So the H200 Qwen3.8 worker's pinned PLE table, whose zero tail motivated #3, was already skipped bymain; this change does not shrink that image.Writer. After the residency scan reports a resident run, it is split by content into MiBs on MiB address boundaries (
mtcp_get_next_aligned_range; a partial MiB at either end is data); the check stops at the first nonzero word, so ordinary data costs almost nothing. Resident zeros are notMADV_DONTNEED'd: the process may run on, and a page pinned for DMA must keep its frame. Runs under 1 MiB are saved as before.DMTCP_DISABLE_ZERO_SCAN=1writes the old images byte for byte.Restart. Zero runs cut a large area into runs below the 1 GiB one payload needs for the readers, so they would have been read by one thread. With
DMTCP_RESTART_READERS, writable payloads of at least 1 MiB are now recorded and skipped, and the helpers read them together (every 1024, and at the end of the image) in equal shares of bytes. Smaller and read-only payloads are read at once: a first version that deferred thousands of small areas made the header pass seek per area, and a cold restart got slower (8.66 → 9.36 s). The batch array is on the restart stack, never at an address an area could need. Readers off: nothing changes.Compatibility. No format change. Images from this branch restart under
mainand images frommainunder this branch, with 1 and 16 readers, for dense, sparse, zeroed and many-area processes (16/16).Measured on a 24-thread Xeon VM, local disk, 16 readers + huge pages, memory pass from
DMTCP_RESTART_PROFILE(cold = image evicted withPOSIX_FADV_DONTNEED):In the striped case all 6 GiB were read by the helpers (
parallel_kib), though every run is 64 MiB.Tests.
restore-memory-check.pygainszeroed,zeroed16andmmap-zeroed(the bench clears every other stripe after writing it; the image must hold none of it, every word must return, and with readers the runs must be shared), andmany-areas-1/-16: 2,400 areas, writable ones of at least 1 MiB between small read-only ones, so more payloads than one batch holds and shares that start inside areas. It verifies every byte and each protection.make checkwith 1 and with 16 readers: 75/75 unit tests and the same results asmainon this VM.checkpoint-open-files-alias,checkpoint-open-files-envandsyscall-testerfail in the full run on bothmainand this branch, and pass on both when run alone.GPU. warmctl beta.17
build --in-place --backend dmtcpandupof Qwen3-0.6B on one RTX 3090 (Runpod, 8 readers, huge pages),mainvs this branch through--dmtcp-dir, two cold (images evicted) and two warm restores each:/health, cold/health, warmEvery restore answered the same greedy completion. A standalone process with 6 GiB pinned and 6 GiB pageable memory (zero tails) next to 6 GiB on the GPU, checkpointed after cuda-checkpoint: image 17.43 → 14.18 GiB, memory pass 3.10 → 2.38 s.
Fixes #3