From 19f5fc56ea63d3e06707af2763fd83ff7adf4ce7 Mon Sep 17 00:00:00 2001 From: Kirill Bespalov Date: Wed, 4 Nov 2026 01:24:18 +0300 Subject: [PATCH 1/4] Skip resident zero pages; share zero-split areas between restart readers The residency scan saved every present page of private anonymous memory, so a pinned buffer larger than its contents or a cleared pool went into the image as zeros and was read back on every restart. On the H200 Qwen3.8-Flash-Next worker a quarter of the process image is such a tail. The writer now also reads each resident run 1 MiB at a time and records an all-zero MiB as a DMTCP_ZERO_PAGE child, the header absent pages already get, so restarts old and new read the image. The check stops at the first nonzero word, so ordinary data costs almost nothing. Resident zeros are not dropped with MADV_DONTNEED: the process may run on, and a page pinned for DMA must keep its frame. DMTCP_DISABLE_ZERO_SCAN=1 writes the old images. Zero runs cut a large area into runs below the 1 GiB a payload needs for the parallel readers, which would then be read by one thread. With readers, restart now records writable payloads of at least 1 MiB and seeks past them; the helpers read them together, every 1024 and at the end of the image, in equal shares of bytes. Smaller and read-only payloads are read at once: deferring thousands of small areas made the header pass seek per area and a cold restart slower. The batch array lives on the restart stack. On a CPU process like that worker (12 GiB with a 25% zero tail, 10,000 small areas; 16 readers, huge pages) the image went from 14.07 to 11.07 GiB and the memory pass from 8.66 to 6.86 s cold, 1.35 to 1.19 s warm. With zeros in every other 64 MiB stripe it went from 7.28 to 3.68 s cold, all runs still read in parallel. Images restart across both versions. Fixes #3 --- doc/restart-memory-profiling.md | 54 ++++++++-- doc/warmctl-release-notes.md | 17 +++ src/constants.h | 4 + src/mtcp/mtcp_restart.c | 184 +++++++++++++++++++++++++++----- src/mtcp/mtcp_restart.h | 6 ++ src/writeckpt.cpp | 44 ++++++-- test/restore-many-areas.c | 82 ++++++++++++++ test/restore-memory-bench.c | 13 ++- test/restore-memory-check.py | 75 +++++++++---- 9 files changed, 411 insertions(+), 68 deletions(-) create mode 100644 test/restore-many-areas.c diff --git a/doc/restart-memory-profiling.md b/doc/restart-memory-profiling.md index e0a71887ce..5158b1766f 100644 --- a/doc/restart-memory-profiling.md +++ b/doc/restart-memory-profiling.md @@ -8,7 +8,8 @@ normally. The default reader and checkpoint format stay unchanged. | `DMTCP_RESTART_PROFILE` | `1` | Log memory restoration, large/slow regions, page faults, the restart barrier and plugin phase. | | `DMTCP_RESTART_HUGEPAGES` | `1` | Advise huge pages for eligible large private anonymous mappings before loading their contents. | | `DMTCP_RESTART_MMAP` | `1` | Map eligible raw anonymous payloads of at least 64 MiB directly from the image using private mappings. | -| `DMTCP_RESTART_READERS` | `2`–`8`, `16` | On x86-64, copy raw regular-file payloads of at least 1 GiB with parallel `pread` helpers. Default is one reader. | +| `DMTCP_RESTART_READERS` | `2`–`8`, `16` | On x86-64, copy raw regular-file payloads with parallel `pread` helpers: writable payloads of at least 1 MiB together, in batches, and any other payload of at least 1 GiB on its own. Default is one reader. | +| `DMTCP_DISABLE_ZERO_SCAN` | `1` | At checkpoint, save resident anonymous pages that hold only zeros, as before, instead of recording them as zero runs. | Use one optimization at a time for A/B measurements. Example: @@ -23,8 +24,10 @@ bake a new snapshot before testing its post-restart instrumentation. The MTCP memory reader uses raw syscalls and stores its counters in relocated state. `phase=memory` reports microseconds spent in the memory pass and payload reads, -anonymous mmap setup time, bytes read or file-mapped in KiB, and minor/major -faults. `pid` is the real process PID; `virtual_pid` is the checkpoint PID. +anonymous mmap setup time, bytes read or file-mapped in KiB, minor/major +faults, and `batches`, the number of batched reads. `phase=batch` reports each +batch: its payloads (`segments`), KiB, microseconds and whether the helpers +(`pread`) or the parent alone (`read`) read it. `pid` is the real process PID; `virtual_pid` is the checkpoint PID. `phase=area` reports large or slow payloads, mmap eligibility, offset alignment and mmap errors. An eligible payload with nonzero `offset_mod` falls back to reading. `restart_barrier`, @@ -56,10 +59,21 @@ registration may force private copies and remove some or all of the benefit. Neither experiment is qualified for GPU production solely by CPU tests. Parallel readers can be combined with huge-page advice. Their temporary stacks -are unmapped and every helper is reaped before moving to another memory area or -resuming the application. Helpers share the address space but do not use libc, -TLS or pthreads. Failed helper creation falls back to reading its range in the -parent; failed reads fall back to the ordinary reader after all helpers exit. +are unmapped and every helper is reaped before the next memory area is mapped +or the application resumes. Helpers share the address space but do not use +libc, TLS or pthreads. Failed helper creation falls back to reading its range in +the parent; failed reads fall back to the ordinary reader after all helpers exit. + +With readers, the payload of a writable area of at least 1 MiB is not read +when its header is: the pass records where it goes and seeks past it, and the +helpers read the recorded payloads together, every 1024 of them and at the end +of the image, each helper taking an equal share of the bytes. A zero run cuts +a large area into runs that are smaller than the 1 GiB a single payload needs +for the helpers; batched, they still share them. Read-only areas are read at +once, as they are protected right after, and so are areas under 1 MiB: putting +thousands of them off turns the pass over the headers into a seek per area, +which costs more on a cold or network file than the helpers save. The batch +array lives on the restart stack, never at an address an area could need. Helpers arm a parent-death signal and recheck their parent, so cancellation does not leave them holding the restore's address space. `parallel_kib` identifies actual use. `child_minflt` and `child_majflt` are the @@ -75,3 +89,29 @@ During instrumentation, `dmtcp_get_restart_env()` was found to bound its scan by `sizeof(char *)`, which examined only the first environment entry. The fix uses the actual read length and requires an exact name followed by `=`. The benchmark verifies a later variable whose longer prefixed name occurs first. + +## Resident zero pages + +A process can hold gigabytes of resident zeros: a pinned host buffer sized +larger than what it holds, a pool cleared after use. The page residency scan +saves every resident page. The writer now also reads resident private +anonymous memory 1 MiB at a time and records each all-zero MiB as a zero run, +the header the residency scan already writes for absent pages. A restart maps +the area and reads only the other runs, so zeros cost neither image bytes nor +reads. Restarts that predate this read such images too: the format is the same. + +The check stops at the first nonzero word of each MiB, so ordinary data costs +almost nothing; only zero MiBs are read whole, and they are no longer written. +Resident zeros are left in place (no `MADV_DONTNEED`): the process may run on +after the checkpoint, and a page pinned for DMA must keep its frame. Runs +under 1 MiB are saved as before, so a run never costs more header than data. +`DMTCP_DISABLE_ZERO_SCAN=1` restores the old images byte for byte. + +`test/restore-memory-bench.c` has a `zeroed` layout, which writes every other +64 MiB stripe and then clears it. `test/restore-memory-check.py` checks that +none of those stripes reach the image, that every word comes back, and that +with readers the runs between them are still shared. +`test/restore-many-areas.c` checkpoints 2,400 areas: writable ones of at least +1 MiB between small read-only ones, so restarts batch more payloads than one +batch holds, with shares starting and ending inside areas, and read the +read-only ones at once. It verifies every byte and each protection. diff --git a/doc/warmctl-release-notes.md b/doc/warmctl-release-notes.md index 2d9d86610c..49faf1b4ab 100644 --- a/doc/warmctl-release-notes.md +++ b/doc/warmctl-release-notes.md @@ -1,3 +1,20 @@ +# Unreleased + +- The checkpoint writer records resident all-zero MiBs of private anonymous + memory as zero runs instead of saving them. Images are smaller and restarts + read less; existing restarts read the new images, and + `DMTCP_DISABLE_ZERO_SCAN=1` turns it off. +- With `DMTCP_RESTART_READERS`, writable payloads of at least 1 MiB are read by + the helpers in batches, so a large area cut by zero runs is still read in + parallel. Smaller and read-only payloads are read as before. + +On a CPU test process like the H200 worker (a 12 GiB area with a 25% zero +tail, plus 10,000 small areas), 16 readers and huge pages: the image fell from +14.07 to 11.07 GiB and the checkpoint from 16.5 to 14.3 s; the memory pass from +8.66 to 6.86 s with a cold file cache and from 1.35 to 1.19 s with a warm one. +With zeros in every other 64 MiB stripe of 12 GiB, the image halved and the +pass fell from 7.28 to 3.68 s cold. Not yet measured on a GPU restore. + # v5.0.0-warmctl.1 First versioned TensorRails fork release, based on DMTCP 5.0. It includes the diff --git a/src/constants.h b/src/constants.h index 0a659d5577..eff63e45d9 100644 --- a/src/constants.h +++ b/src/constants.h @@ -148,6 +148,10 @@ // restart. #define ENV_VAR_DISABLE_FILE_MAPPED "DMTCP_DISABLE_FILE_MAPPED" +// Set to "1" to save resident anonymous pages that hold only zeros, as before, +// instead of recording each 1 MB run of them as a zero run with no data. +#define ENV_VAR_DISABLE_ZERO_SCAN "DMTCP_DISABLE_ZERO_SCAN" + // this list should be kept up to date with all "protected" environment vars #define ENV_VARS_ALL \ ENV_VAR_NAME_HOST, \ diff --git a/src/mtcp/mtcp_restart.c b/src/mtcp/mtcp_restart.c index f8667fc341..250ab17159 100644 --- a/src/mtcp/mtcp_restart.c +++ b/src/mtcp/mtcp_restart.c @@ -886,12 +886,34 @@ __asm__(".text\n" "ud2\n" "1: ret\n" ".size mtcp_read_clone,.-mtcp_read_clone\n"); +#endif -typedef struct { - int fd; +// A run of bytes of the image to be read into memory. +typedef struct RestartSegment { char *address; size_t size; off_t offset; +} RestartSegment; + +// Segments are put off until this many are pending, then read together. +#define RESTART_DEFERRED_MAX 1024 + +// Smaller areas are read at once, in order: putting their reads off would +// turn the pass over the headers into a seek per area, which costs more on a +// cold or network file than the readers save. +#define RESTART_DEFER_MIN (1024UL * 1024) + +// Below this many bytes, starting readers costs more than it saves. +#define RESTART_PARALLEL_MIN (64UL * 1024 * 1024) + +// One reader's share: size bytes of the segments, from byte skip of +// segments[first] on. +typedef struct { + int fd; + const RestartSegment *segments; + size_t first; + size_t skip; + size_t size; volatile int result; pid_t parent; } RestartReadJob; @@ -901,19 +923,27 @@ restart_read_job(void *raw) { int mtcp_sys_errno; RestartReadJob *job = (RestartReadJob *) raw; - size_t done = 0; - while (done < job->size) { - size_t chunk = job->size - done; + size_t i = job->first, skip = job->skip, left = job->size; + while (left > 0) { + const RestartSegment *s = &job->segments[i]; + size_t chunk = s->size - skip; + if (chunk > left) chunk = left; if (chunk > 64 * 1024 * 1024) chunk = 64 * 1024 * 1024; - long n = mtcp_inline_syscall(pread64, 4, job->fd, job->address + done, - chunk, job->offset + done); + long n = mtcp_inline_syscall(pread64, 4, job->fd, s->address + skip, + chunk, s->offset + skip); if (n < 0 && mtcp_sys_errno == EINTR) continue; if (n <= 0) return job->result = 1; - done += n; + skip += n; + left -= n; + if (skip == s->size) { + i++; + skip = 0; + } } return job->result = 0; } +#if defined(__x86_64__) static int restart_read_child(void *raw) { @@ -927,15 +957,16 @@ restart_read_child(void *raw) } #endif +// Reads total bytes of the segments with the readers, each taking an equal +// share of the bytes whatever the segments' sizes. Returns 0, having changed +// nothing but memory, if they could not; the fd's offset is never moved. static int -restart_parallel_read(RestoreInfo *rinfo, int fd, void *address, size_t size) +restart_read_segments(RestoreInfo *rinfo, int fd, const RestartSegment *segments, + size_t count, uint64_t total) { #if defined(__x86_64__) int mtcp_sys_errno; - if (rinfo->restartReaders <= 1 || !rinfo->restartImageRegular || - size < 1024UL * 1024 * 1024) return 0; - off_t offset = mtcp_sys_lseek(fd, 0, SEEK_CUR); - if (offset < 0) return 0; + if (rinfo->restartReaders <= 1 || total < RESTART_PARALLEL_MIN) return 0; const size_t stackSize = 64 * 1024; int readers = rinfo->restartReaders; char *stacks = mtcp_sys_mmap(NULL, stackSize * (readers - 1), @@ -945,18 +976,29 @@ restart_parallel_read(RestoreInfo *rinfo, int fd, void *address, size_t size) RestartReadJob jobs[16]; long pids[16]; const size_t huge = 2 * 1024 * 1024; - size_t stride = ((size + readers - 1) / readers + huge - 1) & ~(huge - 1); + size_t stride = ((total + readers - 1) / readers + huge - 1) & ~(huge - 1); pid_t parent = mtcp_sys_getpid(); + size_t seg = 0, skip = 0; // where the next share starts for (int i = 0; i < readers; i++) { - size_t start = stride * i; - size_t length = size > start ? size - start : 0; + uint64_t start = (uint64_t) stride * i; + size_t length = total > start ? total - start : 0; if (length > stride) length = stride; - jobs[i] = (RestartReadJob){fd, (char *) address + start, - length, offset + start, -1, parent}; - pids[i] = i + 1 < readers ? + jobs[i] = (RestartReadJob){fd, segments, seg, skip, length, -1, parent}; + for (size_t left = length; left > 0;) { + size_t rest = segments[seg].size - skip; + if (rest > left) { + skip += left; + break; + } + left -= rest; + seg++; + skip = 0; + } + pids[i] = i + 1 < readers && length > 0 ? mtcp_read_clone(restart_read_child, stacks + stackSize * (i + 1), &jobs[i]) : -1; } - // Failed clone calls fall back to doing that helper's range in the parent. + MTCP_ASSERT(seg == count && skip == 0); + // Failed clone calls fall back to doing that helper's share in the parent. for (int i = 0; i < readers; i++) { if (pids[i] < 0) restart_read_job(&jobs[i]); } @@ -978,12 +1020,89 @@ restart_parallel_read(RestoreInfo *rinfo, int fd, void *address, size_t size) if (jobs[i].result != 0) ok = 0; } MTCP_ASSERT(mtcp_sys_munmap(stacks, stackSize * (readers - 1)) == 0); - if (!ok) return 0; // pread did not move fd: the ordinary reader can retry. + if (ok) rinfo->profileParallelBytes += total; + return ok; +#else + return 0; +#endif +} + +// Reads one large area at the fd's offset with the readers and moves the +// offset past it. Returns 0, leaving the offset, if it did not. +static int +restart_parallel_read(RestoreInfo *rinfo, int fd, void *address, size_t size) +{ + int mtcp_sys_errno; + if (!rinfo->restartImageRegular || size < 1024UL * 1024 * 1024) return 0; + off_t offset = mtcp_sys_lseek(fd, 0, SEEK_CUR); + if (offset < 0) return 0; + RestartSegment segment = {(char *) address, size, offset}; + if (!restart_read_segments(rinfo, fd, &segment, 1, size)) return 0; MTCP_ASSERT(mtcp_sys_lseek(fd, offset + size, SEEK_SET) == offset + size); - rinfo->profileParallelBytes += size; + return 1; +} + +// Reads every deferred segment: with the readers when there are enough bytes, +// else (or if they fail) in this thread. An image that ends early is fatal, +// as it is to mtcp_readfile(). +static void +restart_flush_reads(RestoreInfo *rinfo, int fd) +{ + int mtcp_sys_errno; + if (rinfo->deferredCount == 0) return; + uint64_t start = rinfo->profileRestart ? restart_time_us() : 0; + uint64_t total = rinfo->deferredBytes; + int parallel = restart_read_segments(rinfo, fd, rinfo->deferred, + rinfo->deferredCount, total); + if (!parallel) { + RestartReadJob job = {fd, rinfo->deferred, 0, 0, total, -1, 0}; + if (restart_read_job(&job) != 0) { + MTCP_PRINTF("error %d reading %u deferred segments (%p bytes) of the " + "checkpoint image\n", mtcp_sys_errno, + (unsigned) rinfo->deferredCount, (void *) total); + mtcp_abort(); + } + } + rinfo->profileBatches++; + if (rinfo->profileRestart) { + uint64_t us = restart_time_us() - start; + rinfo->profileReadUs += us; + rinfo->profileReadBytes += total; + MTCP_PRINTF("DMTCP_PROFILE pid=%d virtual_pid=%d phase=batch segments=%u " + "read_kib=%u read_us=%u mode=%s\n", mtcp_sys_getpid(), + rinfo->ckptHdr.pid, (unsigned) rinfo->deferredCount, + (unsigned) (total / 1024), (unsigned) us, + parallel ? "pread" : "read"); + } + rinfo->deferredCount = 0; + rinfo->deferredBytes = 0; +} + +// Puts off reading size bytes at the fd's offset into address, moving the +// offset past them, so the readers can share them with other large runs: +// zero runs cut a large area into runs too small to share on their own. +// Only into memory that stays writable, since a read-only area is protected +// as soon as it is read. Returns 0 if the bytes must be read now. +static int +restart_defer_read(RestoreInfo *rinfo, int fd, void *address, size_t size, + int prot) +{ +#if defined(__x86_64__) + int mtcp_sys_errno; + if (rinfo->restartReaders <= 1 || !rinfo->restartImageRegular || + !(prot & PROT_WRITE) || size < RESTART_DEFER_MIN) return 0; + off_t offset = mtcp_sys_lseek(fd, 0, SEEK_CUR); + if (offset < 0) return 0; + if (rinfo->deferredCount == RESTART_DEFERRED_MAX) { + restart_flush_reads(rinfo, fd); + } + rinfo->deferred[rinfo->deferredCount++] = + (RestartSegment){(char *) address, size, offset}; + rinfo->deferredBytes += size; + MTCP_ASSERT(mtcp_sys_lseek(fd, size, SEEK_CUR) == offset + (off_t) size); return 1; #else - return 0; + return 0; // there are no readers to share them #endif } @@ -999,6 +1118,9 @@ mtcp_readmemoryareas(RestoreInfo *rinfo, int fd, VA endOfStack) rinfo->profileFilemapBytes = rinfo->profileFilemapUs = 0; rinfo->profileParallelBytes = rinfo->profileChildMinorFaults = 0; rinfo->profileChildMajorFaults = 0; + RestartSegment deferred[RESTART_DEFERRED_MAX]; + rinfo->deferred = deferred; + rinfo->deferredCount = rinfo->deferredBytes = rinfo->profileBatches = 0; rinfo->mmapParentStart = rinfo->mmapParentEnd = 0; struct stat imageStat; rinfo->restartImageRegular = (rinfo->restartMmap || rinfo->restartReaders > 1) && @@ -1012,6 +1134,8 @@ mtcp_readmemoryareas(RestoreInfo *rinfo, int fd, VA endOfStack) break; /* error */ } } + restart_flush_reads(rinfo, fd); + rinfo->deferred = NULL; if (rinfo->profileRestart) { uint64_t elapsed = restart_time_us() - start; usage_ok = usage_ok && @@ -1020,7 +1144,7 @@ mtcp_readmemoryareas(RestoreInfo *rinfo, int fd, VA endOfStack) "map_us=%u read_kib=%u areas=%u minflt=%u majflt=%u " "usage_ok=%u huge_advice=%u huge_failed=%u " "filemap_kib=%u filemap_us=%u readers=%u parallel_kib=%u " - "child_minflt=%u child_majflt=%u\n", + "child_minflt=%u child_majflt=%u batches=%u\n", mtcp_sys_getpid(), rinfo->ckptHdr.pid, (unsigned) elapsed, (unsigned) rinfo->profileReadUs, (unsigned) rinfo->profileMapUs, (unsigned) (rinfo->profileReadBytes / 1024), rinfo->profileAreas, @@ -1031,7 +1155,8 @@ mtcp_readmemoryareas(RestoreInfo *rinfo, int fd, VA endOfStack) (unsigned) rinfo->profileFilemapUs, rinfo->restartReaders, (unsigned) (rinfo->profileParallelBytes / 1024), (unsigned) rinfo->profileChildMinorFaults, - (unsigned) rinfo->profileChildMajorFaults); + (unsigned) rinfo->profileChildMajorFaults, + rinfo->profileBatches); } #if defined(__arm__) || defined(__aarch64__) @@ -1348,12 +1473,15 @@ read_one_memory_area(RestoreInfo *rinfo, int fd, VA endOfStack) } } } - int parallelRead = !mappedImage && + int deferredRead = !mappedImage && + restart_defer_read(rinfo, fd, area.addr, readSize, area.prot); + int parallelRead = !mappedImage && !deferredRead && restart_parallel_read(rinfo, fd, area.addr, readSize); - if (!mappedImage && !parallelRead) { + if (!mappedImage && !deferredRead && !parallelRead) { mtcp_readfile(fd, area.addr, readSize); } - if (rinfo->profileRestart) { + // A deferred read is timed and counted with its batch. + if (rinfo->profileRestart && !deferredRead) { uint64_t us = restart_time_us() - readStart; if (mappedImage) { rinfo->profileFilemapUs += us; diff --git a/src/mtcp/mtcp_restart.h b/src/mtcp/mtcp_restart.h index 24ed7f591e..cc4fe214f8 100644 --- a/src/mtcp/mtcp_restart.h +++ b/src/mtcp/mtcp_restart.h @@ -155,6 +155,12 @@ typedef struct RestoreInfo { uint64_t profileParallelBytes; uint64_t profileChildMinorFaults; uint64_t profileChildMajorFaults; + // Reads put off to be done together by the readers (restart_defer_read()). + // The array lives in mtcp_readmemoryareas()'s frame. + struct RestartSegment *deferred; + size_t deferredCount; + uint64_t deferredBytes; + unsigned profileBatches; } RestoreInfo; // int mtcp_restart_process_args(RestoreInfo *rinfo, int argc, char *argv[], char **environ); diff --git a/src/writeckpt.cpp b/src/writeckpt.cpp index 7718cc6068..b00776223d 100644 --- a/src/writeckpt.cpp +++ b/src/writeckpt.cpp @@ -332,8 +332,12 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) // - zero runs are found by page residency (Util::scanOccupiedRangeBatch, // via PAGEMAP_SCAN/pagemap) for anonymous memory, or by reading contents // (mtcp_get_next_page_range -> Util::areZeroPages) for deleted-file maps; + // - resident anonymous runs are read too, 1 MB at a time: a process may + // hold gigabytes of zeroed buffers (a CUDA host buffer, a cleared pool), + // and those are zero runs as well; // - a zero run is saved as a DMTCP_ZERO_PAGE header with no data, then - // dropped from RSS with madvise(MADV_DONTNEED); non-zero runs save bytes. + // dropped from RSS with madvise(MADV_DONTNEED), unless it is resident; + // non-zero runs save bytes. // On restart the parent header re-mmaps the whole region as fresh anonymous // memory (zero-fill-on-demand: zero runs are left unallocated, not physically // zeroed) and only the non-zero child runs are read back in. @@ -355,6 +359,11 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) // content scan. The caller decides this (before any reclassification) and // passes it in as residencyScanSafe. bool useResidencyScan = residencyScanSafe; + bool scanResident = useResidencyScan && + !Util::readBooleanEnv(ENV_VAR_DISABLE_ZERO_SCAN, false); + // End of the resident run the residency scan reported last: its contents + // are split into zero and non-zero runs without scanning its pages again. + uintptr_t residentEnd = 0; // Addresses advance monotonically and the list is sorted, so re-query only // after passing the current range. @@ -366,6 +375,7 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) while (area.size > 0) { size_t size; int is_zero; + bool resident = false; Area a = area; const uintptr_t addr = (uintptr_t)area.addr; const uintptr_t areaEnd = addr + area.size; @@ -404,15 +414,25 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) size = chunkSize; is_zero = 0; } else if (useResidencyScan) { - uintptr_t scanned = 0; - is_zero = Util::scanOccupiedRangeBatch(addr, chunkEnd, &scanned); - size = scanned; - if (size == 0) { - // Defensive: the scan should never report zero progress, but if it - // somehow does, treat the remaining chunk as occupied so the loop below - // (area.size -= size) advances instead of spinning forever. - size = chunkSize; - is_zero = 0; + if (addr >= residentEnd) { + uintptr_t scanned = 0; + is_zero = Util::scanOccupiedRangeBatch(addr, chunkEnd, &scanned); + size = scanned; + if (size == 0) { + // Defensive: the scan should never report zero progress, but if it + // somehow does, treat the remaining chunk as occupied so the loop + // below (area.size -= size) advances instead of spinning forever. + size = chunkSize; + is_zero = 0; + } + if (!is_zero && scanResident) { + residentEnd = addr + size; + } + } + if (addr < residentEnd) { + a.size = MIN(residentEnd, chunkEnd) - addr; + mtcp_get_next_page_range(&a, &size, &is_zero); + resident = true; } } else { a.size = chunkSize; @@ -433,7 +453,9 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) if (!is_zero) { writeCkptAll(fd, a.addr, a.size, "anonymous page data"); - } else { + } else if (!resident) { + // Resident zeros stay: the process may run on after the checkpoint, and + // a page pinned for DMA (cudaHostRegister) must keep its frame. if (madvise(a.addr, a.size, MADV_DONTNEED) == -1) { TRACE("error doing madvise(..., MADV_DONTNEED): errno={} addr={} " "size={}", diff --git a/test/restore-many-areas.c b/test/restore-many-areas.c new file mode 100644 index 0000000000..6d4824bc19 --- /dev/null +++ b/test/restore-many-areas.c @@ -0,0 +1,82 @@ +/* Many areas, writable ones of 1 MiB and more between small read-only ones, + * survive a restart. + * gcc -O2 -Iinclude test/restore-many-areas.c -o test/restore-many-areas + * Launch under dmtcp_launch --no-gzip; it checkpoints once and exits. + * Restart reads the writable areas' bytes in batches, across areas and in + * shares that start and end inside them, and the read-only ones at once; on + * restart this verifies every byte of both and each area's protection. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include "dmtcp.h" + +/* More writable areas than restart defers at once (RESTART_DEFERRED_MAX). */ +#define AREAS 2400 + +/* Writable areas are at least the 1 MiB restart defers (RESTART_DEFER_MIN). */ +static size_t pages(size_t i) { return (i % 2 ? 1 : 256) + (i * 7919) % 23; } + +static uint64_t word(size_t area, size_t i) +{ + return ((area + 1) * UINT64_C(0x9e3779b97f4a7c15)) ^ (i * UINT64_C(0xbf58476d1ce4e5b9)); +} + +int main(void) +{ + size_t page = sysconf(_SC_PAGESIZE), total = 0; + for (size_t i = 0; i < AREAS; i++) total += pages(i) * page; + /* One mapping, then alternate protections, so no two neighbors merge. */ + char *base = mmap(NULL, total, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (base == MAP_FAILED) { perror("mmap"); return 2; } + char *p = base; + for (size_t i = 0; i < AREAS; i++) { + uint64_t *w = (uint64_t *) p; + size_t n = pages(i) * page / 8; + for (size_t j = 0; j < n; j++) w[j] = word(i, j); + if (i % 2 && mprotect(p, pages(i) * page, PROT_READ)) { perror("mprotect"); return 2; } + p += pages(i) * page; + } + printf("MANY_CHECKPOINT areas=%d bytes=%zu\n", AREAS, total); + fflush(stdout); + int result = dmtcp_checkpoint(); + if (result == DMTCP_NOT_PRESENT) return 3; + if (result == DMTCP_AFTER_CHECKPOINT) return 0; + if (result != DMTCP_AFTER_RESTART) return 4; + p = base; + for (size_t i = 0; i < AREAS; i++) { + uint64_t *w = (uint64_t *) p; + size_t n = pages(i) * page / 8; + for (size_t j = 0; j < n; j++) { + if (w[j] != word(i, j)) { + fprintf(stderr, "MANY_CORRUPT area=%zu word=%zu\n", i, j); + return 5; + } + } + p += pages(i) * page; + } + /* Each odd area must be read-only again, and each even one writable. */ + FILE *maps = fopen("/proc/self/maps", "r"); + if (!maps) return 6; + char line[512]; + size_t readonly = 0, writable = 0; + while (fgets(line, sizeof(line), maps)) { + unsigned long start, end; + char perms[5]; + if (sscanf(line, "%lx-%lx %4s", &start, &end, perms) != 3) continue; + if (start < (unsigned long) base || end > (unsigned long) (base + total)) continue; + if (perms[1] == 'w') writable++; else readonly++; + } + fclose(maps); + if (readonly != AREAS / 2 || writable != AREAS / 2) { + fprintf(stderr, "MANY_PROT readonly=%zu writable=%zu\n", readonly, writable); + return 7; + } + puts("MANY_VERIFIED"); + fflush(stdout); + return 0; +} diff --git a/test/restore-memory-bench.c b/test/restore-memory-bench.c index 8f3c91febf..ae52a94496 100644 --- a/test/restore-memory-bench.c +++ b/test/restore-memory-bench.c @@ -32,6 +32,9 @@ int main(int argc, char **argv) { size_t mib = argc > 1 ? strtoull(argv[1], NULL, 10) : 256; int sparse = argc > 2 && strcmp(argv[2], "sparse") == 0; + /* zeroed: the odd stripes are written, then cleared, so they are resident + * pages of zeros rather than holes. */ + int zeroed = argc > 2 && strcmp(argv[2], "zeroed") == 0; int cow = argc > 3 && strcmp(argv[3], "cow") == 0; const size_t stripeWords = 64 * 1024 * 1024 / 8; if (!mib || mib > 65536) return 2; @@ -44,7 +47,11 @@ int main(int argc, char **argv) for (size_t i = 0; i < bytes / 8; ++i) { if (!sparse || (i / stripeWords) % 2 == 0) p[i] = word(i); } - printf("BENCH_CHECKPOINT mib=%zu sparse=%d\n", mib, sparse); + for (size_t i = stripeWords; zeroed && i < bytes / 8; i += 2 * stripeWords) { + size_t n = bytes / 8 - i < stripeWords ? bytes / 8 - i : stripeWords; + memset(p + i, 0, n * 8); + } + printf("BENCH_CHECKPOINT mib=%zu sparse=%d zeroed=%d\n", mib, sparse, zeroed); fflush(stdout); int result = dmtcp_checkpoint(); if (result == DMTCP_NOT_PRESENT) return 3; @@ -64,7 +71,7 @@ int main(int argc, char **argv) printf("BENCH_RESUMED monotonic=%.6f mib=%zu\n", start, mib); fflush(stdout); for (size_t i = 0; i < bytes / 8; ++i) { - uint64_t expected = sparse && (i / stripeWords) % 2 ? 0 : word(i); + uint64_t expected = (sparse || zeroed) && (i / stripeWords) % 2 ? 0 : word(i); if (p[i] != expected) { fprintf(stderr, "BENCH_CORRUPT word=%zu got=%" PRIx64 " expected=%" PRIx64 "\n", i, p[i], expected); @@ -96,7 +103,7 @@ int main(int argc, char **argv) if (secondResult == DMTCP_AFTER_CHECKPOINT) return 0; if (secondResult != DMTCP_AFTER_RESTART) return 8; for (size_t i = 0; i < bytes / 8; ++i) { - uint64_t expected = sparse && (i / stripeWords) % 2 ? 0 : word(i); + uint64_t expected = (sparse || zeroed) && (i / stripeWords) % 2 ? 0 : word(i); if (i % (4096 / 8) == 0) expected ^= UINT64_C(0xfedcba9876543210); if (p[i] != expected) return 9; } diff --git a/test/restore-memory-check.py b/test/restore-memory-check.py index d2adec5d62..20aba7fb43 100644 --- a/test/restore-memory-check.py +++ b/test/restore-memory-check.py @@ -81,7 +81,8 @@ def main(): root, output = args.root.resolve(), args.output.resolve() output.mkdir(parents=True, exist_ok=False) source = Path(__file__).resolve().parent - for fixture in ('restore-memory-bench', 'restore-partial-file', 'restore-fdkeep'): + for fixture in ('restore-memory-bench', 'restore-partial-file', 'restore-fdkeep', + 'restore-many-areas'): subprocess.run(['gcc', '-O2', '-I' + str(source.parent / 'include'), str(source / (fixture + '.c')), '-o', str(output / fixture)], check=True) # Last entries exercise both the old scan-length bug and prefix matching. @@ -92,16 +93,21 @@ def main(): DMTCP_RESTART_READERS='1', DMTCP_RESTART_HUGEPAGES='0', DMTCP_BENCH_RESTART_ENV_SUFFIX='wrong', DMTCP_BENCH_RESTART_ENV='donor') # 1025 MiB crosses the parallel-reader threshold with an uneven tail. - cases = [('ordinary', 257, False, 1, False, False, False), - ('parallel2', 1025, False, 2, False, False, False), - ('parallel4', 1025, False, 4, True, False, False), - ('parallel8', 1025, False, 8, True, False, False), - ('parallel16', 1025, False, 16, True, False, False), - ('sparse', 257, True, 16, True, False, False), - ('mmap', 257, False, 1, False, True, False), - ('mmap-sparse', 257, True, 1, False, True, False), - ('gzip-fallback', 1025, False, 16, True, True, True)] - for name, mib, sparse, readers, huge, mapped, compressed in cases: + # zeroed: resident stripes of zeros, which the image keeps as zero runs. + cases = [('ordinary', 257, 'dense', 1, False, False, False), + ('parallel2', 1025, 'dense', 2, False, False, False), + ('parallel4', 1025, 'dense', 4, True, False, False), + ('parallel8', 1025, 'dense', 8, True, False, False), + ('parallel16', 1025, 'dense', 16, True, False, False), + ('sparse', 257, 'sparse', 16, True, False, False), + ('zeroed', 257, 'zeroed', 1, False, False, False), + ('zeroed16', 1025, 'zeroed', 16, True, False, False), + ('mmap', 257, 'dense', 1, False, True, False), + ('mmap-sparse', 257, 'sparse', 1, False, True, False), + ('mmap-zeroed', 257, 'zeroed', 1, False, True, False), + ('gzip-fallback', 1025, 'dense', 16, True, True, True)] + stored = {} + for name, mib, layout, readers, huge, mapped, compressed in cases: case = output / name case.mkdir() first, second, tmp = [case / n for n in ('first', 'second', 'tmp')] @@ -110,28 +116,40 @@ def main(): env.update(DMTCP_TMPDIR=str(tmp), DMTCP_BENCH_SECOND_DIR=str(second), DMTCP_BENCH_RESTART_ENV='donor', DMTCP_GZIP='0') run(root, case / 'checkpoint.log', ['dmtcp_launch', '--no-gzip', '--ckptdir', str(first), - str(output / 'restore-memory-bench'), str(mib), - 'sparse' if sparse else 'dense', 'cow'], env) + str(output / 'restore-memory-bench'), str(mib), layout, 'cow'], env) files = images(first) + stored[name] = sum(os.path.getsize(f) for f in files) + if layout == 'zeroed': + # The bench clears every odd 64 MiB stripe; none of it may be saved. + # Beyond the payload, the image holds what the ordinary one does. + other = stored['ordinary'] - 257 * 2**20 + zero = sum(min(64, mib - s) for s in range(64, mib, 128)) * 2**20 + limit = mib * 2**20 - zero + other + 2**20 + assert stored[name] <= limit, \ + f'{name}: {stored[name]} bytes stored, at most {limit} expected' if compressed: # ARM64's launcher deliberately disables gzip at checkpoint due to # an upstream SIGRETURN limitation. Compress our owned raw fixture # externally to test the restart pipe/fallback on both platforms. - for name in files: - zipped = name + '.gz.tmp' - with open(name, 'rb') as src, gzip.open(zipped, 'wb', compresslevel=1) as dst: + for image in files: + zipped = image + '.gz.tmp' + with open(image, 'rb') as src, gzip.open(zipped, 'wb', compresslevel=1) as dst: shutil.copyfileobj(src, dst, 8 * 1024 * 1024) - os.replace(zipped, name) - with open(name, 'rb') as f: + os.replace(zipped, image) + with open(image, 'rb') as f: assert f.read(2) == b'\x1f\x8b', 'gzip case must use a compressed image' before = digests(files) env.update(DMTCP_BENCH_RESTART_ENV='receiver', DMTCP_RESTART_READERS=str(readers), DMTCP_RESTART_HUGEPAGES=str(int(huge)), DMTCP_RESTART_MMAP=str(int(mapped))) text = run(root, case / 'restart.log', ['dmtcp_restart', *files], env) assert 'BENCH_VERIFIED' in text, text[-6000:] + if layout == 'zeroed' and readers > 1 and mib >= 1024 and not mapped: + # Zero runs cut the area into 64 MiB runs: read together, not one by one. + assert any(map(int, re.findall(r'\bparallel_kib=(\d+)', text))), \ + f'{name}: the readers did not share the runs between zero runs' field = 'filemap_kib' if mapped else 'parallel_kib' values = list(map(int, re.findall(r'\b' + field + r'=(\d+)', text))) - parallel = platform.machine() == 'x86_64' and readers > 1 and mib >= 1024 and not sparse + parallel = platform.machine() == 'x86_64' and readers > 1 and mib >= 1024 and layout == 'dense' if not compressed and (mapped or parallel): assert any(values), f'{name}: optimized reader was not exercised' if platform.machine() != 'x86_64' and not mapped: @@ -145,6 +163,25 @@ def main(): for d in (first, second, tmp): shutil.rmtree(d) print(name + ': PASS (all words, dirty pages, second checkpoint, source SHA-256)', flush=True) + # More writable areas than restart defers at once, between read-only ones. + for readers in (1, 16): + case = output / f'many-areas-{readers}' + case.mkdir() + env.update(DMTCP_TMPDIR=str(case), DMTCP_RESTART_MMAP='0', DMTCP_GZIP='0', + DMTCP_RESTART_READERS=str(readers), DMTCP_RESTART_HUGEPAGES='0') + run(root, case / 'checkpoint.log', ['dmtcp_launch', '--no-gzip', '--ckptdir', + str(case), str(output / 'restore-many-areas')], env) + text = run(root, case / 'restart.log', ['dmtcp_restart', *images(case)], env) + assert 'MANY_VERIFIED' in text, text[-6000:] + batches = sum(map(int, re.findall(r'\bbatches=(\d+)', text))) + parallel = sum(map(int, re.findall(r'\bparallel_kib=(\d+)', text))) + if readers == 1: + assert batches == 0 and parallel == 0, f'one reader deferred reads: {text[-3000:]}' + elif platform.machine() == 'x86_64': + assert batches >= 2 and parallel > 0, f'deferred reads were not shared: {text[-3000:]}' + for f in case.glob('ckpt_*.dmtcp'): + f.unlink() + print(f'many-areas-{readers}: PASS (every byte, protections, {batches} batches)', flush=True) case = output / 'partial-file' case.mkdir() env.update(DMTCP_TMPDIR=str(case), DMTCP_RESTART_MMAP='0', DMTCP_GZIP='0') From 994630442e79d784dacd7713a9c292cfda06d5f8 Mon Sep 17 00:00:00 2001 From: Kirill Bespalov Date: Wed, 4 Nov 2026 07:59:22 +0300 Subject: [PATCH 2/4] Find zero MiBs on MiB boundaries; check parallel use only on x86-64 The writer split a resident run into MiBs counted from the run's start, which is wherever the mapping begins or merges with its neighbor. A zero buffer then lost up to a MiB at each end, and CI's layouts put the zeroed bench a MiB over its bound. MiBs now sit on address boundaries; a partial MiB at either end of a run is data, as a short run was. The bench aligns its region to 2 MiB, so every zeroed stripe is whole MiBs and the image bound is exact. ARM64 has no parallel readers, so the zeroed16 check for them runs on x86-64 only. --- doc/restart-memory-profiling.md | 8 +++++--- src/writeckpt.cpp | 36 +++++++++++++++++++++++++++++---- test/restore-memory-bench.c | 12 ++++++++--- test/restore-memory-check.py | 3 ++- 4 files changed, 48 insertions(+), 11 deletions(-) diff --git a/doc/restart-memory-profiling.md b/doc/restart-memory-profiling.md index 5158b1766f..6a95b83160 100644 --- a/doc/restart-memory-profiling.md +++ b/doc/restart-memory-profiling.md @@ -95,7 +95,8 @@ benchmark verifies a later variable whose longer prefixed name occurs first. A process can hold gigabytes of resident zeros: a pinned host buffer sized larger than what it holds, a pool cleared after use. The page residency scan saves every resident page. The writer now also reads resident private -anonymous memory 1 MiB at a time and records each all-zero MiB as a zero run, +anonymous memory a MiB at a time, on MiB address boundaries, and records each +all-zero MiB as a zero run, the header the residency scan already writes for absent pages. A restart maps the area and reads only the other runs, so zeros cost neither image bytes nor reads. Restarts that predate this read such images too: the format is the same. @@ -103,8 +104,9 @@ reads. Restarts that predate this read such images too: the format is the same. The check stops at the first nonzero word of each MiB, so ordinary data costs almost nothing; only zero MiBs are read whole, and they are no longer written. Resident zeros are left in place (no `MADV_DONTNEED`): the process may run on -after the checkpoint, and a page pinned for DMA must keep its frame. Runs -under 1 MiB are saved as before, so a run never costs more header than data. +after the checkpoint, and a page pinned for DMA must keep its frame. Partial +MiBs at the ends of a run are saved as before, so a zero run is never shorter +than a MiB and never costs more header than data. `DMTCP_DISABLE_ZERO_SCAN=1` restores the old images byte for byte. `test/restore-memory-bench.c` has a `zeroed` layout, which writes every other diff --git a/src/writeckpt.cpp b/src/writeckpt.cpp index b00776223d..1dc6b41b1e 100644 --- a/src/writeckpt.cpp +++ b/src/writeckpt.cpp @@ -324,6 +324,33 @@ mtcp_get_next_page_range(Area *area, size_t *size, int *is_zero) } } +// Like mtcp_get_next_page_range(), but in MBs on MB boundaries, so a zero MB +// is found wherever the area starts. A part of an MB, at either end, counts +// as data. +static void +mtcp_get_next_aligned_range(Area *area, size_t *size, int *is_zero) +{ + const uintptr_t one_MB = 1024 * 1024; + const uintptr_t start = (uintptr_t)area->addr; + const uintptr_t end = start + area->size; + uintptr_t pos = start; + int zero = -1; + + while (pos < end) { + uintptr_t next = MIN(end, (pos + one_MB) & ~(one_MB - 1)); + int z = next - pos == one_MB && + Util::areZeroPages((void *)pos, one_MB / MTCP_PAGE_SIZE); + if (zero < 0) { + zero = z; + } else if (z != zero) { + break; + } + pos = next; + } + *size = pos - start; + *is_zero = zero > 0; +} + static void mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) { @@ -332,9 +359,10 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) // - zero runs are found by page residency (Util::scanOccupiedRangeBatch, // via PAGEMAP_SCAN/pagemap) for anonymous memory, or by reading contents // (mtcp_get_next_page_range -> Util::areZeroPages) for deleted-file maps; - // - resident anonymous runs are read too, 1 MB at a time: a process may - // hold gigabytes of zeroed buffers (a CUDA host buffer, a cleared pool), - // and those are zero runs as well; + // - resident anonymous runs are read too, an aligned MB at a time + // (mtcp_get_next_aligned_range): a process may hold gigabytes of + // zeroed buffers (a CUDA host buffer, a cleared pool), and those are + // zero runs as well; // - a zero run is saved as a DMTCP_ZERO_PAGE header with no data, then // dropped from RSS with madvise(MADV_DONTNEED), unless it is resident; // non-zero runs save bytes. @@ -431,7 +459,7 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) } if (addr < residentEnd) { a.size = MIN(residentEnd, chunkEnd) - addr; - mtcp_get_next_page_range(&a, &size, &is_zero); + mtcp_get_next_aligned_range(&a, &size, &is_zero); resident = true; } } else { diff --git a/test/restore-memory-bench.c b/test/restore-memory-bench.c index ae52a94496..a62d60aa57 100644 --- a/test/restore-memory-bench.c +++ b/test/restore-memory-bench.c @@ -39,9 +39,15 @@ int main(int argc, char **argv) const size_t stripeWords = 64 * 1024 * 1024 / 8; if (!mib || mib > 65536) return 2; size_t bytes = mib * 1024 * 1024; - uint64_t *p = mmap(NULL, bytes, PROT_READ | PROT_WRITE, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - if (p == MAP_FAILED) { perror("mmap"); return 2; } + /* Start on a 2 MiB boundary, so each zeroed stripe is whole MiBs for the + * checkpoint writer, whatever mapping lies next to this one. */ + const size_t align = 2 * 1024 * 1024; + char *raw = mmap(NULL, bytes + align, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (raw == MAP_FAILED) { perror("mmap"); return 2; } + uint64_t *p = (uint64_t *) (((uintptr_t) raw + align - 1) & ~(align - 1)); + if ((char *) p > raw) munmap(raw, (char *) p - raw); + munmap((char *) p + bytes, raw + align - (char *) p); /* Match the baseline's ordinary anonymous pages at checkpoint creation. */ if (madvise(p, bytes, MADV_NOHUGEPAGE)) { perror("madvise"); return 2; } for (size_t i = 0; i < bytes / 8; ++i) { diff --git a/test/restore-memory-check.py b/test/restore-memory-check.py index 20aba7fb43..c58c1168d8 100644 --- a/test/restore-memory-check.py +++ b/test/restore-memory-check.py @@ -143,7 +143,8 @@ def main(): DMTCP_RESTART_HUGEPAGES=str(int(huge)), DMTCP_RESTART_MMAP=str(int(mapped))) text = run(root, case / 'restart.log', ['dmtcp_restart', *files], env) assert 'BENCH_VERIFIED' in text, text[-6000:] - if layout == 'zeroed' and readers > 1 and mib >= 1024 and not mapped: + if (layout == 'zeroed' and readers > 1 and mib >= 1024 and not mapped + and platform.machine() == 'x86_64'): # Zero runs cut the area into 64 MiB runs: read together, not one by one. assert any(map(int, re.findall(r'\bparallel_kib=(\d+)', text))), \ f'{name}: the readers did not share the runs between zero runs' From bc5261e4acf005953e9898346a7fbbeb1c94945f Mon Sep 17 00:00:00 2001 From: Kirill Bespalov Date: Wed, 4 Nov 2026 08:06:49 +0300 Subject: [PATCH 3/4] Bound a zeroed image by the dense image of the same size CI images of a 1025 MiB bench carry about 2 MiB more besides the payload than those of a 257 MiB one, so measuring zeroed16 against the 257 MiB ordinary image failed by that much, though no zero stripe was saved. Each zeroed image is now bounded by the dense image of its own size. --- test/restore-memory-check.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/test/restore-memory-check.py b/test/restore-memory-check.py index c58c1168d8..617bc2b118 100644 --- a/test/restore-memory-check.py +++ b/test/restore-memory-check.py @@ -120,11 +120,11 @@ def main(): files = images(first) stored[name] = sum(os.path.getsize(f) for f in files) if layout == 'zeroed': - # The bench clears every odd 64 MiB stripe; none of it may be saved. - # Beyond the payload, the image holds what the ordinary one does. - other = stored['ordinary'] - 257 * 2**20 + # The bench clears every odd 64 MiB stripe; none of it may be saved: + # the image is a dense one of the same size less those stripes. + dense = stored['ordinary' if mib == 257 else 'parallel16'] zero = sum(min(64, mib - s) for s in range(64, mib, 128)) * 2**20 - limit = mib * 2**20 - zero + other + 2**20 + limit = dense - zero + 2**20 assert stored[name] <= limit, \ f'{name}: {stored[name]} bytes stored, at most {limit} expected' if compressed: From 4c12195208427b8e46a3f60ca9ff5bb59024f018 Mon Sep 17 00:00:00 2001 From: Kirill Bespalov Date: Wed, 4 Nov 2026 08:47:42 +0300 Subject: [PATCH 4/4] Record the GPU restore result; pinned host memory was already covered On one RTX 3090, warmctl beta.17 baking and restoring Qwen3-0.6B: process images 8,582 -> 6,065 MiB, the vLLM worker's memory pass 1.05 -> 0.74 s, /health 4.6-5.1 -> 4.0-4.4 s, same greedy answers. Pinned CUDA host memory is a shared mapping of /dev/zero, which the content scan already split into zero runs, so the comments no longer name a CUDA host buffer as what the resident scan finds. --- doc/restart-memory-profiling.md | 7 ++++--- doc/warmctl-release-notes.md | 10 +++++++++- src/writeckpt.cpp | 5 +++-- 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/doc/restart-memory-profiling.md b/doc/restart-memory-profiling.md index 6a95b83160..81430378e3 100644 --- a/doc/restart-memory-profiling.md +++ b/doc/restart-memory-profiling.md @@ -92,9 +92,10 @@ benchmark verifies a later variable whose longer prefixed name occurs first. ## Resident zero pages -A process can hold gigabytes of resident zeros: a pinned host buffer sized -larger than what it holds, a pool cleared after use. The page residency scan -saves every resident page. The writer now also reads resident private +A process can hold gigabytes of resident zeros: a tensor or pool sized larger +than what it holds, or cleared after use. The page residency scan saves every +resident page of private anonymous memory. (Pinned CUDA host memory is a +shared mapping of `/dev/zero`, which the content scan already covered.) The writer now also reads resident private anonymous memory a MiB at a time, on MiB address boundaries, and records each all-zero MiB as a zero run, the header the residency scan already writes for absent pages. A restart maps diff --git a/doc/warmctl-release-notes.md b/doc/warmctl-release-notes.md index 49faf1b4ab..34e937b8fa 100644 --- a/doc/warmctl-release-notes.md +++ b/doc/warmctl-release-notes.md @@ -13,7 +13,15 @@ tail, plus 10,000 small areas), 16 readers and huge pages: the image fell from 14.07 to 11.07 GiB and the checkpoint from 16.5 to 14.3 s; the memory pass from 8.66 to 6.86 s with a cold file cache and from 1.35 to 1.19 s with a warm one. With zeros in every other 64 MiB stripe of 12 GiB, the image halved and the -pass fell from 7.28 to 3.68 s cold. Not yet measured on a GPU restore. +pass fell from 7.28 to 3.68 s cold. + +On a GPU, warmctl beta.17 baking and restoring Qwen3-0.6B on one RTX 3090 +(8 readers, huge pages): the process images fell from 8,582 to 6,065 MiB (the +vLLM worker's from 5,701 to 3,215 MiB), the worker's memory pass from 1.05 to +0.74 s, and time to `/health` from 4.6–5.1 s to 4.0–4.4 s, with the same +greedy answers. Pinned CUDA host memory is a shared mapping, whose zeros the +content scan already skipped, so the H200 Qwen3.8 worker's pinned embedding +table gains nothing from the writer change. # v5.0.0-warmctl.1 diff --git a/src/writeckpt.cpp b/src/writeckpt.cpp index 1dc6b41b1e..bf026303e4 100644 --- a/src/writeckpt.cpp +++ b/src/writeckpt.cpp @@ -361,8 +361,9 @@ mtcp_write_anonymous_pages(int fd, Area area, bool residencyScanSafe) // (mtcp_get_next_page_range -> Util::areZeroPages) for deleted-file maps; // - resident anonymous runs are read too, an aligned MB at a time // (mtcp_get_next_aligned_range): a process may hold gigabytes of - // zeroed buffers (a CUDA host buffer, a cleared pool), and those are - // zero runs as well; + // zeroed buffers (a cleared tensor or pool), and those are zero runs + // as well. Pinned CUDA host memory is a shared mapping and takes the + // content scan anyway; // - a zero run is saved as a DMTCP_ZERO_PAGE header with no data, then // dropped from RSS with madvise(MADV_DONTNEED), unless it is resident; // non-zero runs save bytes.