Skip resident zero pages; share zero-split areas between restart readers

doc/restart-memory-profiling.md

@@ -8,7 +8,8 @@ normally. The default reader and checkpoint format stay unchanged.
 | `DMTCP_RESTART_PROFILE` | `1` | Log memory restoration, large/slow regions, page faults, the restart barrier and plugin phase. |
 | `DMTCP_RESTART_HUGEPAGES` | `1` | Advise huge pages for eligible large private anonymous mappings before loading their contents. |
 | `DMTCP_RESTART_MMAP` | `1` | Map eligible raw anonymous payloads of at least 64 MiB directly from the image using private mappings. |
-| `DMTCP_RESTART_READERS` | `2`–`8`, `16` | On x86-64, copy raw regular-file payloads of at least 1 GiB with parallel `pread` helpers. Default is one reader. |
+| `DMTCP_RESTART_READERS` | `2`–`8`, `16` | On x86-64, copy raw regular-file payloads with parallel `pread` helpers: writable payloads of at least 1 MiB together, in batches, and any other payload of at least 1 GiB on its own. Default is one reader. |
+| `DMTCP_DISABLE_ZERO_SCAN` | `1` | At checkpoint, save resident anonymous pages that hold only zeros, as before, instead of recording them as zero runs. |
 
 Use one optimization at a time for A/B measurements. Example:
 
@@ -23,8 +24,10 @@ bake a new snapshot before testing its post-restart instrumentation. The MTCP
 memory reader uses raw syscalls and stores its counters in relocated state.
 
 `phase=memory` reports microseconds spent in the memory pass and payload reads,
-anonymous mmap setup time, bytes read or file-mapped in KiB, and minor/major
-faults. `pid` is the real process PID; `virtual_pid` is the checkpoint PID.
+anonymous mmap setup time, bytes read or file-mapped in KiB, minor/major
+faults, and `batches`, the number of batched reads. `phase=batch` reports each
+batch: its payloads (`segments`), KiB, microseconds and whether the helpers
+(`pread`) or the parent alone (`read`) read it. `pid` is the real process PID; `virtual_pid` is the checkpoint PID.
 `phase=area` reports large or slow payloads, mmap eligibility, offset alignment
 and mmap errors. An eligible payload with nonzero `offset_mod` falls back to
 reading. `restart_barrier`,
@@ -56,10 +59,21 @@ registration may force private copies and remove some or all of the benefit.
 Neither experiment is qualified for GPU production solely by CPU tests.
 
 Parallel readers can be combined with huge-page advice. Their temporary stacks
-are unmapped and every helper is reaped before moving to another memory area or
-resuming the application. Helpers share the address space but do not use libc,
-TLS or pthreads. Failed helper creation falls back to reading its range in the
-parent; failed reads fall back to the ordinary reader after all helpers exit.
+are unmapped and every helper is reaped before the next memory area is mapped
+or the application resumes. Helpers share the address space but do not use
+libc, TLS or pthreads. Failed helper creation falls back to reading its range in
+the parent; failed reads fall back to the ordinary reader after all helpers exit.
+
+With readers, the payload of a writable area of at least 1 MiB is not read
+when its header is: the pass records where it goes and seeks past it, and the
+helpers read the recorded payloads together, every 1024 of them and at the end
+of the image, each helper taking an equal share of the bytes. A zero run cuts
+a large area into runs that are smaller than the 1 GiB a single payload needs
+for the helpers; batched, they still share them. Read-only areas are read at
+once, as they are protected right after, and so are areas under 1 MiB: putting
+thousands of them off turns the pass over the headers into a seek per area,
+which costs more on a cold or network file than the helpers save. The batch
+array lives on the restart stack, never at an address an area could need.
 Helpers arm a parent-death signal and recheck their parent, so cancellation
 does not leave them holding the restore's address space.
 `parallel_kib` identifies actual use. `child_minflt` and `child_majflt` are the
@@ -75,3 +89,32 @@ During instrumentation, `dmtcp_get_restart_env()` was found to bound its scan
 by `sizeof(char *)`, which examined only the first environment entry. The fix
 uses the actual read length and requires an exact name followed by `=`. The
 benchmark verifies a later variable whose longer prefixed name occurs first.
+
+## Resident zero pages
+
+A process can hold gigabytes of resident zeros: a tensor or pool sized larger
+than what it holds, or cleared after use. The page residency scan saves every
+resident page of private anonymous memory. (Pinned CUDA host memory is a
+shared mapping of `/dev/zero`, which the content scan already covered.) The writer now also reads resident private
+anonymous memory a MiB at a time, on MiB address boundaries, and records each
+all-zero MiB as a zero run,
+the header the residency scan already writes for absent pages. A restart maps
+the area and reads only the other runs, so zeros cost neither image bytes nor
+reads. Restarts that predate this read such images too: the format is the same.
+
+The check stops at the first nonzero word of each MiB, so ordinary data costs
+almost nothing; only zero MiBs are read whole, and they are no longer written.
+Resident zeros are left in place (no `MADV_DONTNEED`): the process may run on
+after the checkpoint, and a page pinned for DMA must keep its frame. Partial
+MiBs at the ends of a run are saved as before, so a zero run is never shorter
+than a MiB and never costs more header than data.
+`DMTCP_DISABLE_ZERO_SCAN=1` restores the old images byte for byte.
+
+`test/restore-memory-bench.c` has a `zeroed` layout, which writes every other
+64 MiB stripe and then clears it. `test/restore-memory-check.py` checks that
+none of those stripes reach the image, that every word comes back, and that
+with readers the runs between them are still shared.
+`test/restore-many-areas.c` checkpoints 2,400 areas: writable ones of at least
+1 MiB between small read-only ones, so restarts batch more payloads than one
+batch holds, with shares starting and ending inside areas, and read the
+read-only ones at once. It verifies every byte and each protection.

doc/warmctl-release-notes.md

@@ -1,3 +1,28 @@
+# Unreleased
+
+- The checkpoint writer records resident all-zero MiBs of private anonymous
+  memory as zero runs instead of saving them. Images are smaller and restarts
+  read less; existing restarts read the new images, and
+  `DMTCP_DISABLE_ZERO_SCAN=1` turns it off.
+- With `DMTCP_RESTART_READERS`, writable payloads of at least 1 MiB are read by
+  the helpers in batches, so a large area cut by zero runs is still read in
+  parallel. Smaller and read-only payloads are read as before.
+
+On a CPU test process like the H200 worker (a 12 GiB area with a 25% zero
+tail, plus 10,000 small areas), 16 readers and huge pages: the image fell from
+14.07 to 11.07 GiB and the checkpoint from 16.5 to 14.3 s; the memory pass from
+8.66 to 6.86 s with a cold file cache and from 1.35 to 1.19 s with a warm one.
+With zeros in every other 64 MiB stripe of 12 GiB, the image halved and the
+pass fell from 7.28 to 3.68 s cold.
+
+On a GPU, warmctl beta.17 baking and restoring Qwen3-0.6B on one RTX 3090
+(8 readers, huge pages): the process images fell from 8,582 to 6,065 MiB (the
+vLLM worker's from 5,701 to 3,215 MiB), the worker's memory pass from 1.05 to
+0.74 s, and time to `/health` from 4.6–5.1 s to 4.0–4.4 s, with the same
+greedy answers. Pinned CUDA host memory is a shared mapping, whose zeros the
+content scan already skipped, so the H200 Qwen3.8 worker's pinned embedding
+table gains nothing from the writer change.
+
 # v5.0.0-warmctl.1
 
 First versioned TensorRails fork release, based on DMTCP 5.0. It includes the

src/constants.h

@@ -148,6 +148,10 @@
 // restart.
 #define ENV_VAR_DISABLE_FILE_MAPPED     "DMTCP_DISABLE_FILE_MAPPED"
 
+// Set to "1" to save resident anonymous pages that hold only zeros, as before,
+// instead of recording each 1 MB run of them as a zero run with no data.
+#define ENV_VAR_DISABLE_ZERO_SCAN       "DMTCP_DISABLE_ZERO_SCAN"
+
 // this list should be kept up to date with all "protected" environment vars
 #define ENV_VARS_ALL                  \
   ENV_VAR_NAME_HOST,                  \