Skip to content

Do not save or read back resident zero pages; keep zero-split areas on the parallel readers #3

Closed

Description

@kbespalov

Checkpoint images save resident pages that hold only zeros, and restart reads them back. On the Qwen3.8-Flash-Next-NVFP4 H200 worker, one 47.68 GiB anonymous payload (the FP8 PLE n-gram embedding, pinned on the CPU) is most of the 53.5 GiB worker image and 16.2 s of its 20.9 s cold memory pass on Runpod NFS (release-cold-cycle run 09: read_kib=50000896 read_us=16241444 mode=pread). A 15 GiB sample of the same model's CRIU images is 24.7% zero 4 KiB pages, all in runs of at least 64 MiB, in one tail.

Problems:

  • The page residency scan (Util::scanOccupiedRangeBatch) skips only absent pages. A page that is present but zero (a pinned buffer larger than its contents, a cleared pool) is written to the image and read back on every restart.
  • The non-zero rest of these images does not compress (zstd 0.977 of it, LZ4 worse), so compressing images cannot replace skipping zeros, and decompression costs restart CPU.
  • If zero runs were skipped, they would cut a large area into runs smaller than the 1 GiB a payload needs for the parallel readers (restart_parallel_read), and those runs would be read by one thread: a regression on exactly the images that benefit.

Acceptance:

  • The writer records resident all-zero runs of private anonymous memory as DMTCP_ZERO_PAGE children, with no image format change: existing restarts read new images, new restarts read old ones.
  • Ordinary data costs no measurable checkpoint time; resident pages are left in place (no MADV_DONTNEED on memory that may be pinned for DMA); an environment switch restores the old images byte for byte.
  • With DMTCP_RESTART_READERS, runs between zero runs are still read in parallel, without slowing restarts of images with many small areas, cold or warm.
  • CPU regressions verify every byte, protections, image size, parallel use and cross-version restarts; the full suite passes with one and with 16 readers.
  • GPU qualification on the H200 worker (pinned memory re-registered by CUDA after restart) before a warmctl release ships it by default.

Activity

  1. kbespalov commented on Nov 4, 2026

    @kbespalov
    Author

    Correction after testing on a GPU: the problem statement above is wrong about the H200 worker. Its 47.68 GiB payload is the PLE n-gram table in pinned CUDA host memory, and torch's pinned memory is a shared mapping of /dev/zero. DMTCP never used the residency scan for it: the content scan already recorded its zero MiBs as zero runs, which is why that DMTCP image is smaller than the CRIU one. #4 does not shrink it.

    What #4 does fix is the same problem in private anonymous memory, which is real in vLLM: on one RTX 3090, Qwen3-0.6B baked and restored with warmctl beta.17, the process images fell from 8,582 to 6,065 MiB (the worker's from 5,701 to 3,215 MiB), the worker's memory pass from 1.05 to 0.74 s and /health from 4.6–5.1 s to 4.0–4.4 s, with the same answers. The batched restart reads now also share the API server's and engine core's large payloads between the readers (0.70 → 0.47 s each). The acceptance items are met on CPU and that GPU; the H200 item stays open until a Flash-Next restore is measured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions