Tile AMI: prefetch the first boot's root volume pieces by ejc3 · Pull Request #1407 · ejc3/fcvm

ejc3 · 2026-10-28T12:57:50Z

Prefetch the tile image's first-boot root volume pieces so a fresh EC2 tile reaches its first frame in about 33 s instead of 45 to 84 s.

Cause: the root volume of a tile is made from the AMI's snapshot and fetches each block from S3 on its first read. On the 728be8c tile image systemd-modules-load took 35.4 s (190 MiB of NVIDIA modules) and the renderer unit 8.7 s, with 14 s from podman's image pull to the container's init. Same mechanism as #1370 and #1397 for the host image.

Changes (all under fleet/ami):

tile/resident.py + build.sh trace: list the byte ranges of root volume files in the page cache of a tile that served its first frame (2,825 pieces of 2,422 files, 421 MiB, modules first). trace refuses a boot that ran the prefetch; bake refuses an image without a list (NO_PREFETCH_LIST=1 for the first image of a lineage).
tile/prefetch.c: static reader, 256 threads, 1 MiB pieces, list order. Built by build.sh.
tile/fcvm-fleet-prefetch.service: wanted by sysinit.target, ordered after nothing, so it runs beside systemd-modules-load.
fcvm-fleet-renderer.service waits for /dev/nvidia-uvm-tools: with the faster boot every launch logged failed to stat CDI host device and restarted after 2 s.
measure-boot.sh tile prints systemd-analyze, cloud-init stages, the renderer journal and the root volume read counters after the first frame; VOLUME=Throughput=..,Iops=.. sets the root volume.

Measured (g4dn.xlarge, on demand, launch request to first frame over HTTPS, spot had no capacity):

image	launches (s)
`728be8c` tile AMI	84.2, 45.5, 58.6 (one spot launch: 86.1)
this PR's recipe, built on it	36.6, 32.0, 33.5

systemd-analyze blame: modules-load 35.4 s -> 4.0 s, renderer unit 8.7 s -> 2.9 s, prefetch unit 5 s beside them.

EBS settings (reading the 421 MiB, 256 readers, from a volume made from the AMI's snapshot): gp3 default 2.8 to 6.2 s; 500 MiB/s 1.8 s; 1000 MiB/s + 4000 IOPS 0.55 s; 1000 MiB/s + 16000 IOPS 0.42 s; volume initialization rate 300 MiB/s 6.2 s (no better than default, as in #1397). Whole-tile launches: 1000 MiB/s + 4000 IOPS 19.5 s, 500 MiB/s + 3000 IOPS 24.9 s, default 32 to 37 s. This PR does not change the volume; the launch templates are in ejc3/aws and the throughput costs extra per volume-hour, so that is a separate decision.

Tests: bash fleet/ami/check.sh (shellcheck, bash -n, and fixtures for the unit's ordering, enable and build lines, the bake and trace refusals, the renderer's wait, prefetch.c compiled and run on listed pieces, resident.py's ranges, skip list, module ordering and recent-file filter). New checks fail on main because the files and lines are absent. CI does not run check.sh.

Contract: tile image recipes only; the AMI is rebuilt from main after this merges and moved in ejc3/aws.

Summary by CodeRabbit

New Features
- Tile images now preload previously accessed root-volume data during startup, helping reduce time to the first frame.
- Boot measurements now include startup timing, renderer logs, and root-volume read statistics. An optional volume setting is also available for measurements.
Bug Fixes
- The renderer now waits for its required graphics device to appear before starting, and reports an error if it does not appear in time.
Documentation
- Updated tile image documentation with prefetch behavior and boot measurement details.

…uvm-tools node A tile launched from the AMI took 45 to 84 s to its first frame (on demand, three launches; 86 s on one spot launch). The root volume is made from the AMI's snapshot and fetches each block from S3 on its first read: on the 728be8c image systemd-modules-load took 35.4 s (190 MiB of NVIDIA modules) and the renderer unit 8.7 s, with 14 s from podman's image pull to the container's init. - tile/resident.py lists, with mincore, the byte ranges of root volume files that a tile's page cache holds after its first frame (2,825 pieces of 2,422 files, 421 MiB), the NVIDIA modules first. build.sh `trace` writes it to /etc/fcvm-fleet/prefetch.list and refuses a boot that already ran the prefetch; `bake` refuses an image without a list unless NO_PREFETCH_LIST=1. - tile/prefetch.c is a static reader (256 threads, 1 MiB pieces, list order) that build.sh compiles into /usr/local/lib/fcvm-fleet/prefetch. - fcvm-fleet-prefetch.service is wanted by sysinit.target and ordered after nothing, so it runs beside systemd-modules-load instead of ahead of it. - fcvm-fleet-renderer.service waits for /dev/nvidia-uvm-tools before podman: with the faster boot every launch logged "failed to stat CDI host device" and restarted after 2 s. - measure-boot.sh tile prints systemd-analyze, cloud-init stages, the renderer journal and the root volume's read counters after the first frame, and takes VOLUME=Throughput=...,Iops=... for the root volume. Measured on a g4dn.xlarge, on demand, launch request to first frame over HTTPS: 84.2, 45.5, 58.6 s before; 36.6, 32.0, 33.5 s with the prefetch. Reading the 421 MiB (256 readers) from a volume made from the AMI's snapshot took 2.8 to 6.2 s at gp3 defaults, 1.8 s at 500 MiB/s, 0.55 s at 1000 MiB/s and 4000 IOPS; a volume initialization rate of 300 MiB/s took 6.2 s. Tile launches with 1000 MiB/s and 4000 IOPS reached the first frame in 19.5 s, with 500 MiB/s and 3000 IOPS in 24.9 s. Tests: fleet/ami/check.sh pins the unit (no ordering, wanted by sysinit.target, conditioned on the list), build.sh enabling it and building the binary, bake and trace refusals, the renderer's wait before podman, prefetch.c's reads of listed pieces (compiled and run on a fixture), and resident.py's ranges, skip list, module ordering and recent-file filter.

ejc3 · 2026-10-28T12:57:57Z

@codex review

chatgpt-codex-connector · 2026-10-28T12:58:09Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-28T13:02:53.211324Z	`363d6d6`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

coderabbitai · 2026-10-28T12:58:14Z

Caution

Review failed

The pull request is closed.

Walkthrough

The tile AMI build now records page-cache-resident root-volume ranges and reads them during startup through a systemd service. The change also adds renderer device-node ordering, boot diagnostics, volume configuration for measurements, and tests for the new behavior.

Changes: Tile root-volume prefetch

Layer / File(s)	Summary
Resident range generation and prefetch reading `fleet/ami/tile/resident.py`, `fleet/ami/tile/prefetch.c`, `fleet/ami/check.sh`	`resident.py` writes eligible page-cache-resident file ranges to a list. The static prefetch program reads listed ranges in chunks using worker threads. Checks cover range selection, ordering, and reads.
AMI trace, bake, and startup integration `fleet/ami/tile/build.sh`, `fleet/ami/tile/fcvm-fleet-prefetch.service`, `fleet/ami/tile/fcvm-fleet-renderer.service`, `fleet/ami/check.sh`, `fleet/ami/README.md`	The trace phase creates the prefetch list, and bake requires a nonempty list unless `NO_PREFETCH_LIST=1`. The service runs the prefetch binary when the list exists. The renderer waits for `/dev/nvidia-uvm-tools`.
Boot measurements and diagnostics `fleet/ami/measure-boot.sh`, `fleet/ami/README.md`	The measurement script accepts an optional gp3 root-volume mapping and collects boot diagnostics. The README records first-frame times and volume measurements.

Priority: Pending latest changes

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant build.sh
  participant resident.py
  participant fcvm-fleet-prefetch.service
  participant prefetch
  participant root_volume
  build.sh->>resident.py: Generate resident file ranges
  resident.py-->>build.sh: Write prefetch.list
  fcvm-fleet-prefetch.service->>prefetch: Run with prefetch.list
  prefetch->>root_volume: Read listed file ranges

Possibly related issues

ejc3/fcvm#1257 — The change addresses root-volume reads during tile startup, which are part of the issue’s measured boot phases. It does not address the other phases described there.

Possibly related PRs

ejc3/fcvm#1228 — Both changes pre-read AMI-backed files to reduce cold-read latency; this PR targets tile root-volume ranges.

Merge Risk: Low · up to 99018

Host measurements with a volume override may fail to launch, and the reported disk counters may include diagnostic activity. These bounded measurement issues should be fixed or explicitly accepted before merge.

Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 990188c466

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ot's active state A oneshot without RemainAfterExit never enters the active state, so ActiveEnterTimestampMonotonic read 0 after a successful run (checked with a oneshot unit on a systemd host) and the guard in build.sh trace never fired. The prefetch unit now touches /run/fcvm-fleet-prefetched and trace refuses when it exists. check.sh fails on the previous build.sh.

ejc3

NOT-A-DEFECT: the Codex receipt and CodeRabbit summary bodies carry no finding of their own; the one inline finding is answered and resolved above.

ejc3 · 2026-10-28T13:01:17Z

@codex review

chatgpt-codex-connector · 2026-10-28T13:02:51Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 363d6d6bc7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread fleet/ami/tile/build.sh Outdated

ejc3 commented Oct 28, 2026

View reviewed changes

ejc3 merged commit 71bb448 into main Oct 28, 2026
14 checks passed

ejc3 deleted the perf/tile-boot-prefetch branch October 28, 2026 13:03

ejc3 mentioned this pull request Oct 28, 2026

fcvm-fleet: tile AMI from ejc3/fcvm main 71bb448e (boot prefetch) ejc3/aws#717

Merged

Navigation Menu

Repository navigation

Tile AMI: prefetch the first boot's root volume pieces - #1407

Tile AMI: prefetch the first boot's root volume pieces#1407
ejc3 merged 2 commits into
mainfrom
perf/tile-boot-prefetch

ejc3 commented Oct 28, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

ejc3 commented Oct 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 28, 2026 •

edited

Loading

Review failed

Sequence Diagram(s)

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

ejc3 left a comment

Uh oh!

ejc3 commented Oct 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ejc3 commented Oct 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

ejc3 commented Oct 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

coderabbitai Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Sequence Diagram(s)

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ejc3 left a comment

Choose a reason for hiding this comment

Uh oh!

ejc3 commented Oct 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ejc3 commented Oct 28, 2026 •

edited by coderabbitai Bot

Loading

chatgpt-codex-connector Bot commented Oct 28, 2026 •

edited

Loading

coderabbitai Bot commented Oct 28, 2026 •

edited

Loading