Repository navigation
fcvm-fleet: tile root volumes at gp3 1000 MiB/s and 4000 IOPS - #720
Conversation
The three tile launch templates (spot, on demand, pool) get a root block device mapping of gp3 with 1000 MiB/s throughput and 4000 IOPS, from one local (tile_root_volume), instead of the AMI's 125 MiB/s and 3000 IOPS. The size and snapshot stay the AMI's. On the 71bb448e tile image, launch request to first frame over HTTPS (median): 35.1 s default, 29.7 s at 1000/4000, 19.8 s at 1000/16000 (five on-demand launches each, interleaved). A stopped pool tile started again reached its first frame in 27.5 s on default (3 starts), 16.6 s at 1000/4000 (6 starts) and 15.7 s at 1000/16000 (3 starts). The 16000 IOPS setting is left out: it adds nothing to the pool start and costs 65 USD a month more per volume. The GPU host template is unchanged: on a g4dn.metal launch from the 7f5fb7e6 host image, the store was active 64.9 s after the request on default and 64.4 s at 1000/4000, and the first GPU VM's first frame took 4.5 s and 5.0 s. scripts/test-fcvm-fleet.py: test_every_tile_template_sets_the_root_volume_to_gp3_1000_mibps_4000_iops fails against main's module and passes with this change.
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 59 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review Please review current head f08ea17. Independent HIGH code/security review confirms the root mapping inherits the AMI snapshot/size and the exact-secret/tag/type launch boundaries are unchanged. The pool’s additional provisioned-volume cost has been checked against AWS Pricing. |
|
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Merged as 7cf2ccd and applied with #717. AWS readback verifies all three latest tile templates use the reviewed AMI and /dev/sda1 gp3 at 4,000 IOPS / 1,000 MiB/s. IAM stays within its existing 10,235-character inline budget. Coordinator revision 55 remains healthy; no running instance was forcibly recycled. Combined merged-main 52-test suite passed on ARM, and the final full Terraform plan is empty. The eight existing pool volumes retain their current settings until replaced; eight upgraded roots add $320/month to the baseline storage cost. CodeRabbit’s quota-limited/skipped review on this PR was not counted as approval; independent HIGH code/security and exact-head Codex code reviews passed, with no unresolved threads. |
Sets the root volume of the three EC2 tile launch templates (spot, on demand, pool) to gp3 at 1000 MiB/s and 4000 IOPS. The AMI's default is 125 MiB/s and 3000 IOPS. Owner: "Faster disks are fine". The size and snapshot stay the AMI's; the setting is one local,
tile_root_volume, inmodules/fcvm-fleet-region/main.tf. No other file changes except one test.Measurements
Tile image: the one in #717 (ejc3/fcvm main 71bb448e), g4dn.xlarge, us-east-1,
fleet/ami/measure-boot.sh tile(launch request to first frame over HTTPS, the volume overridden per arm). Spot had no g4dn.xlarge capacity in any zone for the last set, so it ran on demand; the boot does not depend on the market.Fresh launch, five interleaved launches per setting, milliseconds:
Earlier launches on the same image (13 of them, mostly spot) read the same direction: medians 35.1 s default (n=9, one run at 57.8 s), 31.2 s at 4000 (n=9), 21.3 s at 16000 (n=10).
Stopped pool tile started again (what a wake does with
--ec2-pool),StartInstancesto first frame over HTTPS:The 46.1 s start at 4000 is the one outlier of the six, in the first set of three; the repeat set of three read 18.6, 15.9 and 15.8 s.
16000 IOPS is left out
It shortens a fresh launch by about 10 s, but a pool start, which is the standing path with
fcvm_fleet_tile_pool = true, is the same at 4000 (median 16.6 s) and 16000 (15.7 s). It costs 65 USD a month more per volume than 4000, and the pool holds 8 volumes while stopped.Cost (us-east-1 gp3 list prices, per volume per month)
32 GiB storage at 0.08 USD/GiB: 2.56 USD, unchanged. Throughput above the 125 MiB/s included, 875 MiB/s at 0.04 USD per MiB/s (40.96 USD per GiB/s): 35.00 USD. IOPS above the 3000 included, 1000 at 0.005 USD: 5.00 USD. Total 42.56 USD against 2.56 USD for the default, so +40.00 USD a month, +0.055 USD per hour for a tile that is launched and terminated.
The 8 stopped pool volumes keep their provisioned throughput and IOPS while stopped: 340.48 USD a month against 20.48 USD, +320 USD. At 16000 IOPS the pool would be +840 USD.
GPU hosts (g4dn.metal): not changed
Host image 7f5fb7e6, on demand,
measure-boot.sh hostwith the template'sVolumeInitializationRate=300on both arms, 64 GiB volume. Two default-throughput launches, two at 1000/4000, plus one more at 1000/4000 that got no capacity:The differences are inside the launch-to-launch spread. The kernel starts about 37 s after the request on a metal instance, and the store restore reads its 39 GiB from S3, not from the root volume (8.5 to 10.3 s). The script stops at the first GPU VM; it does not run the agent or the coordinator, so "agent connected" was not measured. I left the host template alone.
Check
python3 -I scripts/test-fcvm-fleet.py: 52 tests pass.test_every_tile_template_sets_the_root_volume_to_gp3_1000_mibps_4000_iopsfails against main's module (no block device mapping on the three templates).terraform fmt -checkis clean. ARunInstancesdry run against the tile image with a mapping that gives type, throughput and IOPS but no size is accepted, and so is the same inCreateLaunchTemplate.Not run:
terraform plan(the jumpbox applies). The coordinator's launch policy has no condition on volume type, throughput or IOPS. A stopped pool tile keeps the volume it was launched with, so the 8 pool tiles get the new setting only when the coordinator next launches them from the template, not at apply.