diff --git a/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md b/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md index 7eefd3200..fcd5c2a67 100644 --- a/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md +++ b/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md @@ -1,14 +1,14 @@ ID: ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP Title: agent-record flaky: strix disk-headroom exhausts on shared runner Row: CI-AGENT-RECORD-DISK -State: OPEN +State: CLOSED Kind: bug GitHub: - Mirror: PENDING Availability: FULL Created: 2026-10-27 -Updated: 2026-10-27 -Closed: - +Updated: 2026-10-28 +Closed: 2026-10-28 ## Problem @@ -16,4 +16,4 @@ The agent-record CI job fails intermittently on main (runs 40521291883, 40452744 ## Resolution -- +Landed 2026-10-28 as f484c76ad (PR #3515). The wall was not runner disk pressure on / but the /tmp tmpfs of the ubuntu-26.04 image (actions/runner-images#14777): the strix worker's headroom() measures disk_usage of self.local and self.output, both under /tmp (worker.py:226,256), so the 20 GiB floor was unmeetable on 26.04 runners regardless of how empty / was. Proof the first attempt was insufficient: run 40586147690 job 122477980562 showed 113 GiB free on /dev/root and still failed 49+2 strix tests; build-test-cpu (122477933406) and sanitize-cpu (122477933291) failed identically in the same run. Fix: mask tmp.mount and lazy-unmount /tmp as the first step of the six ubuntu-latest compile/ctest lanes. Verified green: run 40637909804 -- agent-record and build-test-vulkan passed with df -h / /tmp showing /tmp on /dev/root at 88 GiB free. diff --git a/.agents/specs/ci-agent-record-disk.md b/.agents/specs/ci-agent-record-disk.md index 0114ef916..201f19798 100644 --- a/.agents/specs/ci-agent-record-disk.md +++ b/.agents/specs/ci-agent-record-disk.md @@ -149,6 +149,33 @@ locally. workaround (mask `tmp.mount`, lazy-unmount `/tmp` as the first step) is the one this change lands, and runner-images#14814 is the image-side fix. +## Outcome + +Landed as f484c76ad (PR #3515, merged 2026-10-28T11:05Z). What settled it: + +- The first hypothesis (shared-runner disk pressure on `/`) was **rejected + by evidence from its own fix**: PR #3515's first commit freed `/` to + 113 GiB and the job still failed 49+2 strix tests. The worker measures + `/tmp` filesystems, not `/`. +- The root cause is the ubuntu-26.04 image's `/tmp` tmpfs + (actions/runner-images#14777), which makes the worker's 20 GiB floor + unmeetable by construction; the flakiness is runner-image assignment + during the 24.04→26.04 rollout. +- Defaults: the mask+lazy-unmount step lands FIRST in six ubuntu-latest + compile/ctest lanes (`agent-record`, `build-test-cpu`, `sanitize-cpu`, + `build-test-vulkan`, `build-newest-gcc`, `cuda-fat-build`), and only + those — the trivial record/document lanes do not write `/tmp`, and the + pinned `ubuntu-24.04-arm` lanes keep `/tmp` on the root disk. The + agent-record reclamation step stays because `/tmp` returning to the root + disk makes the freed space reachable by the suites and compilers. The + worker's 20 GiB floor is untouched: a safety property of the oracle + runner is not weakened to green a gate. +- Verified green on run 40637909804: `agent-record` and + `build-test-vulkan` passed with `df -h / /tmp` showing `/tmp` on + `/dev/root` at 88 GiB free; the long ctest lanes and the pre-existing + windows-msvc flint.cpp warning-as-error break are recorded separately + and do not belong to this row. + ## Stop conditions - If GitHub's runner image changes so that the reclaimed paths no longer