Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
ID: ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP
Title: agent-record flaky: strix disk-headroom exhausts on shared runner
Row: CI-AGENT-RECORD-DISK
State: OPEN
State: CLOSED
Kind: bug
GitHub: -
Mirror: PENDING
Availability: FULL
Created: 2026-10-27
Updated: 2026-10-27
Closed: -
Updated: 2026-10-28
Closed: 2026-10-28

## Problem

The agent-record CI job fails intermittently on main (runs 40521291883, 40452744901) and reproduced on the #3513 rerun (job 122404172875): 49 tests in tests.tools.test_strix_vllm_oracle fail with ValueError('disk headroom exhausted') from tools/bench/strix_vllm_oracle/worker.py:257, plus CacheDropError (cache-drop proof retained 6291456 resident bytes). Environmental shared-runner pressure, not owned by a product row. Spec: .agents/specs/ci-agent-record-disk.md. Fix: free runner disk before the tools suite in .github/workflows/ci.yml.

## Resolution

-
Landed 2026-10-28 as f484c76ad (PR #3515). The wall was not runner disk pressure on / but the /tmp tmpfs of the ubuntu-26.04 image (actions/runner-images#14777): the strix worker's headroom() measures disk_usage of self.local and self.output, both under /tmp (worker.py:226,256), so the 20 GiB floor was unmeetable on 26.04 runners regardless of how empty / was. Proof the first attempt was insufficient: run 40586147690 job 122477980562 showed 113 GiB free on /dev/root and still failed 49+2 strix tests; build-test-cpu (122477933406) and sanitize-cpu (122477933291) failed identically in the same run. Fix: mask tmp.mount and lazy-unmount /tmp as the first step of the six ubuntu-latest compile/ctest lanes. Verified green: run 40637909804 -- agent-record and build-test-vulkan passed with df -h / /tmp showing /tmp on /dev/root at 88 GiB free.
27 changes: 27 additions & 0 deletions .agents/specs/ci-agent-record-disk.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,33 @@ locally.
workaround (mask `tmp.mount`, lazy-unmount `/tmp` as the first step) is
the one this change lands, and runner-images#14814 is the image-side fix.

## Outcome

Landed as f484c76ad (PR #3515, merged 2026-10-28T11:05Z). What settled it:

- The first hypothesis (shared-runner disk pressure on `/`) was **rejected
by evidence from its own fix**: PR #3515's first commit freed `/` to
113 GiB and the job still failed 49+2 strix tests. The worker measures
`/tmp` filesystems, not `/`.
- The root cause is the ubuntu-26.04 image's `/tmp` tmpfs
(actions/runner-images#14777), which makes the worker's 20 GiB floor
unmeetable by construction; the flakiness is runner-image assignment
during the 24.04→26.04 rollout.
- Defaults: the mask+lazy-unmount step lands FIRST in six ubuntu-latest
compile/ctest lanes (`agent-record`, `build-test-cpu`, `sanitize-cpu`,
`build-test-vulkan`, `build-newest-gcc`, `cuda-fat-build`), and only
those — the trivial record/document lanes do not write `/tmp`, and the
pinned `ubuntu-24.04-arm` lanes keep `/tmp` on the root disk. The
agent-record reclamation step stays because `/tmp` returning to the root
disk makes the freed space reachable by the suites and compilers. The
worker's 20 GiB floor is untouched: a safety property of the oracle
runner is not weakened to green a gate.
- Verified green on run 40637909804: `agent-record` and
`build-test-vulkan` passed with `df -h / /tmp` showing `/tmp` on
`/dev/root` at 88 GiB free; the long ctest lanes and the pre-existing
windows-msvc flint.cpp warning-as-error break are recorded separately
and do not belong to this row.

## Stop conditions

- If GitHub's runner image changes so that the reclaimed paths no longer
Expand Down
Loading