Skip to content

Commit 3ddd4e4

Browse files
committed
Close ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP and record the row outcome
The fix landed as f484c76 (PR localai-org#3515). The issue closes with the dated resolution evidence: the wall was the ubuntu-26.04 /tmp tmpfs, not root disk pressure; run 40637909804 proves the mask step green on agent-record and build-test-vulkan. The spec gains its ## Outcome: the rejected first hypothesis, the default values and why each holds, and what was left out of scope. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
1 parent f484c76 commit 3ddd4e4

2 files changed

Lines changed: 31 additions & 4 deletions

File tree

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
ID: ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP
22
Title: agent-record flaky: strix disk-headroom exhausts on shared runner
33
Row: CI-AGENT-RECORD-DISK
4-
State: OPEN
4+
State: CLOSED
55
Kind: bug
66
GitHub: -
77
Mirror: PENDING
88
Availability: FULL
99
Created: 2026-10-27
10-
Updated: 2026-10-27
11-
Closed: -
10+
Updated: 2026-10-28
11+
Closed: 2026-10-28
1212

1313
## Problem
1414

1515
The agent-record CI job fails intermittently on main (runs 40521291883, 40452744901) and reproduced on the #3513 rerun (job 122404172875): 49 tests in tests.tools.test_strix_vllm_oracle fail with ValueError('disk headroom exhausted') from tools/bench/strix_vllm_oracle/worker.py:257, plus CacheDropError (cache-drop proof retained 6291456 resident bytes). Environmental shared-runner pressure, not owned by a product row. Spec: .agents/specs/ci-agent-record-disk.md. Fix: free runner disk before the tools suite in .github/workflows/ci.yml.
1616

1717
## Resolution
1818

19-
-
19+
Landed 2026-10-28 as f484c76ad (PR #3515). The wall was not runner disk pressure on / but the /tmp tmpfs of the ubuntu-26.04 image (actions/runner-images#14777): the strix worker's headroom() measures disk_usage of self.local and self.output, both under /tmp (worker.py:226,256), so the 20 GiB floor was unmeetable on 26.04 runners regardless of how empty / was. Proof the first attempt was insufficient: run 40586147690 job 122477980562 showed 113 GiB free on /dev/root and still failed 49+2 strix tests; build-test-cpu (122477933406) and sanitize-cpu (122477933291) failed identically in the same run. Fix: mask tmp.mount and lazy-unmount /tmp as the first step of the six ubuntu-latest compile/ctest lanes. Verified green: run 40637909804 -- agent-record and build-test-vulkan passed with df -h / /tmp showing /tmp on /dev/root at 88 GiB free.

‎.agents/specs/ci-agent-record-disk.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,33 @@ locally.
149149
workaround (mask `tmp.mount`, lazy-unmount `/tmp` as the first step) is
150150
the one this change lands, and runner-images#14814 is the image-side fix.
151151

152+
## Outcome
153+
154+
Landed as f484c76ad (PR #3515, merged 2026-10-28T11:05Z). What settled it:
155+
156+
- The first hypothesis (shared-runner disk pressure on `/`) was **rejected
157+
by evidence from its own fix**: PR #3515's first commit freed `/` to
158+
113 GiB and the job still failed 49+2 strix tests. The worker measures
159+
`/tmp` filesystems, not `/`.
160+
- The root cause is the ubuntu-26.04 image's `/tmp` tmpfs
161+
(actions/runner-images#14777), which makes the worker's 20 GiB floor
162+
unmeetable by construction; the flakiness is runner-image assignment
163+
during the 24.04→26.04 rollout.
164+
- Defaults: the mask+lazy-unmount step lands FIRST in six ubuntu-latest
165+
compile/ctest lanes (`agent-record`, `build-test-cpu`, `sanitize-cpu`,
166+
`build-test-vulkan`, `build-newest-gcc`, `cuda-fat-build`), and only
167+
those — the trivial record/document lanes do not write `/tmp`, and the
168+
pinned `ubuntu-24.04-arm` lanes keep `/tmp` on the root disk. The
169+
agent-record reclamation step stays because `/tmp` returning to the root
170+
disk makes the freed space reachable by the suites and compilers. The
171+
worker's 20 GiB floor is untouched: a safety property of the oracle
172+
runner is not weakened to green a gate.
173+
- Verified green on run 40637909804: `agent-record` and
174+
`build-test-vulkan` passed with `df -h / /tmp` showing `/tmp` on
175+
`/dev/root` at 88 GiB free; the long ctest lanes and the pre-existing
176+
windows-msvc flint.cpp warning-as-error break are recorded separately
177+
and do not belong to this row.
178+
152179
## Stop conditions
153180

154181
- If GitHub's runner image changes so that the reclaimed paths no longer

0 commit comments

Comments
 (0)