Repository navigation
Free runner disk before the agent-record tools suite - #3515
Merged
Merged
Conversation
The agent-record job fails intermittently on main on shared-runner disk pressure: 49 strix-oracle tests die on the worker's 20 GiB headroom check and the cache-drop proof cannot evict. The spec scopes a runner disk-reclamation step; no checker, test, or product change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
The shared runner's free disk drops under the strix worker's 20 GiB
min_disk_bytes floor mid-job, so the tools-suite step reds intermittently
on pressure that has nothing to do with the tree: 49
tests.tools.test_strix_vllm_oracle failures on ValueError('disk headroom
exhausted') at worker.py:257, plus CacheDropError where the cache-drop
proof cannot evict 6291456 resident bytes. Main carries the same
intermittent failure (runs 40521291883, 40452744901) and the localai-org#3513 rerun
reproduced it (job 122404172875).
Reclaim the image's largest unused tenants -- Android SDK, .NET, GHC,
JVM, stale docker layers -- in a best-effort step right after checkout,
and log df -h / so the run carries the evidence. No workflow in this
repository uses any reclaimed tree. The worker's 20 GiB floor is
unchanged: the fix removes the pressure, not the check.
Red-before is main itself: two of its last five agent-record runs failed
the same way without this branch. Green-after is this pull request's
agent-record job passing with the df line in its log.
Fixes ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
The first attempt (reclamation only) did not fix the flaky agent-record
job: run 40586147690's job 122477980562 showed 113 GiB free on /dev/root
and still failed 49+2 strix-oracle tests on ValueError('disk headroom
exhausted') from worker.py:257. The reason is that the worker's headroom()
never measures / -- it measures shutil.disk_usage of self.local and
self.output (worker.py:226,256), and both live under /tmp. The same run's
build-test-cpu (122477933406) and sanitize-cpu (122477933291) failed on
the identical raise, because ctest registers the tools suites there too.
Root cause: ubuntu-latest is rolling from 24.04 to 26.04, and on 26.04
/tmp is a 50%-of-RAM tmpfs with a per-user quota (~6.3 GiB usable) while /
has ~90+ GiB free (actions/runner-images#14777). A runner on 26.04
unmeets the worker's 20 GiB floor by construction; the intermittency is
runner-image assignment, not workload. The fix is the runner-images
maintainer-endorsed workaround: mask tmp.mount and lazy-unmount /tmp as
the FIRST step of the job, before anything writes there, with df -h /
/tmp as the log evidence. On a 24.04 runner both commands are harmless
no-ops. The step lands in the six ubuntu-latest lanes that compile or
run test suites: agent-record, build-test-cpu, sanitize-cpu,
build-test-vulkan, build-newest-gcc, cuda-fat-build. The pinned
ubuntu-24.04-arm lanes keep /tmp on the root disk already.
The worker's 20 GiB floor is unchanged -- the fix removes the pressure,
not the check. The agent-record reclamation step stays: with /tmp back
on the root disk it buys the headroom the suites and compilers consume.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
added a commit
to lu-zero/vllm.cpp
that referenced
this pull request
Oct 28, 2026
The fix landed as f484c76 (PR localai-org#3515). The issue closes with the dated resolution evidence: the wall was the ubuntu-26.04 /tmp tmpfs, not root disk pressure; run 40637909804 proves the mask step green on agent-record and build-test-vulkan. The spec gains its ## Outcome: the rejected first hypothesis, the default values and why each holds, and what was left out of scope. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
added a commit
that referenced
this pull request
Oct 28, 2026
The fix landed as f484c76 (PR #3515). The issue closes with the dated resolution evidence: the wall was the ubuntu-26.04 /tmp tmpfs, not root disk pressure; run 40637909804 proves the mask step green on agent-record and build-test-vulkan. The spec gains its ## Outcome: the rejected first hypothesis, the default values and why each holds, and what was left out of scope. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The agent-record job (and the C++ ctest lanes) fail intermittently on the
shared runners, on pressure that has nothing to do with the tree.
Mechanism, traced twice. On
main2 of the last 5 agent-record runsfail with 49
tests.tools.test_strix_vllm_oraclefailures onValueError("disk headroom exhausted")fromtools/bench/strix_vllm_oracle/worker.py:257, plusCacheDropErrorin thecache-drop test. PR #3515's first commit (disk reclamation only) was
insufficient: run 40586147690, job 122477980562, showed 113 GiB free on
/dev/rootand still failed the same 49+2. The reason: the worker'sheadroom()never measures/-- it measuresshutil.disk_usageofself.localandself.output(worker.py:226,256), and both live under/tmp. The same run'sbuild-test-cpu(job 122477933406) andsanitize-cpu(job 122477933291) failed on the identical raise, becausectestregisters the tools suites there too.Root cause.
ubuntu-latestis rolling from 24.04 to 26.04(actions/runner-images#14777), and on 26.04
/tmpis a 50%-of-RAM tmpfswith a per-user quota (~6.3 GiB usable) while
/has ~90+ GiB free. Arunner on 26.04 unmeets the worker's 20 GiB floor by construction; the
intermittency is runner-image assignment, not workload.
The fix. The runner-images maintainer-endorsed workaround: mask
tmp.mountand lazy-unmount/tmpas the FIRST step of each affected job,before anything writes there, with
df -h / /tmpas the log evidence. On a24.04 runner both commands are harmless no-ops. The step lands in the six
ubuntu-latestlanes that compile or run test suites:agent-record,build-test-cpu,sanitize-cpu,build-test-vulkan,build-newest-gcc,cuda-fat-build. The pinnedubuntu-24.04-armlaneskeep
/tmpon the root disk already. Theagent-recordjob additionallykeeps the reclamation step (Android/.NET/GHC/JVM trees, stale docker
layers): with
/tmpback on the root disk it buys the headroom the suitesand compilers consume.
The worker's 20 GiB floor is unchanged -- the fix removes the pressure,
not the check. The spec and its correction are committed first:
.agents/specs/ci-agent-record-disk.md.Named local gate failure:
check-windows-portability.pyfails on thishost's broken dotnet SDK (/opt/dotnet-sdk-bin-10.0/dotnet exits 1), a
pre-existing environment breakage unrelated to this workflow-only change.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]