From 3ddd4e48bdaf3d51d260c878e0c8b4ffbfa2240e Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Wed, 28 Oct 2026 12:07:46 +0100 Subject: [PATCH] Close ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP and record the row outcome The fix landed as f484c76ad (PR #3515). The issue closes with the dated resolution evidence: the wall was the ubuntu-26.04 /tmp tmpfs, not root disk pressure; run 40637909804 proves the mask step green on agent-record and build-test-vulkan. The spec gains its ## Outcome: the rejected first hypothesis, the default values and why each holds, and what was left out of scope. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki] --- .../ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md | 8 +++--- .agents/specs/ci-agent-record-disk.md | 27 +++++++++++++++++++ 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md b/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md index 7eefd3200..fcd5c2a67 100644 --- a/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md +++ b/.agents/issues/CI-AGENT-RECORD-DISK/ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP.md @@ -1,14 +1,14 @@ ID: ISSUE-LOCAL-01M5ZEVF7HQ8SFK9F8A9WSREEP Title: agent-record flaky: strix disk-headroom exhausts on shared runner Row: CI-AGENT-RECORD-DISK -State: OPEN +State: CLOSED Kind: bug GitHub: - Mirror: PENDING Availability: FULL Created: 2026-10-27 -Updated: 2026-10-27 -Closed: - +Updated: 2026-10-28 +Closed: 2026-10-28 ## Problem @@ -16,4 +16,4 @@ The agent-record CI job fails intermittently on main (runs 40521291883, 40452744 ## Resolution -- +Landed 2026-10-28 as f484c76ad (PR #3515). The wall was not runner disk pressure on / but the /tmp tmpfs of the ubuntu-26.04 image (actions/runner-images#14777): the strix worker's headroom() measures disk_usage of self.local and self.output, both under /tmp (worker.py:226,256), so the 20 GiB floor was unmeetable on 26.04 runners regardless of how empty / was. Proof the first attempt was insufficient: run 40586147690 job 122477980562 showed 113 GiB free on /dev/root and still failed 49+2 strix tests; build-test-cpu (122477933406) and sanitize-cpu (122477933291) failed identically in the same run. Fix: mask tmp.mount and lazy-unmount /tmp as the first step of the six ubuntu-latest compile/ctest lanes. Verified green: run 40637909804 -- agent-record and build-test-vulkan passed with df -h / /tmp showing /tmp on /dev/root at 88 GiB free. diff --git a/.agents/specs/ci-agent-record-disk.md b/.agents/specs/ci-agent-record-disk.md index 0114ef916..201f19798 100644 --- a/.agents/specs/ci-agent-record-disk.md +++ b/.agents/specs/ci-agent-record-disk.md @@ -149,6 +149,33 @@ locally. workaround (mask `tmp.mount`, lazy-unmount `/tmp` as the first step) is the one this change lands, and runner-images#14814 is the image-side fix. +## Outcome + +Landed as f484c76ad (PR #3515, merged 2026-10-28T11:05Z). What settled it: + +- The first hypothesis (shared-runner disk pressure on `/`) was **rejected + by evidence from its own fix**: PR #3515's first commit freed `/` to + 113 GiB and the job still failed 49+2 strix tests. The worker measures + `/tmp` filesystems, not `/`. +- The root cause is the ubuntu-26.04 image's `/tmp` tmpfs + (actions/runner-images#14777), which makes the worker's 20 GiB floor + unmeetable by construction; the flakiness is runner-image assignment + during the 24.04→26.04 rollout. +- Defaults: the mask+lazy-unmount step lands FIRST in six ubuntu-latest + compile/ctest lanes (`agent-record`, `build-test-cpu`, `sanitize-cpu`, + `build-test-vulkan`, `build-newest-gcc`, `cuda-fat-build`), and only + those — the trivial record/document lanes do not write `/tmp`, and the + pinned `ubuntu-24.04-arm` lanes keep `/tmp` on the root disk. The + agent-record reclamation step stays because `/tmp` returning to the root + disk makes the freed space reachable by the suites and compilers. The + worker's 20 GiB floor is untouched: a safety property of the oracle + runner is not weakened to green a gate. +- Verified green on run 40637909804: `agent-record` and + `build-test-vulkan` passed with `df -h / /tmp` showing `/tmp` on + `/dev/root` at 88 GiB free; the long ctest lanes and the pre-existing + windows-msvc flint.cpp warning-as-error break are recorded separately + and do not belong to this row. + ## Stop conditions - If GitHub's runner image changes so that the reclaimed paths no longer