Skip to content

Two timing tests turn dev gates red under load: the 40ms check-clock cut and the settled-errand temp-dir teardown #1664

Description

@AbirAbbas

What happened

On 2026-09-28, two dev push gates went red in touched packages on tests the merged changes do not reach. A re-run of each failed job went green.

dev@8470cdc39 (#1650), cmd/codeaf, run 36472317002. The test body passed; the temp-dir teardown found something still writing:

--- FAIL: TestASettledErrandLeavesNoNodeOfItsOwnStillRunning (0.65s)
    testing.go:1464: TempDir RemoveAll cleanup: unlinkat /tmp/codeaf-ci/TestASettledErrandLeavesNoNodeOfItsOwnStillRunning2657892707: directory not empty

dev@73ade0354 (#1654), internal/session, run 36474315029. The cut case gives the check a 40 ms window (checkwindow_test.go:432). On a busy runner the window expires before the hanging call is even made, so the report takes the other wording:

--- FAIL: TestACheckTheClockCutNamesTheClockNotTheWork/cut (0.41s)
    checkwindow_test.go:472: the report does not lead with the clock and the call it cut:
        the check ran out of time before a call could be made — it had 40ms
        Wrote greet.go with the greeting.

Locally (8 cores, load ~0.2) on dev@73ade0354, 1 of 60 runs failed. That failure was in the control subcase, which hung for 38 s. The same test passed 30/30 on dev@e7976208e.

Replication

Deterministic (no model). Both tests use stub completers, so no key is needed. Run them under CPU pressure, for example with another package's tests running beside them:

go test -count=40 -run '^TestASettledErrandLeavesNoNodeOfItsOwnStillRunning$' ./cmd/codeaf
go test -count=60 -run '^TestACheckTheClockCutNamesTheClockNotTheWork$' ./internal/session

At idle both usually pass (40/40 and 59/60 here). Under load, expect the two failures quoted above.

Acceptance

  • TestACheckTheClockCutNamesTheClockNotTheWork: the cut case no longer depends on a 40 ms wall clock racing the scheduler. For example, inject the check clock, or wait until the hanging call has started before letting the window lapse. The control case must not hang.
  • TestASettledErrandLeavesNoNodeOfItsOwnStillRunning: whatever writes into the test's temp dir after the errand settles is waited for (or owned) before teardown, the same repair session: a stopped job reads as ended at once; the log-folder sweep moves to the registry #1641 made for the job log sweep.
  • Each passes -count=100 with another heavy package's tests running beside it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions