Skip to content

test kit and CI: a command has the time its case has left; the pr tier's gate judges what it runs; the unit shards follow the longest shard - #998

Merged
Sunrisepeak merged 4 commits into
mainfrom
fix/kit-command-time-from-case
Nov 22, 2026
Merged

Sunrisepeak merged 4 commits into
mainfrom
fix/kit-command-time-from-case

Conversation

@Sunrisepeak

@Sunrisepeak Sunrisepeak commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Design 2026-11-20 §5.8 (K1) gives a command of the test kit the time its case has left. mtest 0.11.1 holds a command to that time, but the kit still passed ten minutes as each command's own, and the smaller of the two held. In the pr-tier run of #997 (44937152470, two Linux shards) sets/985 packed every system beside images and stack on a four-processor runner; the step ended at ten minutes while its suite was well within the 45 minutes it states (sets took 2140 s there, 1162 s in a run of three shards).

  • A command that states no time takes the case's remaining time (mcpplibs::mtest::time_left()); where neither the case nor a scope around it states one, ten minutes bound it, since the runner's limit would end the program without the case's report.
  • The two commands raised to 30 minutes for the same reason (images 1033, trees 955) no longer state a time; images states its suite's time from measurement (30 minutes; 1102 s in a shard of two, 707 s in a shard of three, of the run's 1200 s).
  • Cases 1141 and 1142 (e2e/test, protocol) assert the time the kit gives a command; with the previous kit 1141 fails.
  • The design record states the finding (17).

The engine and the released 2026.11.21.1 are unchanged: the pull request changes the test kit, its cases, CI and their records.

The pr tier's gate

The surface ledger's requirements are met by passing cases of the run's group, and the pr tier runs the jobs and cases a change affects. In run 44937152470 (#997, a change of release.yml alone) the gate listed 14 items only the full tier's jobs use (coff-def on Windows, dyndep --split-module and link-output --aside called from inside the build graph, reasons reached on some rows), so the gate failed on every pr run whatever the change.

  • mtest 0.11.3 (0.11.3: a ledger is judged by a set that runs every case mtest#30) judges a ledger for a set that runs every case and notes for another set that it did not; .github/versions.env pins it (the bundle's sha256 is the release's).
  • ci.yml passes the run's set to the gate (gate --set "$SET"): full judges the budget and the surface, pr the budget.
  • The contributing skill states that a pull request adding a command, flag, manifest key or reason token confirms its case with ci:full.

The unit shards' count

Main's full run of #997 (44945453323) took two Linux unit shards, 2544 and 2432 s, and trees ran past the 40 minutes it states (1369 s in a run of three shards); sets, images and stack ran in one shard together. The count was the programs' durations of main's last run summed, over what four workers do in 30 minutes: the run of three shards (1236, 834 and 1438 s) gave two. That measure assumes four workers always busy and a program's duration independent of what runs beside it; the openkal programs build in parallel within the machine's one budget, so in fewer shards each takes longer, and a shard's time is that of its longest programs and of those beside them.

  • ci-unit.yml takes the shards of main's last run, N, and the longest of them, W: N × W of shard time at 30 minutes a shard (both runs give three), at least one, at most eight; four with no history of shards. N and W are read from the stores the shard jobs kept (mtest-unit-<name>-<shard>-a<attempt>): a job's store is imported as mtest's run records, which state no shard, and the history also holds the subsystems job's runs.
  • The design record states the finding (18).

…where no scope states one

Design 2026-11-20 §5.8 (K1) gives a command of the kit the time its case has left. mtest 0.11.1
holds a command to that time, but the kit still passed ten minutes as each command's own, and the
smaller of the two held: `sets/985`'s pack of every system ended at ten minutes in a pr-tier shard
of two (run 44937152470), its suite within the 45 minutes it states. A command that states no time
now takes the case's remaining time, and ten minutes where neither the case nor a scope states one.
The two commands raised to 30 minutes for the same reason (images 1033, trees 955) no longer state
a time; `images` states its suite's time from measurement (30 minutes). Cases 1141 and 1142 assert
the time the kit gives a command. The design record states the finding (17).
…is the full tier's (mtest 0.11.3)

The surface ledger's requirements are met by passing cases of the run's group, and the pr tier runs
the jobs and cases a change affects: 14 items only the full tier's jobs use had no passing case, so
the gate failed on every pr run whatever the change (run 44937152470). mtest 0.11.3 judges a ledger
for a set that runs every case and notes it for another; ci.yml passes the run's set to the gate.
The contributing skill states that a pull request adding a surface item confirms its case with
ci:full; the design record states the finding (17).
@Sunrisepeak Sunrisepeak changed the title test kit: a command has the time its case has left; ten minutes only where no scope states one test kit and CI: a command has the time its case has left; the pr tier's gate judges what it runs Nov 21, 2026
…st of them, not from the programs' durations over four workers
@Sunrisepeak Sunrisepeak changed the title test kit and CI: a command has the time its case has left; the pr tier's gate judges what it runs test kit and CI: a command has the time its case has left; the pr tier's gate judges what it runs; the unit shards follow the longest shard Nov 21, 2026
…jobs kept, since the imported records state no shard
@Sunrisepeak
Sunrisepeak merged commit 8c44dac into main Nov 22, 2026
98 of 100 checks passed
@Sunrisepeak
Sunrisepeak deleted the fix/kit-command-time-from-case branch November 22, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant