Skip to content

Add TrainCheck agent workflow and report paired RSI pilot outcomes - #164

Open
Essoz wants to merge 14 commits into
mainfrom
codex/rsi-pilot-20261030
Open

Essoz wants to merge 14 commits into
mainfrom
codex/rsi-pilot-20261030

Conversation

@Essoz

@Essoz Essoz commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Agents need a TrainCheck workflow that distinguishes executed invariant checks from process success, and evidence that the diagnostics help research decisions. This adds an exportable skill, typed coverage summary, bounded CPU experiment runner, artifact/provenance checks, and a completed early RSI pilot.

Four fresh agents tackled the public adaptive-latent-reasoning task in two control/TrainCheck pairs. Each had two 10,000-update training requests; treatment additionally had one three-update diagnostic. All eight revised training requests completed. Final selections were frozen before independent evaluation on 1,949 shared held-out expressions.

Held-out accuracy Without TrainCheck With TrainCheck Difference
Pair 1 56.769% 51.411% -5.358 percentage points
Pair 2 42.259% 48.921% +6.662 percentage points
Mean 49.514% 50.166% +0.652 percentage points

The baseline scores 21.198%. Both arms improve it; this is not evidence of a TrainCheck effect. Both treatment agents report that diagnostics did not change their design or selection. The two probes took 8.955/8.935 seconds, activated all 357 rules, and reported 355/354 passes plus 2/3 violations. Two alarms also occur in healthy reference runs; the additional identity-metadata alarm remains unresolved, not a confirmed training defect. This pilot supports workflow usability, not a general agent-quality or speedup claim.

The implementation preserves failures, dormant/unknown outcomes, rule coverage, trace provenance and attempt budgets. Plain/full/selective reference runs match recorded model, gradient, optimizer and relevant RNG state. A separate phase-supervisor fix handles outer TERM/INT signals without leaving owned descendants running; two regressions fail on the baseline and four Linux tests pass with the fix. The frozen experiment runtime was not modified mid-trial.

Validation:

  • 70 focused tests passed locally; four Linux process regressions passed on labmach. Explicit strict mypy passed 13 helper/test modules; configured hooks and diff checks passed.
  • 134 composed core regression tests passed on the integrated TrainCheck source before trials.
  • Real upstream finite-panel regression and five completed checkpoint evaluations; checkpoint/configuration/panel hashes and exact native rewards verified. Use repository Python 3.10 for the exact reward audit; Python 3.12 changes floating-point summation by one ULP on one result, documented without altering stored scores or relaxing the check.

The initial reduced launcher omitted the upstream free syntax validator. Three setup trials are retained with costs but excluded from paired outcomes; fresh agents received the restored validator equally. After final selection, the original test sampler failed before any model evaluation because the two-operation partition has only 399 examples. The repair uses the upstream census/sampling function, preserves the failed attempt and original freeze, and verifies all 303 unaffected frozen files unchanged. These amendments and all limitations are explicit in the measurement report.

This is one reduced CPU task with two pairs, API-only diagnostics, a three-step probe horizon, and instruction-isolated agents. It is not an official RSI score or statistical evidence of benefit/harm. Recommendation: validate a task with a known training fault and demonstrated relevant invariant coverage before scaling the study.

The branch also integrates already-reviewed #163, which had entered #162's branch after #162 merged to main. No new instrumenter performance optimization is introduced here. All commits are signed. Large traces, datasets, checkpoints and raw agent artifacts remain outside Git in outputs/rsi-pilot-20261030/; the report records hashes and reproduction locations.

@Essoz Essoz changed the title Add an agent TrainCheck workflow and early RSI validation Add TrainCheck agent workflow and report paired RSI pilot outcomes Oct 30, 2026
@Essoz
Essoz marked this pull request as ready for review October 30, 2026 09:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant