Repository navigation
Conversation
Essoz
marked this pull request as ready for review
October 30, 2026 09:20
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents need a TrainCheck workflow that distinguishes executed invariant checks from process success, and evidence that the diagnostics help research decisions. This adds an exportable skill, typed coverage summary, bounded CPU experiment runner, artifact/provenance checks, and a completed early RSI pilot.
Four fresh agents tackled the public
adaptive-latent-reasoningtask in two control/TrainCheck pairs. Each had two 10,000-update training requests; treatment additionally had one three-update diagnostic. All eight revised training requests completed. Final selections were frozen before independent evaluation on 1,949 shared held-out expressions.The baseline scores 21.198%. Both arms improve it; this is not evidence of a TrainCheck effect. Both treatment agents report that diagnostics did not change their design or selection. The two probes took 8.955/8.935 seconds, activated all 357 rules, and reported 355/354 passes plus 2/3 violations. Two alarms also occur in healthy reference runs; the additional identity-metadata alarm remains unresolved, not a confirmed training defect. This pilot supports workflow usability, not a general agent-quality or speedup claim.
The implementation preserves failures, dormant/unknown outcomes, rule coverage, trace provenance and attempt budgets. Plain/full/selective reference runs match recorded model, gradient, optimizer and relevant RNG state. A separate phase-supervisor fix handles outer TERM/INT signals without leaving owned descendants running; two regressions fail on the baseline and four Linux tests pass with the fix. The frozen experiment runtime was not modified mid-trial.
Validation:
The initial reduced launcher omitted the upstream free syntax validator. Three setup trials are retained with costs but excluded from paired outcomes; fresh agents received the restored validator equally. After final selection, the original test sampler failed before any model evaluation because the two-operation partition has only 399 examples. The repair uses the upstream census/sampling function, preserves the failed attempt and original freeze, and verifies all 303 unaffected frozen files unchanged. These amendments and all limitations are explicit in the measurement report.
This is one reduced CPU task with two pairs, API-only diagnostics, a three-step probe horizon, and instruction-isolated agents. It is not an official RSI score or statistical evidence of benefit/harm. Recommendation: validate a task with a known training fault and demonstrated relevant invariant coverage before scaling the study.
The branch also integrates already-reviewed #163, which had entered #162's branch after #162 merged to main. No new instrumenter performance optimization is introduced here. All commits are signed. Large traces, datasets, checkpoints and raw agent artifacts remain outside Git in
outputs/rsi-pilot-20261030/; the report records hashes and reproduction locations.