Repository navigation
Conversation
The adapter is the one file that holds everything about an agent: its checkout, its commands with their placeholders, the limits of the run and the commands that cost money. The loop checks each ref in a detached worktree and removes it when the check ends.
read runs health and analyze of run-analysis, refuses a run that is not analysable or belongs to another agent, and records the round's input. classify gives each trial one of seven classes and a signature, groups the failing trials by signature with their warnings, and adds columns for found classes and for the agent's own classes.
compare gives the pass rates with their standard error, the won and lost tasks with an exact two-sided binomial test, the targets' trials with Fisher's exact test, and the stop rule, then closes the round. read asks for the person's words before a round that follows a stop. Each stage lives in its own module, and loop.py parses the command line.
candidate registers a fix of a cause and the trials it predicts. check runs the cause's test before and after the fix, the agent's suite against its failures on the base ref, and the replay on both refs, each in a worktree that is removed when the check ends; a paid command is written for a person to start. decide writes the pages of stop 1 and stop 2, answer stores the person's words, and select prices the trial run, builds it and writes its run command.
decide --stop 2 adds each trial job to the round's list in the ledger. The money spent in select, decide and compare comes from that list, and the exact test of the targets reads the trials of every trial run on the same code.
Two exceptions that differ only in the model text they quote now share a signature. The mask runs before the detail is cut, a quote that is not closed masks to the end, and an apostrophe inside a word masks nothing. Each group keeps the detail of its first trial as written in example. The baseline warning names the benchmark only for groups of 3 tasks or more.
… fixes The made-up trials get their own costs, failed ones more than passing ones, and the expected prices follow the price of each task. New tests cover the cheapest group of the stop rule, a task without a cost, a trial job named as PATH::JOB, and a result that cannot be written.
A run of chosen tasks costs the sum over its tasks of the mean recorded cost of each task's trials, times the trials it gets. The cost comes from this round's trial runs on the same commit when they hold the task, else from the run that was read; compare prices from the new run. A task with no cost takes the mean of the others, and the line counts such tasks. select, decide --stop 2 and the stop rule share one function, and the stop rule names its lowest-priced group. decide --stop 2 reads the job folder that health found, so PATH::JOB works. compare writes result.json before the ledger entry. stats.py loses standard_error, which had no caller.
…ds, and braces in commands
read asks git, before it writes anything, whether the loop folder lies in a work tree and whether that repository ignores it. A folder outside every repository, or an ignored one, is read; any other is refused with the repository root and the two ways out. Any other answer of git stops read with git's output.
…of its own source in adapter.json may be null. read, classify and compare never open the agent's code and run without it; candidate, check, select and decide --stop 2 refuse with a message that names source. An adapter with no source refuses every command that runs on the code; commands.classes then runs in the loop folder.
…gent with no checkout, and what a round counts classes.md: model calls from n_model_calls when result.json holds it, output_limit switched off when no trial records a reply's tokens, and found classes counted on the round's run. adapter.md: a null source and unit ids in the units file. files.md: a round's own run is not money it spent.
…te amount of 0 or more
…e amount of 0 or more A trial's cost_usd, the budget in input.json and the cost of each trial run in ledger.json are read only as finite amounts of 0 or more; any other value stops the loop with the file, the trial and the value named, before a ledger entry or a command is written.
… in the answer and in select
…in the answer and in select checks.json records the SHA-256 of the adapter it was made with. decide --stop 1, answer --stop 1 and select read a check only under that same adapter and otherwise ask for loop.py check again.
…ats count over the round's trial jobs
…ts count over the round's trial jobs decide --stop 2 refuses a job that lacks a guard or a canary, and a job whose tasks have unequal numbers of trials. It counts the trials per task over the round's trial jobs on the selected commit, so a runner that gives one trial per job is started three times. trial.json and the page say how many trials per task the jobs hold together and whether the selection is complete; the targets are tested only when it is. The rerun option runs every selected task.
…unit ids on a job of the standard
… a task with none or with two run_names gives each task the one unit id its trials give on a job of the standard. A task whose trials give no unit id while other trials of the job give one, or two unit ids, is refused before trial-tasks.txt, the ledger or a command is written. A job of a Harbor recipe keeps its task names.
…tial count of output tokens, and the checkout first at stop 2
…new one for a round of another stop rule
…heck the checkout first at stop 2, describe model_calls rules_off says how many trials record no tokens of one reply when the adapter sets an output limit. decide --stop 2 checks for the agent's checkout before it reads the trial job. classes.md and files.md say that model_calls in trials.csv is the standard's own count where a result holds one, and that the run-analysis page counts from the trajectory.
…el_calls note without a trailing so
…n compare, unit ids from the job's runner, the ledger unchanged on a refused cost
… by the job's runner, ledger costs checked before a save coverage_of counts the trials per selected task once; targets_of gives no p value and no verdict for an incomplete selection, at stop 2 and in compare. run_names reads the runner from config.json: a job of the standard needs one unit id per selected task, also when every id is missing, and a job of a Harbor recipe keeps its task names. decide --stop 2 checks the costs already in the ledger before it saves the new trial run.
…plete selection, and pin it with a test Stop 2 counts the round's trial jobs on the selected commit together; compare counts the trials of its new job alone. The test sends three jobs of one trial per task through stop 2, which tests the targets after the third, and compares the third job, which gives the targets no test.
Contributor
Author
|
/judge |
This comment has been minimized.
This comment has been minimized.
Contributor
Author
|
/judge |
This comment has been minimized.
This comment has been minimized.
The order of reading groups and the groups not read, what the budget pays for, which answers are stored as words and which as values, who names the baseline and the pass threshold, a trial run again in another job, the file and the commands of the stage 5 page, the verdict of compare beside the gap in standard errors, writes of adapter commands outside the loop folder, and a round from stage 2 to stage 5. The table of files lists the logs of classify, check and select, the paid classes command and prose.json.
A cause names a found class as its rule. compare grouped the new run by the seven classes only, and a round could not say how many trials of the new run a cause still counts. compare now counts each rule of found-classes.json, as the file holds it when compare runs, on the run compared against and on the new run, and writes the trials, failing trials and unreadable files of both to found_classes of result.json. classify and compare share one count. No file or no rule gives an empty entry; a rule with an error stops compare with the message of classify. A result.json without the key still starts the next round.
compare counted each found class on every trial of each run. After stage 4 the new run holds the selected tasks alone, and the two counts stood on different trials: a smaller new run read as a cause that went away. compare now counts each rule on the trials of the tasks its verdict pairs, the tasks of solved_tasks on both runs, and each side of a rule gives counted_trials, the trials it read. The printed line names the shared tasks. classify and groups.json are unchanged.
…n it reads a cost from the ledger A rule with an error in its definition stops compare before anything is written. compare reads the files of the shared tasks only, and a file there that leaves its trial folder stops it after compare-metrics.json. A new run that is the round's own run or one of its trial runs takes its cost from the ledger. Stage 4 says what answering stop 1 again and selecting again in one open round replace and keep. Two tests pin the found class behind a link, in a task the new run lacks and in a shared task.
A second answer of stop 1 replaces the kept candidates and their predictions. The selection stays on the earlier commit until select runs again.
Contributor
Author
|
/judge |
🟢 PR judge · 7 of 7 yes · #244 ↔ #241
Acceptance criteria, 10 of 10 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
opPO333
marked this pull request as ready for review
November 20, 2026 22:03
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #241
What
The skill
run-to-fixunder.claude/skills/run-to-fix/takes a finished run to checked fix candidates and acompared rerun. A round has five stages (read, classify, check, select, compare) and stops twice for a person: which
candidates to keep, and what runs next at what price. The script
scripts/loop.pycounts and keeps the records in aloop folder the person names. The assistant reads trials, names causes and writes tests and fixes in the agent's
code. Everything that belongs to one agent lives in one adapter file (
references/adapter.md). The skill callsruns.pyofrun-analysisfor health, metrics and the round page, and imports nothing fromapp/.AGENTS.mdgets one paragraph on the skill.
Review order:
SKILL.md,references/adapter.md,scripts/loop.py, then the stage modules in stage order.Feedback wanted on the adapter contract and on the price of a trial run.
Why one PR: the change is one new folder that no other code imports, and the author asks the reviewer to agree to one pull request.
Size:
verdict: SPLIT: code lines 2488 > hard 500(support lines 3694, 28 files in 2 areas).Touches
CI and skills: the new folder
.claude/skills/run-to-fix/and one paragraph ofAGENTS.md. No file of theapplication changes.
Verified
Merge base
0e6ae8cff683(main), head7de82f9c7096.python3 -m unittest discover -s .claude/skills/run-to-fix/testsfrom the repository root: 184 tests, OK, on Python 3.12.3. Each change of code hastests that failed on the commit before it.
AGENTS.mdchangegit diff --stat main...HEAD: 27 files under the folder,AGENTS.mdwith 13 added linestest_classify.BothRunPaths,test_classify.RetriedTriestest_classify.Classes,test_compare.Verdicts; a search for agent and benchmark names finds nonetest_check.CheckFive,test_decide.Select,test_check.PaidScriptsRuntest_read.BudgetHasNoDefault,test_decide.TrialBudgetHasNoDefaulttest_check.Check,test_check.Cancellationtest_compare.Verdictstest_read.LoopFolderInARepositorySeven rounds of independent review found 8, 5, 3, 2, 2 and 1 defects; the seventh found none. Each fix followed a
test that failed before it: unknown and invalid costs (
test_decide.IncompleteCosts,RecordedAmounts), the tasksand the commit of a trial run (
TrialRunIdentity,SelectionCoverage), checks of another adapter(
test_check.StaleEvidence,test_decide.ChecksOfAnotherAdapter), unit ids of the standard (StandardUnitIds),written commands and a stopped loop (
test_check.PaidScriptsRun,Cancellation), a found class behind a link(
test_classify.FoundClassConfinement). Stop 2 counts the trials per task over the round's trial jobs on theselected commit.
comparecounts the trials of its new job alone and gives the targets no test below 3 trials pertask.
On recorded runs of a real agent, two Harbor jobs:
read,classifyandcomparegive the counts ofrun-analysisand of a count by hand, class by class. With the agent's own adapter,candidate,checkof twocandidates,
decide --stop 1and the proposal ofselectmatch a recount by hand.On two real jobs of the standard path, an agent of the catalogue on the container backend:
read,classifywithtwo found classes and
comparematch a count by hand. The round found four gaps, fixed here with tests: model callsfrom the standard's
n_model_calls,output_limitnamed as switched off without token counts, unit ids in the tasklists, an adapter with no
source.Not verified: stages 3 and 4 on the standard path; a
retried/folder on a real job (a test covers it); a trialrun with its comparison on a real agent (
select --size, a build,decide --stop 2,compare); Python versionsother than 3.12.
Risk
Door: two-way. A revert removes one folder and one paragraph of
AGENTS.md; no other code imports the skill. Itleaves what the skill wrote outside the repository: loop folders, written paid commands, and a worktree left by a
loop that was killed with SIGKILL, until
git worktree remove --force <path>removes it.readdecides that a run is not analysable from the opening words of the verdicts ofruns.py health. A changeof that wording makes
readaccept a run it should refuse, andtest_read.ReadRefusesfails.classifyandcomparestop when their counts of trials, passes and pairs differ from those ofruns.py.checkruns the adapter's commands with bash in detached worktrees of the agent's checkout. It removes each onewhen its check ends, on failure, on SIGTERM and on Ctrl+C.
makes a trial run of other code count for the selected commit.