Skip to content

feat(runs): add the run-to-fix skill, from a finished run to checked fixes and a compared rerun - #244

Open
opPO333 wants to merge 101 commits into
ThinkFlowLab:mainfrom
opPO333:feat/run-to-fix-skill
Open

opPO333 wants to merge 101 commits into
ThinkFlowLab:mainfrom
opPO333:feat/run-to-fix-skill

Conversation

@opPO333

@opPO333 opPO333 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #241

What

The skill run-to-fix under .claude/skills/run-to-fix/ takes a finished run to checked fix candidates and a
compared rerun. A round has five stages (read, classify, check, select, compare) and stops twice for a person: which
candidates to keep, and what runs next at what price. The script scripts/loop.py counts and keeps the records in a
loop folder the person names. The assistant reads trials, names causes and writes tests and fixes in the agent's
code. Everything that belongs to one agent lives in one adapter file (references/adapter.md). The skill calls
runs.py of run-analysis for health, metrics and the round page, and imports nothing from app/. AGENTS.md
gets one paragraph on the skill.

Review order: SKILL.md, references/adapter.md, scripts/loop.py, then the stage modules in stage order.
Feedback wanted on the adapter contract and on the price of a trial run.

Why one PR: the change is one new folder that no other code imports, and the author asks the reviewer to agree to one pull request.

Size: verdict: SPLIT: code lines 2488 > hard 500 (support lines 3694, 28 files in 2 areas).

Touches

CI and skills: the new folder .claude/skills/run-to-fix/ and one paragraph of AGENTS.md. No file of the
application changes.

Verified

Merge base 0e6ae8cff683 (main), head 7de82f9c7096. python3 -m unittest discover -s .claude/skills/run-to-fix/tests from the repository root: 184 tests, OK, on Python 3.12.3. Each change of code has
tests that failed on the commit before it.

acceptance box of #241 what proves it
1. Only the folder and AGENTS.md change git diff --stat main...HEAD: 27 files under the folder, AGENTS.md with 13 added lines
2. Standard library only the run above; an import scan finds no other module
3. The same classes on both run paths test_classify.BothRunPaths, test_classify.RetriedTries
4. No agent in the scripts test_classify.Classes, test_compare.Verdicts; a search for agent and benchmark names finds none
5. Paid commands are written and not started test_check.CheckFive, test_decide.Select, test_check.PaidScriptsRun
6. No budget has a default test_read.BudgetHasNoDefault, test_decide.TrialBudgetHasNoDefault
7. The checkout stays unchanged test_check.Check, test_check.Cancellation
8. Noise at p of 0.05 or more test_compare.Verdicts
9. The loop folder lies outside git test_read.LoopFolderInARepository
10. Rounds on real runs both below; on the standard path, stages 1, 2 and 5

Seven rounds of independent review found 8, 5, 3, 2, 2 and 1 defects; the seventh found none. Each fix followed a
test that failed before it: unknown and invalid costs (test_decide.IncompleteCosts, RecordedAmounts), the tasks
and the commit of a trial run (TrialRunIdentity, SelectionCoverage), checks of another adapter
(test_check.StaleEvidence, test_decide.ChecksOfAnotherAdapter), unit ids of the standard (StandardUnitIds),
written commands and a stopped loop (test_check.PaidScriptsRun, Cancellation), a found class behind a link
(test_classify.FoundClassConfinement). Stop 2 counts the trials per task over the round's trial jobs on the
selected commit. compare counts the trials of its new job alone and gives the targets no test below 3 trials per
task.

On recorded runs of a real agent, two Harbor jobs: read, classify and compare give the counts of
run-analysis and of a count by hand, class by class. With the agent's own adapter, candidate, check of two
candidates, decide --stop 1 and the proposal of select match a recount by hand.

On two real jobs of the standard path, an agent of the catalogue on the container backend: read, classify with
two found classes and compare match a count by hand. The round found four gaps, fixed here with tests: model calls
from the standard's n_model_calls, output_limit named as switched off without token counts, unit ids in the task
lists, an adapter with no source.

Not verified: stages 3 and 4 on the standard path; a retried/ folder on a real job (a test covers it); a trial
run with its comparison on a real agent (select --size, a build, decide --stop 2, compare); Python versions
other than 3.12.

Risk

Door: two-way. A revert removes one folder and one paragraph of AGENTS.md; no other code imports the skill. It
leaves what the skill wrote outside the repository: loop folders, written paid commands, and a worktree left by a
loop that was killed with SIGKILL, until git worktree remove --force <path> removes it.

  • read decides that a run is not analysable from the opening words of the verdicts of runs.py health. A change
    of that wording makes read accept a run it should refuse, and test_read.ReadRefuses fails. classify and
    compare stop when their counts of trials, passes and pairs differ from those of runs.py.
  • check runs the adapter's commands with bash in detached worktrees of the agent's checkout. It removes each one
    when its check ends, on failure, on SIGTERM and on Ctrl+C.
  • The commit a trial run ran is the person's statement, since a job does not record its code. A wrong statement
    makes a trial run of other code count for the selected commit.

The adapter is the one file that holds everything about an agent: its
checkout, its commands with their placeholders, the limits of the run and
the commands that cost money. The loop checks each ref in a detached
worktree and removes it when the check ends.
read runs health and analyze of run-analysis, refuses a run that is not
analysable or belongs to another agent, and records the round's input.
classify gives each trial one of seven classes and a signature, groups
the failing trials by signature with their warnings, and adds columns
for found classes and for the agent's own classes.
compare gives the pass rates with their standard error, the won and lost
tasks with an exact two-sided binomial test, the targets' trials with
Fisher's exact test, and the stop rule, then closes the round. read asks
for the person's words before a round that follows a stop. Each stage
lives in its own module, and loop.py parses the command line.
candidate registers a fix of a cause and the trials it predicts. check
runs the cause's test before and after the fix, the agent's suite
against its failures on the base ref, and the replay on both refs, each
in a worktree that is removed when the check ends; a paid command is
written for a person to start. decide writes the pages of stop 1 and
stop 2, answer stores the person's words, and select prices the trial
run, builds it and writes its run command.
decide --stop 2 adds each trial job to the round's list in the ledger.
The money spent in select, decide and compare comes from that list, and
the exact test of the targets reads the trials of every trial run on the
same code.
Two exceptions that differ only in the model text they quote now share a
signature. The mask runs before the detail is cut, a quote that is not
closed masks to the end, and an apostrophe inside a word masks nothing.
Each group keeps the detail of its first trial as written in example.
The baseline warning names the benchmark only for groups of 3 tasks or
more.
… fixes

The made-up trials get their own costs, failed ones more than passing
ones, and the expected prices follow the price of each task. New tests
cover the cheapest group of the stop rule, a task without a cost, a
trial job named as PATH::JOB, and a result that cannot be written.
A run of chosen tasks costs the sum over its tasks of the mean recorded
cost of each task's trials, times the trials it gets. The cost comes from
this round's trial runs on the same commit when they hold the task, else
from the run that was read; compare prices from the new run. A task with
no cost takes the mean of the others, and the line counts such tasks.
select, decide --stop 2 and the stop rule share one function, and the
stop rule names its lowest-priced group.

decide --stop 2 reads the job folder that health found, so PATH::JOB
works. compare writes result.json before the ledger entry. stats.py
loses standard_error, which had no caller.
read asks git, before it writes anything, whether the loop folder lies in
a work tree and whether that repository ignores it. A folder outside
every repository, or an ignored one, is read; any other is refused with
the repository root and the two ways out. Any other answer of git stops
read with git's output.
…of its own

source in adapter.json may be null. read, classify and compare never open
the agent's code and run without it; candidate, check, select and
decide --stop 2 refuse with a message that names source. An adapter with
no source refuses every command that runs on the code; commands.classes
then runs in the loop folder.
…gent with no checkout, and what a round counts

classes.md: model calls from n_model_calls when result.json holds it,
output_limit switched off when no trial records a reply's tokens, and
found classes counted on the round's run. adapter.md: a null source and
unit ids in the units file. files.md: a round's own run is not money it
spent.
…e amount of 0 or more

A trial's cost_usd, the budget in input.json and the cost of each trial
run in ledger.json are read only as finite amounts of 0 or more; any other
value stops the loop with the file, the trial and the value named, before
a ledger entry or a command is written.
…in the answer and in select

checks.json records the SHA-256 of the adapter it was made with. decide
--stop 1, answer --stop 1 and select read a check only under that same
adapter and otherwise ask for loop.py check again.
…ts count over the round's trial jobs

decide --stop 2 refuses a job that lacks a guard or a canary, and a job whose
tasks have unequal numbers of trials. It counts the trials per task over the
round's trial jobs on the selected commit, so a runner that gives one trial
per job is started three times. trial.json and the page say how many trials
per task the jobs hold together and whether the selection is complete; the
targets are tested only when it is. The rerun option runs every selected task.
… a task with none or with two

run_names gives each task the one unit id its trials give on a job of the
standard. A task whose trials give no unit id while other trials of the job
give one, or two unit ids, is refused before trial-tasks.txt, the ledger or a
command is written. A job of a Harbor recipe keeps its task names.
…tial count of output tokens, and the checkout first at stop 2
…heck the checkout first at stop 2, describe model_calls

rules_off says how many trials record no tokens of one reply when the
adapter sets an output limit. decide --stop 2 checks for the agent's
checkout before it reads the trial job. classes.md and files.md say that
model_calls in trials.csv is the standard's own count where a result holds
one, and that the run-analysis page counts from the trajectory.
…n compare, unit ids from the job's runner, the ledger unchanged on a refused cost
… by the job's runner, ledger costs checked before a save

coverage_of counts the trials per selected task once; targets_of gives no
p value and no verdict for an incomplete selection, at stop 2 and in compare.
run_names reads the runner from config.json: a job of the standard needs one
unit id per selected task, also when every id is missing, and a job of a
Harbor recipe keeps its task names. decide --stop 2 checks the costs already
in the ledger before it saves the new trial run.
…plete selection, and pin it with a test

Stop 2 counts the round's trial jobs on the selected commit together;
compare counts the trials of its new job alone. The test sends three jobs of
one trial per task through stop 2, which tests the targets after the third,
and compares the third job, which gives the targets no test.
@opPO333

opPO333 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

/judge

@github-actions

This comment has been minimized.

@opPO333

opPO333 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

/judge

@github-actions

This comment has been minimized.

The order of reading groups and the groups not read, what the budget pays for, which answers are stored as words
and which as values, who names the baseline and the pass threshold, a trial run again in another job, the file and
the commands of the stage 5 page, the verdict of compare beside the gap in standard errors, writes of adapter
commands outside the loop folder, and a round from stage 2 to stage 5. The table of files lists the logs of
classify, check and select, the paid classes command and prose.json.
A cause names a found class as its rule. compare grouped the new run by the seven classes only, and a round could
not say how many trials of the new run a cause still counts. compare now counts each rule of found-classes.json,
as the file holds it when compare runs, on the run compared against and on the new run, and writes the trials,
failing trials and unreadable files of both to found_classes of result.json. classify and compare share one count.
No file or no rule gives an empty entry; a rule with an error stops compare with the message of classify. A
result.json without the key still starts the next round.
compare counted each found class on every trial of each run. After stage 4 the new run holds the selected tasks
alone, and the two counts stood on different trials: a smaller new run read as a cause that went away. compare now
counts each rule on the trials of the tasks its verdict pairs, the tasks of solved_tasks on both runs, and each side
of a rule gives counted_trials, the trials it read. The printed line names the shared tasks. classify and
groups.json are unchanged.
…n it reads a cost from the ledger

A rule with an error in its definition stops compare before anything is written. compare reads the files of the
shared tasks only, and a file there that leaves its trial folder stops it after compare-metrics.json. A new run
that is the round's own run or one of its trial runs takes its cost from the ledger. Stage 4 says what answering
stop 1 again and selecting again in one open round replace and keep. Two tests pin the found class behind a link,
in a task the new run lacks and in a shared task.
A second answer of stop 1 replaces the kept candidates and their predictions. The selection stays on the earlier
commit until select runs again.
@opPO333

opPO333 commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

/judge

@github-actions

Copy link
Copy Markdown

🟢 PR judge · 7 of 7 yes · #244 ↔ #241

Merge risk ❔ UNDECIDED the judge gave no clear reading
Priority 🟡 MEDIUM the deepest path is .claude/skills/run-to-fix/SKILL.md
Question Verdict
Changes do what the issue asks · 10 of 10 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown –
Acceptance criteria, 10 of 10 met
# Criterion Verdict
1 .claude/skills/run-to-fix/ holds SKILL.md, the reference pages, the scripts and their tests. Outside that folder on… ✅ met (cut)
2 The scripts import the Python standard library only, and their tests pass with python3 -m unittest. ✅ met (cut)
3 A test builds the same trials as a job of a Harbor recipe and as a job of the standard and gets the same classes for bo… ✅ met (cut)
4 No script names an agent, and a test runs read, classify and compare for a made-up agent whose adapter has no command a… ✅ met (cut)
5 A test proves that a command named as paid is written to a file and not started, for a check and for a trial run. ✅ met (cut)
6 Every budget is an argument with no default value. ✅ met (cut)
7 A test proves that the agent's checkout is unchanged after a check and that no worktree is left. ✅ met (cut)
8 A test gives the verdict noise when the exact test on the won and lost tasks gives p of 0.05 or more, and improved or r… ✅ met (cut)
9 read refuses a loop folder that git would list as untracked. ✅ met (cut)
10 The Verified section of the pull request records one round on recorded runs of a real agent, and one round on a job of … ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@opPO333
opPO333 marked this pull request as ready for review November 20, 2026 22:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skills: take a finished run from its failures to checked fixes and a compared rerun

1 participant