Skip to content

perf(judge): read the judge's calibration once per batch - #272

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:perf/judge-calibration-once
Nov 20, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:perf/judge-calibration-once

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #258

What

When its caller gives no calibration, batch.ask_and_store reads the workspace's calibration, and that read parses every judgement file in the workspace. Judge all, Compare's Judge, the task batch and the feasibility triage gave no calibration. Each batch therefore read the whole store again before each call, and the store grew by one file per call.

  • web.judge.batch_calibration() reads the calibration once per click. judge_trial, judge_side, judge_control and judge_task take it and pass it to ask_and_store.
  • With no sample yet, it is {} and not None, so that no call reads the store again. The estimate treats an empty calibration as having none.
  • A single Judge from the trial page still reads the workspace's calibration.

Touches

routes (web/judge.py, web/judge_tasks.py, web/feasibility.py) · tests

Verified

  • New app/ui_tests/test_judge_calibration_reads.py posts Judge all over a job of n trials into an empty workspace. The judge is mocked and nothing leaves the machine. The test counts the reads of judgement files. On the base: {50: 1044, 100: 4522, 200: 18968}, and the test fails against its bound of 800 for 200 trials. On this branch: {50: 100, 100: 200, 200: 400}, which is one look per trial for for_job and one per save.
  • In a rootless podman python:3.12 container with Node: python3 app/tests/parallel.py: 3118 tests in 162 modules, OK. pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 709 tests in 60 modules, OK.

Risk

  • A batch's pre-flight uses the calibration from the start of the batch. It does not learn from the calls of the batch itself. Before, it learned after each call.
  • Low otherwise. Revert the commit to undo.

ask_and_store reads the workspace's calibration when its caller gives
none, and that read parses every judgement file of the workspace. Judge
all, Compare's Judge, the task batch and the feasibility triage gave
none, so a batch walked the whole store again before each call: judging
200 trials into an empty workspace read about 19,000 files.

Each batch now reads the calibration once (`batch_calibration`) and
passes it to every call, `{}` when the store has no sample yet so no
call reads it again. A single Judge from the trial page still reads the
workspace's own. The same batch of 200 trials reads 400 files.
@harenome
harenome merged commit a3848a7 into ThinkFlowLab:main Nov 20, 2026
2 checks passed
@harenome
harenome deleted the perf/judge-calibration-once branch November 20, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge: Judge all reads the whole judgement store again for every trial it judges

1 participant