Repository navigation
perf(judge): read the judge's calibration once per batch - #272
Merged
Merged
Conversation
ask_and_store reads the workspace's calibration when its caller gives
none, and that read parses every judgement file of the workspace. Judge
all, Compare's Judge, the task batch and the feasibility triage gave
none, so a batch walked the whole store again before each call: judging
200 trials into an empty workspace read about 19,000 files.
Each batch now reads the calibration once (`batch_calibration`) and
passes it to every call, `{}` when the store has no sample yet so no
call reads it again. A single Judge from the trial page still reads the
workspace's own. The same batch of 200 trials reads 400 files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #258
What
When its caller gives no calibration,
batch.ask_and_storereads the workspace's calibration, and that read parses every judgement file in the workspace. Judge all, Compare's Judge, the task batch and the feasibility triage gave no calibration. Each batch therefore read the whole store again before each call, and the store grew by one file per call.web.judge.batch_calibration()reads the calibration once per click.judge_trial,judge_side,judge_controlandjudge_tasktake it and pass it toask_and_store.{}and notNone, so that no call reads the store again. The estimate treats an empty calibration as having none.Touches
routes (
web/judge.py,web/judge_tasks.py,web/feasibility.py) · testsVerified
app/ui_tests/test_judge_calibration_reads.pyposts Judge all over a job of n trials into an empty workspace. The judge is mocked and nothing leaves the machine. The test counts the reads of judgement files. On the base:{50: 1044, 100: 4522, 200: 18968}, and the test fails against its bound of 800 for 200 trials. On this branch:{50: 100, 100: 200, 200: 400}, which is one look per trial forfor_joband one per save.python:3.12container with Node:python3 app/tests/parallel.py: 3118 tests in 162 modules, OK.pip install -r app/requirements.txt, thenpython3 app/tests/parallel.py app/ui_tests: 709 tests in 60 modules, OK.Risk