Repository navigation
fix(judge): give every job and trial a judgement file of its own - #269
Merged
Merged
Conversation
The store spelled a file name by replacing every unsafe character with `_`, so `task:abc` and `task_abc`, or the agent view `job~a-b` and a job named `job_a-b`, wrote one file and one judgement replaced the other. `for_job` listed every file under a job's folder, so a v1 pair record counted as a judged trial on the job page. `store.key` keeps a name that is already safe as it is, and adds a short digest of the name to its safe spelling otherwise; no safe spelling holds a `~`, so two names never meet. Reads and the merge of a save fall back to the older spelling, so existing stores are read as they are: no file is renamed or rewritten until it is judged again, and that save writes the new name. `for_job` takes the job's trials and returns their records by trial name, so the pages and the reconcile view no longer spell keys themselves and a pair record is never a trial.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #253
What
The store named a judgement file by replacing every character outside
[A-Za-z0-9._@+-]with_. Sotask:abcandtask_abc, or the agent viewjob~a-band a job namedjob_a-b, shared a file, and one judgement replaced the other without a message.for_jobreturned every file in the job's folder, so a v1 pair record (pair:<a>__vs__<b>) counted as a judged trial on the job page.store.key(name)keeps a name that is already safe as it is, so the usual job ids and trial names keep today's file names. A name that sanitising changes gets its safe spelling plus~and 10 hex characters of its SHA-256. No safe spelling contains~, so two names never share a file.for_job(ws_dir, job_id, started)now takes the job's trials ({trial name: start}, which every caller already built) and returns{trial name: record}. It reads nothing else under the job's name, so a pair record is never a trial. The job page, Judge all and the reconcile view no longer spell store keys themselves. Thekey=mappers oflanes.boardandreconcile.reconcileare gone.Touches
backend (
nullius/judge/store.py,nullius/judge/lanes.py,nullius/judge/reconcile.py) · routes (web/judge.py,web/reconcile.py) · testsVerified
app/tests/test_judge_store_keys.py: two task names, a split agent view and a job of the lookalike name, a safe name keeps its file name, a record under the older spelling is read, listed, found byjob_dirsand merged into, and a v1 pair record is not a judged trial. On the base, the three original cases fail ('second' != 'first','job' != 'view',['pair_t1__vs__t9', 't1'] != ['t1']).for_joband tostore.keyfor a name with/.python:3.12container with Node:python3 app/tests/parallel.py: 3116 tests in 161 modules, OK.pip install -r app/requirements.txt, thenpython3 app/tests/parallel.py app/ui_tests: 704 tests in 58 modules, OK.Risk
<job>~<label>, a task slottask:<dataset>), the old file is still read. The first new judgement writes the new file and leaves the old one in place, so the workspace's judgement count can count that trial twice.job~a-bandjob_a-bboth existed before, the view still readsjob_a-b's file until the view is judged again. Deleting such a view's run also removes the older folder, as it does today.for_jobopens one file per trial instead of listing the folder. Opening a missing file is cheap, and the job page already read every record.