Skip to content

fix(judge): give every job and trial a judgement file of its own - #269

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/judge-store-keys
Nov 20, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/judge-store-keys

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #253

What

The store named a judgement file by replacing every character outside [A-Za-z0-9._@+-] with _. So task:abc and task_abc, or the agent view job~a-b and a job named job_a-b, shared a file, and one judgement replaced the other without a message. for_job returned every file in the job's folder, so a v1 pair record (pair:<a>__vs__<b>) counted as a judged trial on the job page.

  • store.key(name) keeps a name that is already safe as it is, so the usual job ids and trial names keep today's file names. A name that sanitising changes gets its safe spelling plus ~ and 10 hex characters of its SHA-256. No safe spelling contains ~, so two names never share a file.
  • Reads, and the merge that a save does first, try the new name first and then the older spelling. A store written before this change reads as it is. A file is not renamed or rewritten until that trial is judged again, and that save writes the new name.
  • for_job(ws_dir, job_id, started) now takes the job's trials ({trial name: start}, which every caller already built) and returns {trial name: record}. It reads nothing else under the job's name, so a pair record is never a trial. The job page, Judge all and the reconcile view no longer spell store keys themselves. The key= mappers of lanes.board and reconcile.reconcile are gone.

Touches

backend (nullius/judge/store.py, nullius/judge/lanes.py, nullius/judge/reconcile.py) · routes (web/judge.py, web/reconcile.py) · tests

Verified

  • New app/tests/test_judge_store_keys.py: two task names, a split agent view and a job of the lookalike name, a safe name keeps its file name, a record under the older spelling is read, listed, found by job_dirs and merged into, and a v1 pair record is not a judged trial. On the base, the three original cases fail ('second' != 'first', 'job' != 'view', ['pair_t1__vs__t9', 't1'] != ['t1']).
  • The existing store tests were updated to the trial-name keys of for_job and to store.key for a name with /.
  • In a rootless podman python:3.12 container with Node: python3 app/tests/parallel.py: 3116 tests in 161 modules, OK. pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 704 tests in 58 modules, OK.

Risk

  • Upgrade: no conversion. Ids that are already safe are unchanged. For an id that sanitising changes (an agent view <job>~<label>, a task slot task:<dataset>), the old file is still read. The first new judgement writes the new file and leaves the old one in place, so the workspace's judgement count can count that trial twice.
  • The fallback cannot tell two old names apart. Where job~a-b and job_a-b both existed before, the view still reads job_a-b's file until the view is judged again. Deleting such a view's run also removes the older folder, as it does today.
  • for_job opens one file per trial instead of listing the folder. Opening a missing file is cheap, and the job page already read every record.

The store spelled a file name by replacing every unsafe character with
`_`, so `task:abc` and `task_abc`, or the agent view `job~a-b` and a job
named `job_a-b`, wrote one file and one judgement replaced the other.
`for_job` listed every file under a job's folder, so a v1 pair record
counted as a judged trial on the job page.

`store.key` keeps a name that is already safe as it is, and adds a short
digest of the name to its safe spelling otherwise; no safe spelling holds
a `~`, so two names never meet. Reads and the merge of a save fall back
to the older spelling, so existing stores are read as they are: no file
is renamed or rewritten until it is judged again, and that save writes
the new name. `for_job` takes the job's trials and returns their records
by trial name, so the pages and the reconcile view no longer spell keys
themselves and a pair record is never a trial.
@harenome
harenome merged commit a07e534 into ThinkFlowLab:main Nov 20, 2026
2 checks passed
@harenome
harenome deleted the fix/judge-store-keys branch November 20, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge: the store's file names merge distinct trials and jobs, and a v1 pair record counts as a judged trial

1 participant