Repository navigation
Description
Priority
low. Two saves to the same trial and the same judge must overlap. A batch judges each trial once, so this needs two batches over the same trials at once, for example Judge all clicked again or in a second tab while the first batch runs. The cause is in the backend (app/nullius/judge/store.py), and the effect is lost answers.
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Evidence
store.save (app/nullius/judge/store.py, lines 70-92) reads the trial's judgement file, merges the new answers into the question version, writes the whole file to <file>.tmp, then renames it over the file. Nothing stops a second save between the read and the rename, and every save uses the same temporary name. web/judge.py runs a batch 8 calls at a time (CONCURRENCY = 8).
The test below makes two saves read the file before either writes. On main (56ca756), in a python:3.12 container, it fails on each of three runs:
AssertionError: Tuples differ: (["FileNotFoundError(2, 'No such file or directory')"], ['q2']) != ([], ['q1', 'q2'])
One save renames the shared .tmp away, so the other raises FileNotFoundError on its own rename. The first save's answer is not in the file.
Screenshot
No response
Reproduction
- On main, add
app/tests/test_judge_store_race.py:
import json, shutil, tempfile, threading, unittest
from unittest import mock
from nullius.judge import store
class OverlappingSaves(unittest.TestCase):
def test_two_overlapping_saves_keep_both_answers(self):
ws = tempfile.mkdtemp(prefix="nullius-judge-race-")
self.addCleanup(shutil.rmtree, ws, True)
both_read = threading.Barrier(2, timeout=5)
real_load = json.load
def load_then_wait(fh, *a, **k):
data = real_load(fh, *a, **k)
both_read.wait() # each save has read the file before either writes
return data
store.save(ws, "job", "t1", {"questions": "v1", "answers": {}})
errors = []
def save(question):
try:
store.save(ws, "job", "t1", {"questions": "v1", "answers": {question: "yes"}})
except Exception as e:
errors.append(repr(e))
with mock.patch.object(store.json, "load", load_then_wait):
threads = [threading.Thread(target=save, args=(q,)) for q in ("q1", "q2")]
for t in threads:
t.start()
for t in threads:
t.join()
answers = store.load(ws, "job", "t1", "v1")["answers"]
self.assertEqual((errors, sorted(answers)), ([], ["q1", "q2"]))- From
app/, runpython3 -m unittest tests.test_judge_store_race. - It fails: one save raises
FileNotFoundErrorand its answer is missing.
Expected
Two saves to one trial's judgement that overlap both succeed, and the file keeps both answers.
Acceptance
-
store.savewrites through a temporary file of its own (for exampletempfile.mkstempin the same folder), never a shared.tmpname. - The read, merge and write of one judgement file happen under a lock, so a second save reads the first one's result.
- The test above is added and passes.
- Saves to different trials still run in parallel.
Dependencies and related
None found.
Activity
harenome commentedon Nov 20, 2026
More evidence on main (bdcf918), for concurrent saves only.
Triggers in normal use. Each of these writes one judgement file from two requests at once:
CONCURRENCY = 8,app/web/judge.py, line 28).judge_sidewrites the trial's own record (line 453).judgements/task_<dataset>/<task>.json(task_key, lines 563-566).Stress result. A script runs 30 rounds. In each round, 8 threads start at a barrier and save a different question to one trial's file. The file already holds a record under the earlier question version. Ten runs in a
python:3.12container, 240 saves each:FileNotFoundErrorAbout one answer in eight is kept. In 3 of the 10 runs, one round left a file that
jsoncannot read. In 3 runs, at least one round lost the earlier version's record.After a write that
jsoncannot read, the next save drops every version. The script makes such a file with one extra byte.loadreturnsNone.savecatches theValueError, starts from{}and writes only the new record:The file held
judgement.questions.v1and an answeraunder v2 before. Both are gone.Label. I propose P2 in place of P3. The triggers above are ordinary use, and a bad write deletes records of earlier question versions that cannot be asked again.
The stress script (from
app/:python3 stress_judge_save.py)