Skip to content

Judge: two overlapping saves to one trial's judgement lose an answer #247

Closed

Description

@harenome

Priority

low. Two saves to the same trial and the same judge must overlap. A batch judges each trial once, so this needs two batches over the same trials at once, for example Judge all clicked again or in a second tab while the first batch runs. The cause is in the backend (app/nullius/judge/store.py), and the effect is lost answers.

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Evidence

store.save (app/nullius/judge/store.py, lines 70-92) reads the trial's judgement file, merges the new answers into the question version, writes the whole file to <file>.tmp, then renames it over the file. Nothing stops a second save between the read and the rename, and every save uses the same temporary name. web/judge.py runs a batch 8 calls at a time (CONCURRENCY = 8).

The test below makes two saves read the file before either writes. On main (56ca756), in a python:3.12 container, it fails on each of three runs:

AssertionError: Tuples differ: (["FileNotFoundError(2, 'No such file or directory')"], ['q2']) != ([], ['q1', 'q2'])

One save renames the shared .tmp away, so the other raises FileNotFoundError on its own rename. The first save's answer is not in the file.

Screenshot

No response

Reproduction

  1. On main, add app/tests/test_judge_store_race.py:
import json, shutil, tempfile, threading, unittest
from unittest import mock

from nullius.judge import store


class OverlappingSaves(unittest.TestCase):
    def test_two_overlapping_saves_keep_both_answers(self):
        ws = tempfile.mkdtemp(prefix="nullius-judge-race-")
        self.addCleanup(shutil.rmtree, ws, True)
        both_read = threading.Barrier(2, timeout=5)
        real_load = json.load

        def load_then_wait(fh, *a, **k):
            data = real_load(fh, *a, **k)
            both_read.wait()               # each save has read the file before either writes
            return data

        store.save(ws, "job", "t1", {"questions": "v1", "answers": {}})
        errors = []

        def save(question):
            try:
                store.save(ws, "job", "t1", {"questions": "v1", "answers": {question: "yes"}})
            except Exception as e:
                errors.append(repr(e))

        with mock.patch.object(store.json, "load", load_then_wait):
            threads = [threading.Thread(target=save, args=(q,)) for q in ("q1", "q2")]
            for t in threads:
                t.start()
            for t in threads:
                t.join()
        answers = store.load(ws, "job", "t1", "v1")["answers"]
        self.assertEqual((errors, sorted(answers)), ([], ["q1", "q2"]))
  1. From app/, run python3 -m unittest tests.test_judge_store_race.
  2. It fails: one save raises FileNotFoundError and its answer is missing.

Expected

Two saves to one trial's judgement that overlap both succeed, and the file keeps both answers.

Acceptance

  • store.save writes through a temporary file of its own (for example tempfile.mkstemp in the same folder), never a shared .tmp name.
  • The read, merge and write of one judgement file happen under a lock, so a second save reads the first one's result.
  • The test above is added and passes.
  • Saves to different trials still run in parallel.

Dependencies and related

None found.

Activity

  1. added
    bugSomething isn't working
    P3When convenient: wording, layout, tests
    on Nov 20, 2026
  2. added theissue type on Nov 20, 2026
  3. harenome commented on Nov 20, 2026

    @harenome
    CollaboratorAuthor

    More evidence on main (bdcf918), for concurrent saves only.

    Triggers in normal use. Each of these writes one judgement file from two requests at once:

    • Judge all on a job page, and Judge on the trial page of one of its trials. Judge all runs inside its request for minutes on a large job (CONCURRENCY = 8, app/web/judge.py, line 28).
    • Judge all on a job page, and Judge on a Compare page that has that job as a column. judge_side writes the trial's own record (line 453).
    • The task batch for two jobs of one dataset. Both write judgements/task_<dataset>/<task>.json (task_key, lines 563-566).

    Stress result. A script runs 30 rounds. In each round, 8 threads start at a barrier and save a different question to one trial's file. The file already holds a record under the earlier question version. Ten runs in a python:3.12 container, 240 saves each:

    lowest highest
    saves that raised FileNotFoundError 141 169
    answers missing at the end 207 210

    About one answer in eight is kept. In 3 of the 10 runs, one round left a file that json cannot read. In 3 runs, at least one round lost the earlier version's record.

    After a write that json cannot read, the next save drops every version. The script makes such a file with one extra byte. load returns None. save catches the ValueError, starts from {} and writes only the new record:

    load after a bad write: None
    versions after the next save: ['judgement.questions.v2'] {'b': 1}
    

    The file held judgement.questions.v1 and an answer a under v2 before. Both are gone.

    Label. I propose P2 in place of P3. The triggers above are ordinary use, and a bad write deletes records of earlier question versions that cannot be asked again.

    The stress script (from app/: python3 stress_judge_save.py)
    """Stress store.save: 8 threads x 30 rounds, each thread saving its own question
    to one trial's judgement at the same moment. Counts raised saves, lost answers,
    and rounds that left a file json cannot read."""
    import json, shutil, sys, tempfile, threading
    
    sys.path.insert(0, ".")
    from nullius.judge import questions as Q, store
    
    THREADS, ROUNDS = 8, 30
    ws = tempfile.mkdtemp(prefix="nullius-judge-stress-")
    errors, lost, corrupt, versions_dropped, kinds = 0, 0, 0, 0, set()
    for r in range(ROUNDS):
        trial = f"t{r}"
        store.save(ws, "job", trial, {"questions": Q.EARLIER[0], "answers": {"old": "kept"}})   # an earlier version's record
        gate, errs = threading.Barrier(THREADS), []
    
        def save(i):
            gate.wait()
            try:
                store.save(ws, "job", trial, {"questions": Q.VERSION, "answers": {f"q{i}": "yes"}})
            except Exception as e:
                errs.append(type(e).__name__)
    
        ts = [threading.Thread(target=save, args=(i,)) for i in range(THREADS)]
        [t.start() for t in ts]; [t.join() for t in ts]
        errors += len(errs); kinds.update(errs)
        path = store._path(ws, "job", trial)
        try:
            data = json.load(open(path))
        except ValueError:
            corrupt += 1
            data = {}
        lost += THREADS - len((data.get(Q.VERSION) or {}).get("answers", {}))
        versions_dropped += Q.EARLIER[0] not in data
    print(f"saves: {THREADS * ROUNDS}; raised: {errors}; answers lost: {lost}; "
          f"files json cannot read: {corrupt}; earlier version gone: {versions_dropped}; raised as: {sorted(kinds)}")
    
    # after a file json cannot read: load gives nothing, and the next save writes a file of one version
    p = store._path(ws, "job", "x")
    store.save(ws, "job", "x", {"questions": Q.EARLIER[0], "answers": {"old": "kept"}})
    store.save(ws, "job", "x", {"questions": Q.VERSION, "answers": {"a": 1}})
    open(p, "a").write("}")                                  # what two writers sharing x.json.tmp can leave
    print("load after a bad write:", store.load(ws, "job", "x"))
    store.save(ws, "job", "x", {"questions": Q.VERSION, "answers": {"b": 1}})
    print("versions after the next save:", sorted(json.load(open(p))), json.load(open(p))[Q.VERSION]["answers"])
    shutil.rmtree(ws)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3When convenient: wording, layout, testsbugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions