Repository navigation
fix(judge): keep an earlier try's judgement when its retry is judged - #270
Merged
Merged
Conversation
A retry runs an attempt again under the same trial name. store.save kept one record per question version, so judging the retry replaced the earlier try's record and the paid judgement was gone. The record judged last stays under its version, where every read finds it as before; an earlier try's moves beside it, under `<version>@<its start>`. A read that names a start finds that try's record in either place, and judging an earlier try again merges into its own record. Files written before this change read as they are.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #254
What
A retry runs an attempt again under the same trial name. The store told the tries apart by
trial_started_at, but kept one record per question version. Judging the retry replaced the earlier try's record, and that paid judgement was lost.<version>@<its start>.loadandfor_jobwith astartedvalue find that try's record in either place. Judging an earlier try again merges into that try's own record and makes it the version's record again.Touches
backend (
nullius/judge/store.py) · testsVerified
app/tests/test_judge_store_tries.py: after a retry is judged, the first try's record is still read byloadandfor_jobwith its start, and judging the first try again merges into its own record. On the base, it fails:unexpectedly None : the first try's judgement is gone.python:3.12container with Node:python3 app/tests/parallel.py: 3118 tests in 162 modules, OK.pip install -r app/requirements.txt, thenpython3 app/tests/parallel.py app/ui_tests: 704 tests in 58 modules, OK.Risk
all_recordsnow also yields the earlier tries' records. These are real judge calls, so the cost calibration and the truncation report count them.