Skip to content

Fix long-document reranking verifier timeout budget - #30

Merged
hanhainebula merged 1 commit into
mainfrom
fix/verifier-timeout-budgets
Oct 25, 2026
Merged

hanhainebula merged 1 commit into
mainfrom
fix/verifier-timeout-budgets

Conversation

@hanhainebula

Copy link
Copy Markdown
Collaborator

Summary

long-document-reranking could time out during verification even when its individual stages stayed within their limits. The outer verifier allowed only 4,800 seconds, while the build, two query runs and trajectory judge allow 12,600 seconds in total.

Increase the outer window to 13,200 seconds:

1,800 build + 2 × 3,600 query runs + 3,600 judge + 600 setup/cleanup = 13,200 seconds

Update the task version to 1.0.1, synchronize the task README and verifier notes, and record the budget calculation beside the configuration. Submission stage limits, fixed inputs, determinism checks and scoring remain unchanged. Existing valid completed results retain their scores; any regrade using the expanded window should record the corresponding task version and timeout.

Validation

  • python scripts/check_tasks.py tasks/long-document-reranking: 0 errors.
  • Parsed the task configuration, grader constants and judge configuration to verify the budget calculation.
  • Compared against the base revision: only the task version, outer verifier timeout and corresponding documentation changed.
  • git diff --check: passed.

Full model evaluations were not rerun for this configuration/documentation fix.

@hanhainebula
hanhainebula merged commit ec551c7 into main Oct 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant