harness-opt-bench: rename, explicit trial limits, archive, and documentation clean-up

[varunursekar]

Summary

Prepares harness-opt-bench for the next round of experiments and makes it easier to read.

• Rename harness-engineering-bench → harness-opt-bench, matching the paper. All path references follow. The W&B project name and S3 locations are unchanged on purpose.
• Explicit outer-trial limits. The build config gains fields for the optimizer trial's sandbox timeout, idle timeout, agent clock, internet access and resources, rendered into the compiled task and forwarded to Harbor. Every benchmark now states 86400 / 3600 / 72000 / true; the values match what previously applied implicitly, so measured behaviour is unchanged. A cut-off optimizer is now scored rather than lost.
• Archive the three benchmarks not reported in the paper (swe-atlas-qna, tau3, swe-bench-pro) under harness-opt-bench/archive/, with a README stating why each is out. Docs, CI matrix and config tests describe the four reported benchmarks.
• Scripts. scripts/ keeps only what defines or reproduces the benchmarks: split generation, task-data status, baseline re-scoring, per-trial token attribution. Team-specific operator scripts leave the repository; recover_producer_tokens.py (forensics for a fixed metering bug) is deleted.
• Documentation. New README with the paper title, arXiv badge and architecture figure; CONFIGURATION.md rewritten around the evaluation protocol, shared defaults, per-benchmark values and change rules; the run-benchmark runbook rewritten for a general reader.
• MODAL_ENVIRONMENT is declared among each build's secrets so the launcher and sidecar place sandboxes in the same environment.

Verification

• tests/test_v05_benchmark_configs.py, tests/test_v05_harbor_build.py, tests/test_v05_cli.py pass.
• Every build compiles; the outer task.toml carries the new limits.
• A GAIA cell launched from this stack ran end to end (optimizer, evaluations, agent clock hand-over to the verifier, finalization, session archive).

Also included, via #89

Merged into this branch so the two land together; #89 has its own review record.

• Reproducible compiled tasks. Nothing run-specific is written into a compiled task: the gateway scope tokens and the producer allow-list arrive by environment variables that vero harbor run sets per launch, read once at service start. The baseline commit carries a fixed date, so two compiles of one config are byte-identical. vero harbor build --manifest records a checksum manifest and --check fails on drift; one manifest is committed per reported benchmark.
• vero_requirement lets a build pin a published scaleapi-vero==<version> so the images install it instead of copying the source tree. Unset for now.
• Target-model routing. Target models are named without a provider prefix and mapped to a deployment through model_aliases on the evaluation scope (--param target_model_route=). The Terminal-Bench variant becomes build.routed.yaml.
• inner_env is required; rescore_candidate.py understands required parameters and applies the same target routing.
• Build files and docs rewritten in plain language, with the shared field explanations consolidated into a Deployment settings section of CONFIGURATION.md instead of repeated in every build file.
• A small GAIA variant (build.shell.e2e.yaml) exercises the whole trial in under an hour; its score is not a result. It ran live end to end: optimizer stopped by its clock, verifier scored the submitted candidate, error block empty, both session archives present.

Review findings from Greptile on both PRs are addressed in the final commits.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new changes or outstanding review findings remain.

Summary

• Adds explicit optimizer-trial clocks, sandbox limits, internet policy, and resource settings.
• Makes compiled tasks reproducible while moving launch-specific credentials and routing configuration to runtime environment variables.
• Adds manifest generation and drift-checking support, published-package pinning, required inner-environment configuration, and target-model routing.
• Updates the four reported benchmark definitions, configuration tests, and operational documentation.
• No changes were made after the previous review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  B[Benchmark build.yaml] --> C[vero harbor build]
  P[Partitions and seed target] --> C
  C --> T[Deterministic compiled Harbor task]
  R[Runtime environment variables] --> L[vero harbor run]
  T --> L
  L --> O[Outer optimizer trial]
  O --> E[Nested benchmark evaluations]
  E --> F[Trusted final scoring]
  F --> A[Result and session archives]

Loading

_{Reviews (6) · Last reviewed commit: "Drop the committed compile manifests"}

[varunursekar]

@greptile review