Record the optimizer harness release each reported result used, replicable with --pin-harness

[varunursekar]

Harbor installs the optimizer harness (claude-code, codex, opencode, kimi-cli, goose, mini-swe-agent) from its public registry at the start of every trial, so the release a result was produced with is otherwise only recoverable from the trial's result.json. This change records it in the benchmark definition and gives the launcher a way to replicate with it.

• Each build.yaml records optimizer_harness_versions, a map from Harbor agent name to the release the reported results used: claude-code 2.1.220, codex 0.146.0, kimi-cli 1.49.0, goose 1.45.0, mini-swe-agent 2.4.6, opencode 1.18.9 (OfficeQA), 1.18.10 (BrowseComp-Plus, Terminal-Bench), 1.18.11 (GAIA). Values must be exact releases.
• A plain vero harbor run is unchanged and installs the current release. vero harbor run --pin-harness passes the recorded release to Harbor as --ak version=, and refuses a harness the build does not record. A caller's own --ak version= selects any release and wins in both modes.
• goose also receives GOOSE_VERSION whenever a release is chosen, because its installer script installs the stable release regardless of the tag it was fetched from.
• CONFIGURATION.md gains an "Optimizer harness versions" table; the runbook describes --pin-harness.
• Tests: the launcher helper in both modes, the config validator, and a check that every build records every harness.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no actionable regression remains in the revised harness-version workflow.

Summary

• Adds exact-release validation and recorded versions to benchmark build configurations.
• Forwards recorded releases through Harbor agent arguments when pinning is requested.
• Handles Goose’s separate GOOSE_VERSION requirement while preserving caller overrides.
• Documents the workflow and adds focused configuration and launcher tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[build.yaml] --> B[Validated optimizer_harness_versions]
    B --> C{--pin-harness?}
    C -->|No| D[Install current harness release]
    C -->|Yes| E[Pass recorded release as --ak version]
    F[Caller --ak version override] --> G[Caller release wins]
    E --> H{Goose?}
    G --> H
    H -->|Yes| I[Also set GOOSE_VERSION]
    H -->|No| J[Launch Harbor]
    I --> J
    D --> J

Loading

_{Reviews (3) · Last reviewed commit: "Make the recorded harness release opt-in..."}