Reproducible compiled tasks, target-model routing, and benchmark documentation clean-up - #89
Merged
Merged
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
varunursekar
force-pushed
the
rebuttal-freeze
branch
from
September 13, 2026 02:44
5f4b0ba to
026040e
Compare
Nothing per-run is baked any more. The three scope tokens and the producer allow-list arrive by environment (VERO_PRODUCER_TOKEN, VERO_EVALUATION_TOKEN, VERO_FINALIZATION_TOKEN, VERO_PRODUCER_SCOPE): vero harbor run mints or chooses them per launch, task.toml passes them to compose like secrets, and the gateway and sidecar read them once at container start, so a later change to the launcher's environment cannot move a running gateway's allowances. The baseline commit is stamped with a fixed date so its hash depends on the seed alone. Two compiles of one config are now byte-identical. vero harbor build --manifest writes a SHA-256 manifest (one hash each for the task-data and vero-source subtrees); --check recompiles and fails on drift. A compiled.manifest.json is committed for every baseline that compiles. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New build field vero_requirement (an exact pin such as scaleapi-vero==0.6.0). When set, the compiler skips the source copy and both images install that requirement with the extras each needs; the compiled task then holds no vero and is small enough to commit (GAIA shell: 1.5 MB against 6.4 MB). The pin must equal the vero doing the compiling, since the images read the configs it wrote; a mismatch is refused at compile time. Unset everywhere for now: 0.5.0 on PyPI predates this branch's gateway and outer-limit changes, so a task compiled against it would not start. The GAIA yamls carry the line commented out with that explanation. Manifests are regenerated after the rebase onto the renamed package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Harbor hands the Modal SDK no environment name, so sandboxes went to the workspace default (main), where every cell before 2026-09 ran. Declaring MODAL_ENVIRONMENT among each build's secrets carries it to the launcher and the sidecar alike; the env files set it to enterprise. Manifests regenerated for the extra task.toml line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build.shell.e2e.yaml is build.shell.yaml with partitions-e2e (2/3/3 cases), six-run budgets, rescore_top_k 1, and optimizer_agent_timeout_seconds 1500, so a trial reaches finalization inside an hour. It exercises compile, tokens, evaluations, the agent clock handing over to the verifier, and the session archive after a vero change. Its score is never reported. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…2e yaml comment A plain-language section covering deterministic compiles, the committed manifests and --check, per-run inputs arriving through the environment, and the vero_requirement pin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Build files, READMEs and the runbook are rewritten in plain language. The evaluation environment is a required parameter (inner_env), target models are named without a provider prefix and mapped to a deployment through the evaluation scope's model_aliases (target_model_route), and the Terminal-Bench variant becomes build.routed.yaml with the route as a parameter. Kept as they were: the harbor extra the evaluation environment needs (now overridable), the secrets list and optional telemetry block, and the sandbox options that group and reclaim evaluation sandboxes. rescore_candidate.py learns the required-parameter placeholder and applies the same target routing. Manifests regenerated with inner_env=modal. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The seven build files repeated the same twenty comment lines. Their meaning now lives in a Deployment settings section, and each build file keeps a pointer plus the notes specific to its benchmark. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vero_requirement accepts only an exact pin of the VeRO distribution, and the compiler refuses a pin it cannot check against the installed version. rescore_candidate.py resolves build parameters from --param and then the environment, as the build loader does. The archived builds get back the sandbox options the rewrite dropped, including the keepalive the SWE-Atlas images need. One generic comment left in the GAIA builds is removed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
varunursekar
force-pushed
the
rebuttal-freeze
branch
from
September 13, 2026 02:48
026040e to
11fd44f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #88. Makes a compiled optimization task reproducible and verifiable, and finishes the documentation clean-up.
vero harbor runsets per launch; the gateway and sidecar read them once at start. The baseline commit carries a fixed date, so two compiles of one config are byte-identical.vero harbor build --manifestrecords a SHA-256 manifest of the compiled task;--checkrecompiles and fails on drift. One manifest is committed per reported benchmark.vero_requirement. A build may pin a publishedscaleapi-vero==<version>so the images install it instead of copying the source tree. Unset for now; the pin must equal the compiling version.model_aliaseson the evaluation scope, overridable with--param target_model_route=. The Terminal-Bench variant becomesbuild.routed.yamlwith the optimizer route as a parameter.inner_envis required, andrescore_candidate.pyunderstands required parameters and applies the same target routing.CONFIGURATION.md(including a section on compiled tasks), the runbook, and comments in every build file.build.shell.e2e.yaml, eight cases, 25-minute optimizer clock) exercises the whole trial in under an hour. Its score is not a result.Verification
vero_requirement, and the sandbox-flag plumbing.--check --param inner_env=modal.🤖 Generated with Claude Code
The PR appears safe to merge; no outstanding or newly introduced actionable failures remain.
Summary
environment/vero/checksum consistently across GAIA, OfficeQA, BrowseComp-Plus, and Terminal-Bench.Reviews (3) · Last reviewed commit: "Address review findings on the compiled-..."