Repository navigation
feat(standard): choose a benchmark's own agents like any agent, one job each - #246
Merged
Merged
Conversation
OSWorld 1 and 2 kept their upstream's agent scripts as a benchmark option, written three times (options.agent, agents.json, setups.original.models). Each agents.json entry is now an agent definition that starts with a new, optional start kind of nullius.agent/v1, start.driver: the upstream's own value and the script, argv and env its driver runs. Where an agent comes from and a benchmark agent's only_for are derived from where it sits. The loader rebuilds options.agent's values, names, about and default from the file, so drivers, --bench-option agent=... and kept run scripts see the option as before; problems() keeps the entries' models within the Original setup's. match.fit binds a benchmark's own agent to its benchmark and checks no port. The Agents page lists them behind a From filter whose default leaves them out; their card shows the benchmark and its published models.
…ctor OSWorld's agents were one select in its Original setup's tab, one value for the whole recipe. They are now tiles of the Agent step, first, in place of the benchmark's Original tile; several may be chosen, a job each, and a setting that names one of them (M3's thinking) is in that agent's tab. Other benchmarks' own agents are greyed with the benchmark they belong to, and a filter hides them. A pick <benchmark>:<key> compiles through the Original setup with the upstream's own value as its agent option: one pick's command is the one the agent option gave. Like the paper names the agent a model is not published with. A recipe from before (original with bench_options.agent) is read as that agent by standardFromJSON and by the server's compile.
Every OSWorld job recorded its agent as original, the setup, so Compare read two of OSWorld's agents with one model as one participant. A setup run now records the benchmark's own agent that ran (setup.agent) and names the job by its key (agent_info.name, agents[].name, the live feed), its setup still original. parse names an older job, recorded as original with its agent option, the same way through agents.json, so old and new runs of one agent share a column.
…nder the model setup Under the model setup, setup.asked was setups.model.runs_as (MiniMax-M3) whatever the agent, while OSWorld's driver asks the agent's own first model. The setup now records that model for the agent that runs.
… they run A Runs on filter beside From, Protocol and Proves: its choices are the benchmarks, each entry's values the ones it fits, read from the same fits the Benchmarks it runs count shows. OSWorld1 lists its own agents and the one catalogue agent that proved the computer port, as the builder enables them. The capabilities page has it too.
… script OSWorld1's own agents read reference on their tiles while another benchmark's own agents read its own script: the same kind of agent now reads its own script wherever it is shown.
…spilling it A benchmark's own agent's summary names its script, one long word such as run_multienv_muse_spark.py, which ran past the card's edge. The toolkit's card summary now breaks such a word, on every card page.
…own agents are chosen Each of a benchmark's own agents compiles through the Original setup, and each found no model: two of OSWorld's agents read "Choose a model. Choose a model." on the Model step. compile now records each problem once.
4 of 18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #245
What
OSWorld 1 and OSWorld 2 ship several agent scripts of their own. They were one
agentoption of the Original setup, one value per recipe, and every job recorded its agent asoriginal. They are now agents: listed on the Agents page, chosen in the builder's Agent step, and named in their jobs. Nullius still runs the upstream's own scripts, unchanged.osworld1/agents.jsonandosworld2/agents.jsonis anullius.agent/v1definition withkey,label,summary,modelsand a new optional start kind,start.driver.start.driver.agentkeeps the upstream's own value (openai_cua); the rest is what the driver runs (script,argv,env). Keys are dashed (openai-cua) and the ref is<benchmark>:<key>(osworld1:openai-cua). Where an agent comes from (from: bundled, yours, benchmark) and itsonly_forare derived from where its definition sits, never written.standard.jsonkeeps onlyoptions.agent'slabel,summaryandsetups; the loader rebuildsvalues,names,aboutanddefaultfrom the file, so drivers,--bench-option agent=...andNULLIUS_OPTION_AGENTread as before.problems()refuses a driver agent withoffersorproven, a missing field, a model outsidesetups.original.models, anonly_fornaming another benchmark, and option values written beside the file.match.fitbinds an own agent to its benchmark ("OpenAI CUA belongs to OSWorld1") and checks no port. The Agents page lists them behind a From filter whose default leaves them out; their card shows the benchmark, linked, and the published models.agentselect. Several may be chosen, one job each. A setting that names one agent (M3'sthinking) is in that agent's tab. Other benchmarks' own agents are greyed with the benchmark they belong to, and a chip hides them. A pick compiles through the Original setup with the upstream's value as its agent option: one pick gives the command the option gave, byte for byte. Like the paper names the agent that a model is not published with. A recipe from before (originalwithbench_options.agent) is read as that agent, bystandardFromJSONand by the server's compile; withoutagentit is the first agent, and an unknown value is a compile error that names it.config.standard.setup.agent,agent_info.name,agents[].nameand the live feed carry the agent key (openai-cua); the setup staysoriginal.parsenames an older OSWorld job the same way from its recorded agent option, so old and new runs of one agent share a Compare column.modelmethod,setup.askedis the model the agent is asked as. It wassetups.model.runs_as(MiniMax-M3) for every agent, while the driver asks the agent's own first model.fitsthe builder greys with. For OSWorld1 it keeps its 12 own agents and Codex, the catalogue agent that proved the computer port.run_multienv_muse_spark.py) instead of running past the card's edge, on every catalogue card page.Out of scope: τ³-bench's
upstream_agentoption, and Harbor's reference agent bound to one benchmark. The builder's Agent step grid will be redrawn in a later change.Touches
standard (
standard/definitions.py,match.py,recipe.py,job.py,run.py,cli.py) · runs (runs/parse.py) · catalogue (osworld1/,osworld2/:agents.json,standard.json,driver.py) · routes (web/agents.py) · templates (agents/standard.html,_cards.html,_tables.html) · static (catalogue/catalogue.css) · builder (command-builder-core.js,command-builder.js) · docs (standard.md,builder.md,agents/add-manually.md,field-guide.md, generated Supported pages) · skills (import-standard,import-benchmark,remove-integration) · testsBefore and after
Verified
In a rootless Podman container (
python:3.12):python3 app/tests/parallel.py: 3099 tests in 157 modules, OK.pip install -r app/requirements.txt, thenpython3 app/tests/parallel.py app/ui_tests: 543 tests in 55 modules, OK.Each of the first six commits passes both suites on its own.
python3 app/tests/catalogue_docs.py --writeandpython3 app/tests/field_contract.py --writeleave the tree unchanged.app/ui_tests/test_recipe_execution_fixtures.pyand its fixtures are unchanged.New tests:
test_benchmark_agents.py: the entries load as agents bound to their benchmark; the option is rebuilt from the file; each refusal ofproblems(); the fits table; an own agent compiles to the command of today's agent option; a recipe from before, withoutagent, and with an unknown value; two agents give two jobs named by their keys; Like the paper with a model the agent is not published with; two of its agents without a model give one error, not one each; another benchmark's agent is refused with its reason; a job names its agent, an older job is read the same way;setup.askedunder themodelmethod for a non-M3 agent.thinkingin M3's tab and noagentoption; other benchmarks' own agents greyed; a recipe from before read back by the builder.In a browser, on a local server with an empty profile, at 1280 px: the Agents page with the From and Runs on filters, light and dark, and the builder's Agent and Review steps for OSWorld1 with two own agents chosen. No page error.
Each OSWorld
driver.pychanges by three lines: on load it keysagents.json's entries bystart.driver.agentagain, so the rest of the driver reads the shape it read before. The suite loads both drivers and checks that their keys are the agent option's values (test_osworld2_muse.py) and that each OSWorld1 agent'sargvgives its own step limit (test_standard_bench_options.py).Not checked: a real OSWorld run. For one agent, the compiled command is the one the agent option gave.
Risk
original, and older ones are read the same way, so anoriginalcolumn for OSWorld becomes one column per agent.original. A single pick keeps its name.osworld1:<key>, which main does not read. To undo, revert the commits; recipes saved in the old shape compile on both.