Repository navigation
Closed
Feature
Copy link
Labels
P0Fix now: unsafe, loses money or data, or nothing runsFix now: unsafe, loses money or data, or nothing runsenhancementNew feature or requestNew feature or request
Description
Priority
extreme. It reaches the standard's recipe compiler, matcher and job records (app/nullius/standard/), the catalogue's OSWorld entries, the Agents page and the builder.
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Today
- OSWorld 1 and OSWorld 2 model their upstream's agent scripts as a benchmark option,
options.agentinstandard.json(values,names,about,default,setups: ["original", "model"]), with the scripts' argv and env in a driver data file,agents.json. No other benchmark does this, anddocs/standard.mddoes not describe it. - The builder shows the option inside the Original setup's tab: a select of 12 agents for OSWorld 1, two value cards for OSWorld 2. The Agent step shows one tile, "OSWorld's agents" (
setups.original.reference_agent). - The agent list is written three times:
options.agent,agents.json.agents, andsetups.original.models(the union of every agent's models). - A recipe runs one OSWorld agent:
bench_options.agentis one value for the whole recipe. - Every OSWorld own-agent job records its agent as
original(job.py,agent_info.nameandagents[].name), andparse.pyreads that as the job's agent. The chosen agent is only inconfig.standard.benchmark_options.agent, which no reader uses, so under Any of your choice two different OSWorld agents with one model look like the same participant in Compare. - Under the
modelmethod,config.standard.setup.askedissetups.model.runs_as(MiniMax-M3) whatever the agent, while the driver asks the agent's own first model (osworld1/driver.py,answer). - The Agents page lists only
catalogue/agents/folders. A benchmark's own agents do not appear there, and the page cannot show which agents run a given benchmark. - The requirement "this agent belongs to one benchmark" already exists as
only_for(match.fit), used by Terminus 2 and mini-swe-agent, with its reason shown on a greyed tile.
Wanted
- One list of agents, each saying where it comes from: bundled, the user's own, or a benchmark's own. Where it comes from is derived from where its definition sits, never written in a file.
- A benchmark's own agents are defined once, in the benchmark folder's
agents.json: each entry conforms to the agent schema (key,label,summary, its publishedmodels) and starts with a new, optionalstart.driverkind ofnullius.agent/v1, holding the upstream's own value (agent) and today'sscript,argvandenv. Keys are dashed (openai-cua), the ref is<benchmark>:<key>.only_foris derived from the folder. standard.jsonkeepsoptions.agent'slabel,summaryandsetups; the loader rebuilds its values, names, about and default fromagents.json, so drivers,--bench-option agent=...andNULLIUS_OPTION_AGENTare unchanged.- In the builder, a benchmark's own agents are tiles in the shared Agent step selector, first, and several may be chosen, one job each. The
agentselect and the "OSWorld's agents" tile are gone. Settings that apply to one agent (OSWorld'sthinking, for M3) are in that agent's tab. - Other benchmarks' own agents are greyed with the
only_forreason ("belongs to OSWorld1"); a filter can hide them. A benchmark's own agent has no port to prove, so no port reason is added for it. - The Agents page lists every agent, with a From filter whose default hides benchmark agents. A benchmark agent's card shows its benchmark (linked) and its published models, and its protocol reads "its own script".
- A Runs on filter on the Agents and Capabilities pages lists, for each benchmark, the entries that fit it, from the same
fitsthe builder greys with. - Saved recipes, drafts and a run's recipe keep their OSWorld agent choice when read back.
- A job records the agent that ran (for example
openai-cua), and older OSWorld jobs are read the same way from their recorded option, so Compare shows one column per agent. - Out of scope: tau3-bench's
upstream_agent, and Harbor's "runs only on" binding (a follow-up).
Acceptance
-
osworld1/agents.jsonandosworld2/agents.jsonentries havekey,label,summary,modelsandstart.driver;standard.jsonwrites novalues,names,aboutordefaultforoptions.agentand nosetups.original.reference_agent. -
definitions.problemsrefuses a driver agent withoffersorproven, an entry withoutlabel,summary,modelsorstart.driver.script, an entry whosemodelsare not all insetups.original.models, a writtenonly_fornaming another benchmark, andoptions.agent.valueswritten besideagents.json, each with a test. Every existingagent.jsonstays valid. -
recipe.choices()lists each own agent with the ref<benchmark>:<key>andfrombenchmark;match.tablegives it no reason for its own benchmark and oneonly_forreason for every other, with a test. - A recipe with
others: ["osworld1:openai-cua"]compiles to the same command as today'sothers: ["original"], bench_options: {agent: "openai_cua"}, byte for byte, with a test. - A recipe from before is read as the matching own agent by
standardFromJSONand by the server's compile; withoutagentit is the first agent; with an unknown value the compile names it as an error. Each case has a test. - A recipe with two OSWorld 1 own agents compiles to two jobs whose names end with the agent keys.
- Like the paper with an own agent and a model that agent is not published with is a Model step error naming the agent.
- Under the
modelmethod,config.standard.setup.askedis the chosen agent's first model, with a test for a non-M3 agent. - A new OSWorld job records the agent key in
agent_info.name,agents[].nameandconfig.standard.setup.agent;parsegives an older OSWorld setup job the same agent name, with a test. - The Agent step for OSWorld 1 shows the 12 own agents as tiles and no
agentselect;thinkingappears only in M3 Agent's tab; another agent benchmark shows them greyed with the reason, and a filter hides them. - The Agents page lists own agents with a From filter whose default leaves them out;
_tables.htmlaccepts several default values inonly. - A Runs on filter on the Agents and Capabilities pages: for OSWorld1 it keeps its 12 own agents and the catalogue agents that fit it, with a route test.
-
app/ui_tests/test_recipe_execution_fixtures.pyand its Harbor fixtures are unchanged. -
docs/standard.md,docs/builder.md,docs/agents/add-manually.mdand the import-standard skill describe a benchmark's own agents; the generated Supported pages are refreshed bypython3 app/tests/catalogue_docs.py --write.
Dependencies and related
- feat(catalogue): add OSWorld1 with OSWorld's own agents and a null control #117 (OSWorld 1) and feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134 (OSWorld 2): where
options.agentandagents.jsonwere introduced. - Standard: an agent of the Agents page cannot run a computer-use benchmark #187 and feat(standard): run a desktop benchmark with an agent of the Agents page, through a computer port and its kit #188: a catalogue agent that runs OSWorld through the computer port. It stays a catalogue agent and sits beside the own agents.
- feat(catalogue): sort and filter the catalogue pages, and show them as cards or a table #240: the catalogue pages' filters and sort, which the Agents page From filter uses.
- feat(web): show the Workspaces page as cards, with a card layout shared by every card page #238: the shared card layout of the Agents page cards.
- Follow-ups: tau3-bench's
upstream_agentoption and Harbor's benchmark-bound reference agent (recipes/launch.py, "runs only on").
Metadata
Metadata
Assignees
Labels
P0Fix now: unsafe, loses money or data, or nothing runsFix now: unsafe, loses money or data, or nothing runsenhancementNew feature or requestNew feature or request
Activity