Skip to content

Standard: a benchmark's own agents are one option of its Original setup, not agents to choose #245

Description

@harenome

Priority

extreme. It reaches the standard's recipe compiler, matcher and job records (app/nullius/standard/), the catalogue's OSWorld entries, the Agents page and the builder.

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Today

  • OSWorld 1 and OSWorld 2 model their upstream's agent scripts as a benchmark option, options.agent in standard.json (values, names, about, default, setups: ["original", "model"]), with the scripts' argv and env in a driver data file, agents.json. No other benchmark does this, and docs/standard.md does not describe it.
  • The builder shows the option inside the Original setup's tab: a select of 12 agents for OSWorld 1, two value cards for OSWorld 2. The Agent step shows one tile, "OSWorld's agents" (setups.original.reference_agent).
  • The agent list is written three times: options.agent, agents.json.agents, and setups.original.models (the union of every agent's models).
  • A recipe runs one OSWorld agent: bench_options.agent is one value for the whole recipe.
  • Every OSWorld own-agent job records its agent as original (job.py, agent_info.name and agents[].name), and parse.py reads that as the job's agent. The chosen agent is only in config.standard.benchmark_options.agent, which no reader uses, so under Any of your choice two different OSWorld agents with one model look like the same participant in Compare.
  • Under the model method, config.standard.setup.asked is setups.model.runs_as (MiniMax-M3) whatever the agent, while the driver asks the agent's own first model (osworld1/driver.py, answer).
  • The Agents page lists only catalogue/agents/ folders. A benchmark's own agents do not appear there, and the page cannot show which agents run a given benchmark.
  • The requirement "this agent belongs to one benchmark" already exists as only_for (match.fit), used by Terminus 2 and mini-swe-agent, with its reason shown on a greyed tile.

Wanted

  • One list of agents, each saying where it comes from: bundled, the user's own, or a benchmark's own. Where it comes from is derived from where its definition sits, never written in a file.
  • A benchmark's own agents are defined once, in the benchmark folder's agents.json: each entry conforms to the agent schema (key, label, summary, its published models) and starts with a new, optional start.driver kind of nullius.agent/v1, holding the upstream's own value (agent) and today's script, argv and env. Keys are dashed (openai-cua), the ref is <benchmark>:<key>. only_for is derived from the folder.
  • standard.json keeps options.agent's label, summary and setups; the loader rebuilds its values, names, about and default from agents.json, so drivers, --bench-option agent=... and NULLIUS_OPTION_AGENT are unchanged.
  • In the builder, a benchmark's own agents are tiles in the shared Agent step selector, first, and several may be chosen, one job each. The agent select and the "OSWorld's agents" tile are gone. Settings that apply to one agent (OSWorld's thinking, for M3) are in that agent's tab.
  • Other benchmarks' own agents are greyed with the only_for reason ("belongs to OSWorld1"); a filter can hide them. A benchmark's own agent has no port to prove, so no port reason is added for it.
  • The Agents page lists every agent, with a From filter whose default hides benchmark agents. A benchmark agent's card shows its benchmark (linked) and its published models, and its protocol reads "its own script".
  • A Runs on filter on the Agents and Capabilities pages lists, for each benchmark, the entries that fit it, from the same fits the builder greys with.
  • Saved recipes, drafts and a run's recipe keep their OSWorld agent choice when read back.
  • A job records the agent that ran (for example openai-cua), and older OSWorld jobs are read the same way from their recorded option, so Compare shows one column per agent.
  • Out of scope: tau3-bench's upstream_agent, and Harbor's "runs only on" binding (a follow-up).

Acceptance

  • osworld1/agents.json and osworld2/agents.json entries have key, label, summary, models and start.driver; standard.json writes no values, names, about or default for options.agent and no setups.original.reference_agent.
  • definitions.problems refuses a driver agent with offers or proven, an entry without label, summary, models or start.driver.script, an entry whose models are not all in setups.original.models, a written only_for naming another benchmark, and options.agent.values written beside agents.json, each with a test. Every existing agent.json stays valid.
  • recipe.choices() lists each own agent with the ref <benchmark>:<key> and from benchmark; match.table gives it no reason for its own benchmark and one only_for reason for every other, with a test.
  • A recipe with others: ["osworld1:openai-cua"] compiles to the same command as today's others: ["original"], bench_options: {agent: "openai_cua"}, byte for byte, with a test.
  • A recipe from before is read as the matching own agent by standardFromJSON and by the server's compile; without agent it is the first agent; with an unknown value the compile names it as an error. Each case has a test.
  • A recipe with two OSWorld 1 own agents compiles to two jobs whose names end with the agent keys.
  • Like the paper with an own agent and a model that agent is not published with is a Model step error naming the agent.
  • Under the model method, config.standard.setup.asked is the chosen agent's first model, with a test for a non-M3 agent.
  • A new OSWorld job records the agent key in agent_info.name, agents[].name and config.standard.setup.agent; parse gives an older OSWorld setup job the same agent name, with a test.
  • The Agent step for OSWorld 1 shows the 12 own agents as tiles and no agent select; thinking appears only in M3 Agent's tab; another agent benchmark shows them greyed with the reason, and a filter hides them.
  • The Agents page lists own agents with a From filter whose default leaves them out; _tables.html accepts several default values in only.
  • A Runs on filter on the Agents and Capabilities pages: for OSWorld1 it keeps its 12 own agents and the catalogue agents that fit it, with a route test.
  • app/ui_tests/test_recipe_execution_fixtures.py and its Harbor fixtures are unchanged.
  • docs/standard.md, docs/builder.md, docs/agents/add-manually.md and the import-standard skill describe a benchmark's own agents; the generated Supported pages are refreshed by python3 app/tests/catalogue_docs.py --write.

Dependencies and related

Activity

  1. added
    enhancementNew feature or request
    P0Fix now: unsafe, loses money or data, or nothing runs
    on Nov 20, 2026
  2. added theissue type on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Fix now: unsafe, loses money or data, or nothing runsenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions