Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 40 additions & 5 deletions app/nullius/catalogue/benchmarks/osworld1/benchmark.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"An agent runs it through the computer port: the agent's tool calls drive OSWorld's own loop, one step for each acting call. A step's observation is the screenshot that loop hands its agent for the next decision, with no capture added; the files a step names are read once its actions and their pauses have run. Steps (100) and Pause (3.0 seconds) set that loop.",
"Original runs one of OSWorld's own agents, chosen by the Agent option (M3 by default), with a model it is published with.",
"Providers: OpenAI for OpenAI CUA; OpenAI-compatible for Muse Spark (Meta's API) and for EvoCUA, AutoGLM-OS and Mano (your model server); OpenRouter for the others.",
"Only M3, PromptAgent and Qwen-CUA have completed tasks through Nullius. The other agents were checked up to their first model call.",
"At this version only the null control has run through Nullius. OSWorld's agents are the same code as in main (044bc33), where M3, PromptAgent and Qwen-CUA completed tasks.",
"Each task's VM takes 4 cores and 4 GB of memory; OS-Symphony can start a second. Run no more tasks at once than the machine holds.",
"Prepare the machine before the first run (Docker, KVM, five system libraries, a VM image of about 25 GB): see OSWorld1 machine setup under Benchmarks in the documentation.",
"Scores depend on the machine's speed, because OSWorld waits a fixed time after each action. Compare runs made on similar machines.",
Expand All @@ -37,21 +37,56 @@
"The agent's python and upload calls do not mark OSWorld's desktop as used; each unit's desktop is reset once, before the agent's first call.",
"Only 12 of OSWorld's agents are offered; the others fail at the pinned commit or need services the relay cannot reach.",
"The VMs run only on Docker's default socket, not on rootless Docker or a Docker context. Podman is untested.",
"A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score."
"A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score.",
"Its evaluator fixes, key repeat turned off in the VM and one retired task make its scores comparable only with this version's, not with the paper's or main (044bc33)'s."
],
"datasets": [
{
"key": "osworld1",
"title": "OSWorld, Ubuntu tasks",
"tasks": 369,
"tasks": 368,
"official": "https://github.com/xlang-ai/OSWorld",
"source": "GitHub",
"registry": "",
"hub": "",
"verifier": "OSWorld's own evaluator, DesktopEnv.evaluate: the task's evaluator functions on the VM's final state.",
"notes": "369 tasks in 10 domains; 361 without Google Drive. Apache-2.0."
"notes": "368 tasks in 10 domains; 360 without Google Drive. Apache-2.0."
}
],
"citation": "@misc{OSWorld,\n title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments}, \n author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},\n year={2024},\n eprint={2404.07972},\n archivePrefix={arXiv},\n primaryClass={cs.AI}\n}",
"citation_source": "https://github.com/xlang-ai/OSWorld"
"citation_source": "https://github.com/xlang-ai/OSWorld",
"versions": {
"044bc33": {
"good_to_know": [
"An agent runs it through the computer port: the agent's tool calls drive OSWorld's own loop, one step for each acting call. A step's observation is the screenshot that loop hands its agent for the next decision, with no capture added; the files a step names are read once its actions and their pauses have run. Steps (100) and Pause (3.0 seconds) set that loop.",
"Original runs one of OSWorld's own agents, chosen by the Agent option (M3 by default), with a model it is published with.",
"Providers: OpenAI for OpenAI CUA; OpenAI-compatible for Muse Spark (Meta's API) and for EvoCUA, AutoGLM-OS and Mano (your model server); OpenRouter for the others.",
"Only M3, PromptAgent and Qwen-CUA have completed tasks through Nullius. The other agents were checked up to their first model call.",
"Each task's VM takes 4 cores and 4 GB of memory; OS-Symphony can start a second. Run no more tasks at once than the machine holds.",
"Prepare the machine before the first run (Docker, KVM, five system libraries, a VM image of about 25 GB): see OSWorld1 machine setup under Benchmarks in the documentation.",
"Scores depend on the machine's speed, because OSWorld waits a fixed time after each action. Compare runs made on similar machines.",
"OSWorld's infeasible tasks are listed publicly, and its scorer checks only the last action. An agent that looks a task up online and answers FAIL scores it."
],
"limitations": [
"An agent needs the computer port, proven by its conformance kit. The agent side has been checked on a fake desktop alone.",
"Preparing the VM image (its lock and its download) is not bounded by a unit's setup time; the core's own limit on setup ends it.",
"A VM start that hangs holds the start gate until its unit's setup time ends; units waiting for the gate fail at the end of theirs.",
"On the local backend an agent runs as you and can write where you can, the runner's folder included.",
"A runner let go at the end of scoring's time that then ends by itself is scored from its complete result.",
"A stop during the runner's last clean-up keeps the result already written.",
"A driver killed outright leaves the runner to stop on its own; a kill during a VM's start can leave that VM.",
"A reaper checks a runner's start time before it ends the runner's group; a process number reused in between is not caught.",
"The agent's python and upload calls do not mark OSWorld's desktop as used; each unit's desktop is reset once, before the agent's first call.",
"Only 12 of OSWorld's agents are offered; the others fail at the pinned commit or need services the relay cannot reach.",
"The VMs run only on Docker's default socket, not on rootless Docker or a Docker context. Podman is untested.",
"A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score."
],
"datasets": {
"osworld1": {
"tasks": 369,
"notes": "369 tasks in 10 domains; 361 without Google Drive. Apache-2.0."
}
}
}
}
}
252 changes: 207 additions & 45 deletions app/nullius/catalogue/benchmarks/osworld1/standard.json
Original file line number Diff line number Diff line change
Expand Up @@ -245,57 +245,219 @@
}
}
},
"proven": {
"test_nogdrive": {
"units": {
"units": 361,
"passed": "361 ids, the same twice, unique",
"on": "2026-11-13"
"default": "desktop-env-v1.0.3",
"versions": {
"desktop-env-v1.0.3": {
"upstream": {
"tag": "desktop-env-v1.0.3",
"tested": "9d6f711d234b84cbff1b5199211610baa03b000c",
"tested_date": "2026-11-19"
},
"nop": {
"passed": "0/8",
"value": 0.0,
"expect": {
"at_most": 0.0
"datasets": [
{
"id": "test_nogdrive",
"label": "Without Google Drive",
"summary": "The upstream's test_nogdrive.json: every task but the 8 that need a Google account's credentials. The live MBTA booking task is retired at this release.",
"path": "evaluation_examples/test_nogdrive.json",
"sha256": "9270c11fe809eb4c4e35a78a47cdb50717ea76557c1b9164c183e8e8b8d9ce91",
"units": 360
},
"units": 8,
"job": "2026-10-29__18-38-14__osworld1__test_nogdrive__prove-nop",
"source": "nullius",
"on": "2026-10-29",
"limit": 8
}
},
"test_all": {
"units": {
"units": 369,
"passed": "369 ids, the same twice, unique",
"on": "2026-11-13"
}
},
"test_infeasible": {
"units": {
"units": 25,
"passed": "25 ids, the same twice, unique",
"on": "2026-11-13"
{
"id": "test_all",
"label": "All tasks",
"summary": "The upstream's test_all.json, its runners' default: the 8 Google Drive tasks need credentials under evaluation_examples/settings, which the upstream does not ship. The live MBTA booking task is retired at this release.",
"path": "evaluation_examples/test_all.json",
"sha256": "47c54203369bd438abf279f6b9836bffc5aa43de6cbaccdfed546993f3fe9149",
"units": 368
},
{
"id": "test_small",
"label": "Small",
"summary": "The upstream's test_small.json: a few tasks of each domain.",
"path": "evaluation_examples/test_small.json",
"sha256": "bb29275171b78728d71825b125af311d74e66bb2a6d693c45c497d380b1fc640",
"units": 39
},
{
"id": "test_infeasible",
"label": "Infeasible",
"summary": "The upstream's test_infeasible.json: tasks that cannot be done, passed by answering FAIL.",
"path": "evaluation_examples/test_infeasible.json",
"sha256": "1b447d0462b1bdac4b660e0dd6e6add6d01c17a9e684702c701abcffda409abb",
"units": 25
}
],
"driver": {
"command": [
"{install}/.venv/bin/python",
"{here}/driver.py",
"{here}/agents.json",
"{here}/nop.py",
"{here}/runner.py"
],
"install": {
"python": "3.12",
"project": ".",
"locked": true,
"virtual": true
},
"participant": true,
"unit_end": true,
"task_given": true
},
"setups": {
"original": {
"summary": "OSWorld's own agents as published: the agent option's script (scripts/python/run_*.py) on the task, with its own prompts, parser, history and step limit, asking one of the models it is published with, on the docker provider's VM, scored by DesktopEnv.evaluate. At OSWorld's desktop-env 1.0.3 release: its evaluator fixes change the score of some tasks and one task is retired, so its runs compare with each other only, not with the paper's or the earlier pin's.",
"models": [
"MiniMax-M3",
"gpt-4o",
"claude-sonnet-4-5-20250929",
"claude-opus-4-5",
"claude-opus-4-6",
"claude-opus-4-7",
"gpt-5.4",
"computer-use-preview",
"qwen3.7-plus",
"gpt-5",
"EvoCUA-S2",
"EvoCUA-S1",
"autoglm-os",
"mano",
"muse-spark-1.1"
],
"default": "MiniMax-M3",
"served_as": {
"MiniMax-M3": "minimax/minimax-m3",
"gpt-4o": "openai/gpt-4o",
"claude-sonnet-4-5-20250929": "anthropic/claude-sonnet-4.5",
"claude-opus-4-5": "anthropic/claude-opus-4.5",
"claude-opus-4-6": "anthropic/claude-opus-4.6",
"claude-opus-4-7": "anthropic/claude-opus-4.7",
"gpt-5.4": "openai/gpt-5.4",
"qwen3.7-plus": "qwen/qwen3.7-plus",
"gpt-5": "openai/gpt-5",
"UI-TARS-1.5-7B": "bytedance/ui-tars-1.5-7b"
},
"providers": {
"openrouter": "",
"openai": "",
"custom": ""
},
"turn_seconds": 7200,
"side_models": [
"UI-TARS-1.5-7B"
],
"comparable": false
},
"model": {
"summary": "The same agent with any model in its own model's place: the upstream's code asks for the agent's first published model, with its prompt and action format, and the relay sends the model chosen, every side call included. Its scores compare with other runs of this setup.",
"runs_as": "MiniMax-M3"
}
},
"proven": {
"test_nogdrive": {
"units": {
"units": 360,
"passed": "360 ids, the same twice, unique",
"on": "2026-11-19"
}
},
"test_all": {
"units": {
"units": 368,
"passed": "368 ids, the same twice, unique",
"on": "2026-11-19"
}
},
"test_small": {
"units": {
"units": 39,
"passed": "39 ids, the same twice, unique",
"on": "2026-11-19"
},
"nop": {
"passed": "0/8",
"value": 0.0,
"expect": {
"at_most": 0.0
},
"units": 8,
"job": "2026-11-19__13-12-06__osworld1-desktop-env-v1.0.3__test_small__prove-nop",
"source": "nullius",
"options": {
"max_steps": "default",
"vm_provider": "docker"
},
"on": "2026-11-19",
"limit": 8
}
},
"test_infeasible": {
"units": {
"units": 25,
"passed": "25 ids, the same twice, unique",
"on": "2026-11-19"
}
}
}
},
"test_small": {
"units": {
"units": 39,
"passed": "39 ids, the same twice, unique",
"on": "2026-11-13"
"044bc33": {
"upstream": {
"branch": "main"
},
"nop": {
"passed": "0/8",
"value": 0.0,
"expect": {
"at_most": 0.0
"proven": {
"test_nogdrive": {
"units": {
"units": 361,
"passed": "361 ids, the same twice, unique",
"on": "2026-11-13"
},
"nop": {
"passed": "0/8",
"value": 0.0,
"expect": {
"at_most": 0.0
},
"units": 8,
"job": "2026-10-29__18-38-14__osworld1__test_nogdrive__prove-nop",
"source": "nullius",
"on": "2026-10-29",
"limit": 8
}
},
"test_all": {
"units": {
"units": 369,
"passed": "369 ids, the same twice, unique",
"on": "2026-11-13"
}
},
"units": 8,
"job": "2026-11-13__14-08-41__osworld1__test_small__prove-nop",
"source": "nullius",
"on": "2026-11-13",
"limit": 8
"test_infeasible": {
"units": {
"units": 25,
"passed": "25 ids, the same twice, unique",
"on": "2026-11-13"
}
},
"test_small": {
"units": {
"units": 39,
"passed": "39 ids, the same twice, unique",
"on": "2026-11-13"
},
"nop": {
"passed": "0/8",
"value": 0.0,
"expect": {
"at_most": 0.0
},
"units": 8,
"job": "2026-11-13__14-08-41__osworld1__test_small__prove-nop",
"source": "nullius",
"on": "2026-11-13",
"limit": 8
}
}
}
}
}
Expand Down
Loading