From 9bb9e926b23e3e90dcff283107b27234c9ddc852 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 12:19:28 +0800 Subject: [PATCH 01/10] docs: clarify evaluation tracks, releases and related projects --- CONTRIBUTING.md | 4 +++- README.md | 33 +++++++++++++++++++++++++++----- docs/README.zh-CN.md | 33 +++++++++++++++++++++++++++----- docs/dependencies.md | 10 ++++++++++ docs/releases.md | 45 ++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 114 insertions(+), 11 deletions(-) create mode 100644 docs/releases.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 237700ba7..68ad49325 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -120,4 +120,6 @@ Optional Claude review requires `ENABLE_CLAUDE_REVIEW=true` and `CLAUDE_CODE_OAU Contributions use the [Apache License 2.0](LICENSE). Preserve third-party licenses, copyright notices and source attributions described in [THIRD_PARTY](THIRD_PARTY.md). -Version changes require an explicit human release request. Code synchronization, review fixes, tests and report rebuilds must not automatically bump package, benchmark or report edition numbers, rewrite frozen release catalogs, or create/push release tags. Reuse existing build versions and record source hashes; keep unreleased changes separate from a certified edition. +## Software releases + +Follow the [release guide](docs/releases.md) for software compatibility, version changes and release checks. Benchmark editions and report builds have separate identities. Routine review fixes and report rebuilds retain the current version; bump versions and publish tags only for an explicitly requested release. diff --git a/README.md b/README.md index 90356d3ca..5f8ec8f12 100644 --- a/README.md +++ b/README.md @@ -8,26 +8,26 @@ [![License: Apache 2.0](https://img.shields.io/badge/code-Apache--2.0-51735b)](LICENSE) [![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](pyproject.toml) -[Quickstart](#quickstart) · [Tracks](#evaluation-tracks) · [Reports](#reports) · [Contributing](CONTRIBUTING.md) +[Quickstart](#quickstart) · [Tracks](#evaluation-tracks) · [Reports](#reports) · [Versions](#versions) · [Related projects](#related-repositories) · [Contributing](CONTRIBUTING.md) -Browser Eval is an extensible evaluation harness for browsers and web agents. It combines task contracts, adapters, saved evidence and scoring to produce inspectable reports. +Browser Eval evaluates Web Fetch, fixed browser trajectories and autonomous Browser Use, with Web Search available in preview. Compare task results, latency and resource use through shared adapters, saved evidence and inspectable reports. ## Evaluation tracks -The following workflows have runnable guides. New workflows use the same adapter and task-contract model; see [contributing](CONTRIBUTING.md). +The main branch includes the three browser workflows below. Web Search is implemented on the development branch linked in its row; it is not included in the current main-branch package. | Track | What it tests | Guide | | --- | --- | --- | -| Web Search / SERP | Compare returned search results, relevance, latency and cost | [Offline example](examples/web_search/README.md) · [Track guide](harness/web_search/README.md) | | Web Fetch | Retrieve the requested page content, records, resources or files | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | +| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [Implementation and integration PR #58](https://github.com/lexmount/browser-eval/pull/58) · [Search guide](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | | Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [CowAgent + WebArena](examples/browser_use/README.md) | Web Fetch supports local browsers, hosted services and extraction tools. See [adapters](docs/provider-contract.md) for available methods and [dependencies](docs/dependencies.md) for installation requirements. Replay and Browser Use have separate runners and environments. -Task definitions and scoring come from the selected benchmark: [LexBench Web Search](https://github.com/lexmount/lexbench-websearch), [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), WebMainBench, WCXB or WebArena. See [benchmark integration](docs/benchmarks.md). +Task definitions and scoring come from the selected benchmark: [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) or [WebArena-Verified](https://github.com/ServiceNow/webarena-verified). Search tasks and scoring belong to [LexBench Web Search](https://github.com/lexmount/lexbench-websearch). See [benchmark integration](docs/benchmarks.md). ## Quickstart @@ -61,6 +61,29 @@ Each new report records its Browser Eval build and selected benchmark edition, w To share a report online, publish its `report/` directory through [Artifact Site](https://github.com/lexmount/artifact-site), following the [hosting guide](docs/report-hosting.md). Choose access permissions there. Formal benchmark releases also require the checks in their track guide. +## Versions + +The current package version is `0.2.0`. Software tags use `vMAJOR.MINOR.PATCH`; benchmark editions and hosted report versions are independent. A report records all three identities so rebuilding a report does not relabel its tasks or saved calls. + +Use [the release guide](docs/releases.md) for compatibility rules, release checks and the scope of the first software release. Web Search remains a preview until its integration reaches main. A stable release commits to documented commands, input contracts and saved-evidence compatibility; it does not guarantee that every browser or live website passes every task. + +## Related repositories + +| Repository | Role | +| --- | --- | +| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Versioned Web Fetch tasks, acceptance scoring and frozen input bindings | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Versioned search queries, source references and retrieval scoring | +| [Moli](https://github.com/lexmount/moli) | Browser evaluated through the same task contracts as other participants | +| [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service; separate from the evaluation harness | +| [Artifact Site](https://github.com/lexmount/artifact-site) | Report hosting, immutable versions and access controls | +| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Protocol, automation-driver and web-platform benchmarks | +| [WebMainBench fork](https://github.com/lexmount/WebMainBench/tree/lex-main) · [upstream](https://github.com/opendatalab/WebMainBench) | Frozen-HTML extraction tasks and declared scoring corrections | +| [WebArena-Verified fork](https://github.com/lexmount/webarena-verified/tree/lex-main) · [upstream](https://github.com/ServiceNow/webarena-verified) | Tasks and evaluator for Browser Use and WebArena trajectory replay | +| [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | Independent content-extraction benchmark | +| [CowAgent](https://github.com/zhayujie/CowAgent), [OpenClaw](https://github.com/openclaw/openclaw), [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) | Agent execution and Web Fetch tool baselines | + +[Dependency sources and pins](docs/dependencies.md) cover Playwright, extraction libraries and supporting runtimes. [Third-party notices](THIRD_PARTY.md) cover attribution and data rights. Cite the benchmark and upstream data used alongside Browser Eval. + ## Contribute Read [CONTRIBUTING](CONTRIBUTING.md) for code ownership, tests and PR conventions, or [open an issue](https://github.com/lexmount/browser-eval/issues) with a reproducible example. diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index b305e8c00..d41996ea2 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -8,26 +8,26 @@ [![License: Apache 2.0](https://img.shields.io/badge/code-Apache--2.0-51735b)](../LICENSE) [![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](../pyproject.toml) -[快速开始](#快速开始) · [评测流程](#评测流程) · [报告](#报告) · [参与贡献](../CONTRIBUTING.md) +[快速开始](#快速开始) · [评测流程](#评测流程) · [报告](#报告) · [版本](#版本) · [相关仓库](#相关仓库) · [参与贡献](../CONTRIBUTING.md) -Browser Eval 是面向浏览器和 Web Agent 的可扩展评测框架，将任务规则、适配器、保存的证据和评分结果汇集为可查阅的报告。 +Browser Eval 支持 Web Fetch、固定轨迹重放和自主 Browser Use 评测，并提供 Web Search 预览实现。通过共用的适配器、保存的证据和可查阅的报告，比较任务结果、延迟与资源用量。 ## 评测流程 -以下流程均有可运行的指南，适配器和任务规则见[贡献指南](../CONTRIBUTING.md)。 +主分支包含下列三种浏览器评测流程。Web Search 已在表中链接的开发分支实现，当前主分支的软件包尚未包含它。 | 流程 | 测什么 | 使用指南 | | --- | --- | --- | -| Web Search / SERP | 比较搜索结果、相关性、耗时和费用 | [离线示例](../examples/web_search/README.md) · [专项指南](../harness/web_search/README.md) | | Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | +| Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [实现与接入 PR #58](https://github.com/lexmount/browser-eval/pull/58) · [搜索评测指南](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | | 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [CowAgent + WebArena](../examples/browser_use/README.md) | Web Fetch 支持本地浏览器、在线服务和提取工具。可用方法见[适配器说明](../docs/provider-contract.md)，安装要求见[依赖说明](../docs/dependencies.md)。轨迹重放和 Browser Use 各有自己的运行入口与环境。 -题目和评分规则来自选定的评测集：[LexBench Web Search](https://github.com/lexmount/lexbench-websearch)、[LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、WebMainBench、WCXB 或 WebArena。接入方式见[评测集指南](../docs/benchmarks.md)。 +题目和评分规则来自选定的评测集：[LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) 或 [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)。搜索题目和评分由 [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) 维护。接入方式见[评测集指南](../docs/benchmarks.md)。 ## 快速开始 @@ -61,6 +61,29 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run 在线分享时，按[托管指南](../docs/report-hosting.md)将 `report/` 目录发布到 [Artifact Site](https://github.com/lexmount/artifact-site)，并设置访问权限。正式评测发布还需通过对应流程的检查。 +## 版本 + +当前软件包版本是 `0.2.0`。软件标签采用 `v主版本.次版本.补丁版本`，评测集和托管报告各有独立版本。报告记录这三种身份，因此重建报告不会改换题目版本或保存的调用。 + +兼容规则、发布检查和首个软件发行版的范围见[发布指南](../docs/releases.md)。Web Search 在接入主分支前保持预览状态。稳定版承诺的是文档中的命令、输入约定和历史证据兼容性；浏览器与在线网站的逐题表现仍由评测结果说明。 + +## 相关仓库 + +| 仓库 | 用途 | +| --- | --- | +| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Web Fetch 题目、验收评分和冻结输入绑定 | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 | +| [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 | +| [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | +| [Artifact Site](https://github.com/lexmount/artifact-site) | 报告托管、不可变版本和访问权限 | +| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 协议、自动化驱动和 Web 平台评测 | +| [WebMainBench 分支](https://github.com/lexmount/WebMainBench/tree/lex-main) · [上游](https://github.com/opendatalab/WebMainBench) | 冻结 HTML 提取任务与有记录的评分修正 | +| [WebArena-Verified 分支](https://github.com/lexmount/webarena-verified/tree/lex-main) · [上游](https://github.com/ServiceNow/webarena-verified) | Browser Use 和 WebArena 轨迹重放的题目与评测器 | +| [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | 独立的内容提取评测集 | +| [CowAgent](https://github.com/zhayujie/CowAgent)、[OpenClaw](https://github.com/openclaw/openclaw)、[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) | Agent 运行和 Web Fetch 工具基线 | + +Playwright、提取库和配套运行环境的来源与固定版本见[依赖说明](../docs/dependencies.md)，引用要求和数据权利见[第三方说明](../THIRD_PARTY.md)。引用 Browser Eval 时，也请引用实际使用的评测集与上游数据。 + ## 参与贡献 代码归属、测试和 PR 规范见 [CONTRIBUTING](../CONTRIBUTING.md)。反馈问题请[提交可复现的示例](https://github.com/lexmount/browser-eval/issues)。 diff --git a/docs/dependencies.md b/docs/dependencies.md index ca20054bc..749a0a554 100644 --- a/docs/dependencies.md +++ b/docs/dependencies.md @@ -10,6 +10,16 @@ Install the base CLI with `uv sync --frozen`. Add only the extras needed by your Use the declared Git source for the evaluator. A registry package with the same name and version may contain different code. +## Core task and parsing dependencies + +| Dependency | Role | Version authority | +| --- | --- | --- | +| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | +| [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) | HTML parsing | `pyproject.toml` and `uv.lock` | +| [FastCDC](https://github.com/iscc/fastcdc-py) | Content chunking | `pyproject.toml` and `uv.lock` | + +Search development uses [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) for query definitions and retrieval scoring. Its integration is tracked in [PR #58](https://github.com/lexmount/browser-eval/pull/58); it is not a dependency of the current main-branch package. + ## Git-based runtime dependencies | Dependency | Repository and relevant branch | Required source | diff --git a/docs/releases.md b/docs/releases.md new file mode 100644 index 000000000..4e204f5c3 --- /dev/null +++ b/docs/releases.md @@ -0,0 +1,45 @@ +# Software releases + +Browser Eval versions the execution and reporting software. The current package version is `0.2.0`; a version in source is not a published GitHub release. Software tags use `vMAJOR.MINOR.PATCH` and identify one tested commit. + +## Version rules + +Follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). The public compatibility contract covers documented CLI commands and options, dataset/provider inputs, and reading saved evidence through supported report builders. Internal Python helpers and generated HTML markup are not public APIs. + +| Change | Version change | Example after 1.0.0 | +| --- | --- | --- | +| Compatible bug fix or a documentation-only release | Patch | `1.0.0` → `1.0.1` | +| Compatible new adapter, command, track or optional input field | Minor | `1.0.1` → `1.1.0` | +| Remove or rename a public command, require incompatible inputs, or stop reading supported evidence without migration | Major | `1.1.0` → `2.0.0` | + +While the software remains `0.x`, use a minor release for new functionality or breaking changes and a patch for compatible fixes. Describe breaking changes explicitly. Release when users need an installable, reproducible checkpoint; individual commits and report rebuilds do not need new numbers. Version changes and release tags require an explicit human release request. + +`1.0.0` establishes the compatibility contract above. It can cover stable tracks while others remain explicitly experimental. For the first stable Web Fetch + Web Search release, integrate Search into main, run its offline collection/scoring/report tests and browser evidence workflows, and pass the existing package and CI gates on the release commit. Publish the supported track and dataset matrix in its release notes. A successful historical report alone does not test the installable release. + +## Software, benchmarks and reports + +| Identity | Owner | When it changes | +| --- | --- | --- | +| Browser Eval software, e.g. `v0.2.0` | This repository | Execution, adapters, evidence handling or reporting software is released | +| Benchmark edition, e.g. `bench-v2.0.1` | Selected benchmark repository | Tasks, references or scoring change under that benchmark's rules | +| Published report build | Artifact Site or selected report host | A new report build is published | + +Added cases can require a major benchmark edition even when accepting that edition is a compatible software feature. The [benchmark contract](benchmarks.md#benchmark-editions) defines `direct`, `rescore` and `separate` comparisons. Pin upstream code commits and external input hashes alongside readable versions. Rebuilding presentation preserves saved calls and the selected benchmark edition. + +## First stable release + +Use `v1.0.0` for the first release that commits to Web Fetch and Web Search compatibility. An extra `v0.2.0` publication is not required: `0.2.0` is the current source version, and the first published software release can be `1.0.0`. + +Web Fetch includes LexBench Web Fetch integration and independently selected WebMainBench and WCXB workflows. Search uses LexBench Web Search definitions and scoring. Publish the exact supported dataset editions and provider configurations with the release. Browser Trajectory Replay and Browser Use can remain available as experimental tracks with their own environment requirements. + +Integrate [PR #58](https://github.com/lexmount/browser-eval/pull/58) into main before selecting the stable release commit. Until then, the main-branch package contains Web Fetch, trajectory replay and Browser Use; Search is preview code on its development branch. Run the integrated package and release checks before assigning `1.0.0`. External references, browsers, self-hosted sites and provider accounts remain separate inputs. + +## Release procedure + +1. Select a clean commit on main and declare supported tracks, benchmark pins and migrations. Check that pending work is not described as included. +2. Align the software version in `pyproject.toml`, `harness/web_fetch/__init__.py`, `uv.lock`, `CITATION.cff` and README citations. Set the actual release date when publishing. Run `uv lock` when package metadata changes, retaining immutable dependency pins. +3. Run `make setup-test` and `make ci` on that commit. The package gate checks installation outside the checkout. Follow each track's separate release gate when publishing benchmark reports; a software tag does not certify their measurements. +4. Prepare release notes with supported tracks and suites, installation commands, compatible interfaces, required external inputs and migrations. Check that the source archive and wheel contain the declared version. +5. After release approval, publish an annotated `vMAJOR.MINOR.PATCH` tag and GitHub release for that exact commit. Never move a published tag. Upload software distribution files; keep provider outputs and private report evidence outside the software release. + +Use a feature branch and PR for release preparation. Merging and publishing are separate maintainer actions. Record the exact commit in bug reports and citations, including for unreleased branches. From fa64e3746d0fce53048e608f31046e94e29dadb3 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 12:23:17 +0800 Subject: [PATCH 02/10] docs(releases): defer version selection until Search integration --- docs/releases.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/releases.md b/docs/releases.md index 4e204f5c3..7c4959edd 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -26,13 +26,13 @@ While the software remains `0.x`, use a minor release for new functionality or b Added cases can require a major benchmark edition even when accepting that edition is a compatible software feature. The [benchmark contract](benchmarks.md#benchmark-editions) defines `direct`, `rescore` and `separate` comparisons. Pin upstream code commits and external input hashes alongside readable versions. Rebuilding presentation preserves saved calls and the selected benchmark edition. -## First stable release +## Stable release scope -Use `v1.0.0` for the first release that commits to Web Fetch and Web Search compatibility. An extra `v0.2.0` publication is not required: `0.2.0` is the current source version, and the first published software release can be `1.0.0`. +Choose the first stable software version after the pending Web Search integration PRs have landed on main and the integrated release checks pass. Version `1.0.0` establishes the documented compatibility contract; release preparation alone does not assign that version or create a tag. Web Fetch includes LexBench Web Fetch integration and independently selected WebMainBench and WCXB workflows. Search uses LexBench Web Search definitions and scoring. Publish the exact supported dataset editions and provider configurations with the release. Browser Trajectory Replay and Browser Use can remain available as experimental tracks with their own environment requirements. -Integrate [PR #58](https://github.com/lexmount/browser-eval/pull/58) into main before selecting the stable release commit. Until then, the main-branch package contains Web Fetch, trajectory replay and Browser Use; Search is preview code on its development branch. Run the integrated package and release checks before assigning `1.0.0`. External references, browsers, self-hosted sites and provider accounts remain separate inputs. +Integrate [PR #58](https://github.com/lexmount/browser-eval/pull/58) and its related Web Search changes into main before selecting the stable release commit. Until then, the main-branch package contains Web Fetch, trajectory replay and Browser Use; Search is preview code on its development branch. Run the integrated package and release checks before assigning `1.0.0`. External references, browsers, self-hosted sites and provider accounts remain separate inputs. ## Release procedure From 424d7589f2f01275c40b183a1a0d959459c61dd9 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 12:26:11 +0800 Subject: [PATCH 03/10] docs: introduce Browser Eval as an evaluation harness --- README.md | 2 +- docs/README.zh-CN.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5f8ec8f12..b59839c39 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ -Browser Eval evaluates Web Fetch, fixed browser trajectories and autonomous Browser Use, with Web Search available in preview. Compare task results, latency and resource use through shared adapters, saved evidence and inspectable reports. +Browser Eval is an evaluation harness for browsers and web agents. Connect a benchmark and a browser or service adapter to run tasks, record responses and timing, apply the benchmark's scoring rules, and generate comparison reports with the original evidence. ## Evaluation tracks diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index d41996ea2..f1261f418 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -12,7 +12,7 @@ -Browser Eval 支持 Web Fetch、固定轨迹重放和自主 Browser Use 评测，并提供 Web Search 预览实现。通过共用的适配器、保存的证据和可查阅的报告，比较任务结果、延迟与资源用量。 +Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation harness）。接入评测集和浏览器或服务适配器后，就能运行任务、记录响应和耗时、按评测集的规则评分，并生成带原始证据的对比报告。 ## 评测流程 From 05eb98d6b965b9c1cae9704fbf136b1b2077961c Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 13:21:26 +0800 Subject: [PATCH 04/10] docs: map each evaluation track to its benchmarks --- README.md | 14 +++++++------- docs/README.zh-CN.md | 14 +++++++------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index b59839c39..2f759c9c9 100644 --- a/README.md +++ b/README.md @@ -18,16 +18,16 @@ Browser Eval is an evaluation harness for browsers and web agents. Connect a ben The main branch includes the three browser workflows below. Web Search is implemented on the development branch linked in its row; it is not included in the current main-branch package. -| Track | What it tests | Guide | -| --- | --- | --- | -| Web Fetch | Retrieve the requested page content, records, resources or files | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | -| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [Implementation and integration PR #58](https://github.com/lexmount/browser-eval/pull/58) · [Search guide](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | -| Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | -| Browser Use | Let an Agent use different browsers to complete tasks | [CowAgent + WebArena](examples/browser_use/README.md) | +| Track | What it tests | Benchmarks and task suites | Guide | +| --- | --- | --- | --- | +| Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | +| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Implementation and integration PR #58](https://github.com/lexmount/browser-eval/pull/58) · [Search guide](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | +| Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/tasks.v4.json); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | +| Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | Web Fetch supports local browsers, hosted services and extraction tools. See [adapters](docs/provider-contract.md) for available methods and [dependencies](docs/dependencies.md) for installation requirements. Replay and Browser Use have separate runners and environments. -Task definitions and scoring come from the selected benchmark: [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) or [WebArena-Verified](https://github.com/ServiceNow/webarena-verified). Search tasks and scoring belong to [LexBench Web Search](https://github.com/lexmount/lexbench-websearch). See [benchmark integration](docs/benchmarks.md). +See [benchmark integration](docs/benchmarks.md) for task and scoring ownership, pinned versions and input requirements. The replay suites adapt fixed actions for browser comparison; Browser Use evaluates autonomous Agent decisions. ## Quickstart diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index f1261f418..8b1cb7092 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -18,16 +18,16 @@ Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation har 主分支包含下列三种浏览器评测流程。Web Search 已在表中链接的开发分支实现，当前主分支的软件包尚未包含它。 -| 流程 | 测什么 | 使用指南 | -| --- | --- | --- | -| Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | -| Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [实现与接入 PR #58](https://github.com/lexmount/browser-eval/pull/58) · [搜索评测指南](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | -| 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | -| Browser Use | 让 Agent 使用不同浏览器完成任务 | [CowAgent + WebArena](../examples/browser_use/README.md) | +| 流程 | 测什么 | 对应评测集与任务集 | 使用指南 | +| --- | --- | --- | --- | +| Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | +| Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [实现与接入 PR #58](https://github.com/lexmount/browser-eval/pull/58) · [搜索评测指南](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | +| 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/tasks.v4.json)；[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json)，由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | +| Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)，使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | Web Fetch 支持本地浏览器、在线服务和提取工具。可用方法见[适配器说明](../docs/provider-contract.md)，安装要求见[依赖说明](../docs/dependencies.md)。轨迹重放和 Browser Use 各有自己的运行入口与环境。 -题目和评分规则来自选定的评测集：[LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) 或 [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)。搜索题目和评分由 [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) 维护。接入方式见[评测集指南](../docs/benchmarks.md)。 +题目与评分的维护归属、固定版本和输入要求见[评测集接入指南](../docs/benchmarks.md)。轨迹重放使用固定动作比较浏览器，Browser Use 则评测 Agent 自主决策后的任务完成情况。 ## 快速开始 From 55d066215dda1bb7459f6b81e7c488cbd32d83ba Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 13:23:41 +0800 Subject: [PATCH 05/10] docs: attribute replay scenario designs to Lexbench --- README.md | 4 ++-- docs/README.zh-CN.md | 4 ++-- docs/architecture.md | 1 + docs/benchmarks.md | 2 ++ 4 files changed, 7 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 2f759c9c9..539b01338 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ The main branch includes the three browser workflows below. Web Search is implem | --- | --- | --- | --- | | Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | | Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Implementation and integration PR #58](https://github.com/lexmount/browser-eval/pull/58) · [Search guide](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | -| Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/tasks.v4.json); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | +| Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/tasks.v4.json), adapted from [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | Web Fetch supports local browsers, hosted services and extraction tools. See [adapters](docs/provider-contract.md) for available methods and [dependencies](docs/dependencies.md) for installation requirements. Replay and Browser Use have separate runners and environments. @@ -76,7 +76,7 @@ Use [the release guide](docs/releases.md) for compatibility rules, release check | [Moli](https://github.com/lexmount/moli) | Browser evaluated through the same task contracts as other participants | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service; separate from the evaluation harness | | [Artifact Site](https://github.com/lexmount/artifact-site) | Report hosting, immutable versions and access controls | -| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Protocol, automation-driver and web-platform benchmarks | +| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Scenario design source for curated replay; separate protocol, driver and web-platform benchmarks | | [WebMainBench fork](https://github.com/lexmount/WebMainBench/tree/lex-main) · [upstream](https://github.com/opendatalab/WebMainBench) | Frozen-HTML extraction tasks and declared scoring corrections | | [WebArena-Verified fork](https://github.com/lexmount/webarena-verified/tree/lex-main) · [upstream](https://github.com/ServiceNow/webarena-verified) | Tasks and evaluator for Browser Use and WebArena trajectory replay | | [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | Independent content-extraction benchmark | diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 8b1cb7092..0b132a169 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -22,7 +22,7 @@ Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation har | --- | --- | --- | --- | | Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | | Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [实现与接入 PR #58](https://github.com/lexmount/browser-eval/pull/58) · [搜索评测指南](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | -| 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/tasks.v4.json)；[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json)，由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | +| 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/tasks.v4.json)，改编自 [Lexbench-Headless-Browser 场景设计](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios)；[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json)，由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)，使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | Web Fetch 支持本地浏览器、在线服务和提取工具。可用方法见[适配器说明](../docs/provider-contract.md)，安装要求见[依赖说明](../docs/dependencies.md)。轨迹重放和 Browser Use 各有自己的运行入口与环境。 @@ -76,7 +76,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run | [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | | [Artifact Site](https://github.com/lexmount/artifact-site) | 报告托管、不可变版本和访问权限 | -| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 协议、自动化驱动和 Web 平台评测 | +| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 自建轨迹的场景设计来源；另有独立的协议、自动化驱动和 Web 平台评测 | | [WebMainBench 分支](https://github.com/lexmount/WebMainBench/tree/lex-main) · [上游](https://github.com/opendatalab/WebMainBench) | 冻结 HTML 提取任务与有记录的评分修正 | | [WebArena-Verified 分支](https://github.com/lexmount/webarena-verified/tree/lex-main) · [上游](https://github.com/ServiceNow/webarena-verified) | Browser Use 和 WebArena 轨迹重放的题目与评测器 | | [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | 独立的内容提取评测集 | diff --git a/docs/architecture.md b/docs/architecture.md index 685f6e3cd..5ace788aa 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -15,6 +15,7 @@ Browser Eval supplies execution, evidence handling and reporting for web fetchin Public WebArena human-recording replay is isolated under `harness/trajectory_replay/public_webarena/` with its [own examples and release contract](../examples/trajectory_replay/public_webarena/README.md). +The curated replay tasks also record [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) as a scenario design source. `harness/lexbench/` adapts retained Lexbench comparisons; it does not define a fourth browser score or replace a track's execution and evaluator contracts. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 1a2dae9aa..d51044d02 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -52,6 +52,8 @@ Change task requirements in LexBench, freeze a new dataset/catalog/manifest vers The retained `harness/lexbench/` module concerns historical headless-browser comparisons. It is unrelated to the `lexbench-webfetch` input package and remains an execution/report adapter. +The curated [Browser Trajectory Replay suite](../examples/trajectory_replay/tasks.v4.json) records [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios) in each applicable task's `source.design_basis`. It adapts those capabilities to public-page actions with its own exact expected outputs. This suite does not execute the complete upstream task population. The [public WebArena replay](../examples/trajectory_replay/public_webarena/README.md) instead uses qualified human recordings and the WebArena-Verified evaluator. + ## Benchmark editions Use a benchmark version together with exact task/reference and scoring-source From 32b03145e2e9e2ffa20a814262e2a135d000557d Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 13:45:41 +0800 Subject: [PATCH 06/10] feat(dependencies): support retrieval family workspace sources --- CONTRIBUTING.md | 2 +- Makefile | 2 +- README.md | 11 +++--- docs/README.zh-CN.md | 11 +++--- docs/benchmarks.md | 18 ++++++++- docs/dependencies.md | 5 +-- docs/releases.md | 4 +- harness/lexbench_pin.py | 49 +++++++++++++++++++----- tests/test_lexbench_dependency_update.py | 40 +++++++++++++++++++ tests/test_readme_navigation.py | 31 +++++++++++++++ 10 files changed, 143 insertions(+), 30 deletions(-) create mode 100644 tests/test_readme_navigation.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 68ad49325..5acb839bf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -118,7 +118,7 @@ Optional Claude review requires `ENABLE_CLAUDE_REVIEW=true` and `CLAUDE_CODE_OAU ## License -Contributions use the [Apache License 2.0](LICENSE). Preserve third-party licenses, copyright notices and source attributions described in [THIRD_PARTY](THIRD_PARTY.md). +Contributions use the [Apache License 2.0](LICENSE). Contributors retain copyright in their contributions; submitting a contribution grants the applicable license and does not transfer ownership to LexMount. Preserve existing copyright notices and third-party terms described in [THIRD_PARTY](THIRD_PARTY.md). ## Software releases diff --git a/Makefile b/Makefile index 5d877bd24..d54161a63 100644 --- a/Makefile +++ b/Makefile @@ -216,4 +216,4 @@ test-native-adapters: .PHONY: bump-lexbench bump-lexbench: @test -n "$(REV)" || (echo 'REV= is required' >&2; exit 2) - $(PY) scripts/bump_lexbench.py --revision "$(REV)" --uv "$(UV)" --benchmark "$(or $(BENCHMARK),lexbench-webfetch)" + $(PY) scripts/bump_lexbench.py --revision "$(REV)" --uv "$(UV)" --benchmark "$(or $(BENCHMARK),lexbench-webfetch)" $(if $(REPOSITORY),--repository "$(REPOSITORY)",) $(if $(SUBDIRECTORY),--subdirectory "$(SUBDIRECTORY)",) diff --git a/README.md b/README.md index 539b01338..67c524ac7 100644 --- a/README.md +++ b/README.md @@ -16,12 +16,12 @@ Browser Eval is an evaluation harness for browsers and web agents. Connect a ben ## Evaluation tracks -The main branch includes the three browser workflows below. Web Search is implemented on the development branch linked in its row; it is not included in the current main-branch package. +Each track selects its own benchmark, task population and scoring rules. The harness supplies execution adapters, saved evidence and reports. | Track | What it tests | Benchmarks and task suites | Guide | | --- | --- | --- | --- | -| Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | -| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Implementation and integration PR #58](https://github.com/lexmount/browser-eval/pull/58) · [Search guide](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | +| Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | +| Web Search / SERP | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md): scenario queries and a SimpleQA Verified retrieval adaptation | [Offline example](examples/web_search/README.md) · [Track guide](harness/web_search/README.md) | | Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/tasks.v4.json), adapted from [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | @@ -65,14 +65,13 @@ To share a report online, publish its `report/` directory through [Artifact Site The current package version is `0.2.0`. Software tags use `vMAJOR.MINOR.PATCH`; benchmark editions and hosted report versions are independent. A report records all three identities so rebuilding a report does not relabel its tasks or saved calls. -Use [the release guide](docs/releases.md) for compatibility rules, release checks and the scope of the first software release. Web Search remains a preview until its integration reaches main. A stable release commits to documented commands, input contracts and saved-evidence compatibility; it does not guarantee that every browser or live website passes every task. +Use [the release guide](docs/releases.md) for compatibility rules, release checks and the scope of the first software release. A stable release commits to documented commands, input contracts and saved-evidence compatibility; it does not guarantee that every browser or live website passes every task. ## Related repositories | Repository | Role | | --- | --- | -| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Versioned Web Fetch tasks, acceptance scoring and frozen input bindings | -| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Versioned search queries, source references and retrieval scoring | +| [LexBench Web Retrieval](https://github.com/lexmount/lexbench-web-retrieval) | Benchmark family: [Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) and [Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md), with separate task populations and scoring | | [Moli](https://github.com/lexmount/moli) | Browser evaluated through the same task contracts as other participants | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service; separate from the evaluation harness | | [Artifact Site](https://github.com/lexmount/artifact-site) | Report hosting, immutable versions and access controls | diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 0b132a169..94612eaeb 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -16,12 +16,12 @@ Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation har ## 评测流程 -主分支包含下列三种浏览器评测流程。Web Search 已在表中链接的开发分支实现，当前主分支的软件包尚未包含它。 +每种评测流程分别选择评测集、题目范围和评分规则；脚手架负责适配器调用、证据保存和报告。 | 流程 | 测什么 | 对应评测集与任务集 | 使用指南 | | --- | --- | --- | --- | -| Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | -| Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [实现与接入 PR #58](https://github.com/lexmount/browser-eval/pull/58) · [搜索评测指南](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | +| Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | +| Web Search / SERP | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [离线示例](../examples/web_search/README.md) · [完整指南](../harness/web_search/README.md) | | 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/tasks.v4.json)，改编自 [Lexbench-Headless-Browser 场景设计](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios)；[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json)，由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)，使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | @@ -65,14 +65,13 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run 当前软件包版本是 `0.2.0`。软件标签采用 `v主版本.次版本.补丁版本`，评测集和托管报告各有独立版本。报告记录这三种身份，因此重建报告不会改换题目版本或保存的调用。 -兼容规则、发布检查和首个软件发行版的范围见[发布指南](../docs/releases.md)。Web Search 在接入主分支前保持预览状态。稳定版承诺的是文档中的命令、输入约定和历史证据兼容性；浏览器与在线网站的逐题表现仍由评测结果说明。 +兼容规则、发布检查和首个软件发行版的范围见[发布指南](../docs/releases.md)。稳定版承诺的是文档中的命令、输入约定和历史证据兼容性；浏览器与在线网站的逐题表现仍由评测结果说明。 ## 相关仓库 | 仓库 | 用途 | | --- | --- | -| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Web Fetch 题目、验收评分和冻结输入绑定 | -| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 | +| [LexBench Web Retrieval](https://github.com/lexmount/lexbench-web-retrieval) | 评测家族：[Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) 和 [Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md) 各自维护题目和评分 | | [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | | [Artifact Site](https://github.com/lexmount/artifact-site) | 报告托管、不可变版本和访问权限 | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index d51044d02..c6c3fce2d 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -4,9 +4,9 @@ Browser Eval executes tasks, adapts browsers and providers, calls the selected b | Responsibility | Owner | | --- | --- | -| Self-authored Web Fetch task populations, prompts and acceptance requirements | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | +| Self-authored Web Fetch task populations, prompts and acceptance requirements | [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) | | LexBench task catalogs and frozen source-input bindings | LexBench Web Fetch | -| Web Search tasks, provenance, source qualifications and deterministic retrieval scoring | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | +| Web Search tasks, provenance, source qualifications and deterministic retrieval scoring | [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md) | | Browser/provider adapters, execution policy and credential loading | Browser Eval | | Task model, deterministic acceptance scorer, file-byte checks, compiler and scenario registry | LexBench Web Fetch | | Rescoring orchestration and evidence revision machinery | Browser Eval | @@ -275,3 +275,17 @@ LexBench Web Search is installed from the immutable commit in `pyproject.toml` a Change search queries, references or scoring in LexBench Web Search and pin the resulting commit here. Change provider behavior or report generation in Browser Eval. See the pinned benchmark's [data provenance](https://github.com/lexmount/lexbench-websearch/blob/a6d7b92987e32ef52563ee68bac4343db5686d95/docs/data.md) and [evaluation protocol](https://github.com/lexmount/lexbench-websearch/blob/a6d7b92987e32ef52563ee68bac4343db5686d95/docs/evaluation.md). Update a merged Search benchmark revision with `make bump-lexbench BENCHMARK=lexbench-websearch REV=`. The shared updater checks upstream main ancestry and updates the package, lock and dataset-hash binding together. The initial extraction PR depends on the companion benchmark PR landing first. + +## Select a package from the retrieval family + +[LexBench Web Retrieval](https://github.com/lexmount/lexbench-web-retrieval) keeps Fetch at the repository root and Search in `packages/websearch/`. Their package names and Python imports remain independent. A Git source for Search must include that subdirectory; binding only a repository and commit would select the wrong build root. + +Use the existing updater to select a merged family commit. It stages installation, checks the full main-branch ancestry, and updates the project, lock and definition binding together. For Search: + +```sh +make bump-lexbench BENCHMARK=lexbench-websearch REV= \ + REPOSITORY=https://github.com/lexmount/lexbench-web-retrieval.git \ + SUBDIRECTORY=packages/websearch +``` + +For Fetch, select the same repository without `SUBDIRECTORY`. Installed definitions and frozen scorer bytes retain their exact identities. Historical pins continue to identify their original commits; a repository migration does not authorize changed tasks, reference answers or benchmark editions. diff --git a/docs/dependencies.md b/docs/dependencies.md index 749a0a554..b5ff297c5 100644 --- a/docs/dependencies.md +++ b/docs/dependencies.md @@ -14,12 +14,11 @@ Use the declared Git source for the evaluator. A registry package with the same | Dependency | Role | Version authority | | --- | --- | --- | -| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | +| [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | +| [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md) | Search queries, source references and deterministic retrieval scoring | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md#web-search-integration) | | [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) | HTML parsing | `pyproject.toml` and `uv.lock` | | [FastCDC](https://github.com/iscc/fastcdc-py) | Content chunking | `pyproject.toml` and `uv.lock` | -Search development uses [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) for query definitions and retrieval scoring. Its integration is tracked in [PR #58](https://github.com/lexmount/browser-eval/pull/58); it is not a dependency of the current main-branch package. - ## Git-based runtime dependencies | Dependency | Repository and relevant branch | Required source | diff --git a/docs/releases.md b/docs/releases.md index 7c4959edd..e289e40e4 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -28,11 +28,11 @@ Added cases can require a major benchmark edition even when accepting that editi ## Stable release scope -Choose the first stable software version after the pending Web Search integration PRs have landed on main and the integrated release checks pass. Version `1.0.0` establishes the documented compatibility contract; release preparation alone does not assign that version or create a tag. +Choose a stable software version after all changes in its declared release scope have landed on main and the integrated release checks pass. Version `1.0.0` establishes the documented compatibility contract; release preparation alone does not assign that version or create a tag. Web Fetch includes LexBench Web Fetch integration and independently selected WebMainBench and WCXB workflows. Search uses LexBench Web Search definitions and scoring. Publish the exact supported dataset editions and provider configurations with the release. Browser Trajectory Replay and Browser Use can remain available as experimental tracks with their own environment requirements. -Integrate [PR #58](https://github.com/lexmount/browser-eval/pull/58) and its related Web Search changes into main before selecting the stable release commit. Until then, the main-branch package contains Web Fetch, trajectory replay and Browser Use; Search is preview code on its development branch. Run the integrated package and release checks before assigning `1.0.0`. External references, browsers, self-hosted sites and provider accounts remain separate inputs. +Web Fetch, Web Search / SERP, Browser Trajectory Replay and Browser Use have main-branch implementations. Validate the selected tracks together on the release commit before assigning a stable version. External references, browsers, self-hosted sites and provider accounts remain separate inputs. ## Release procedure diff --git a/harness/lexbench_pin.py b/harness/lexbench_pin.py index 4435f4869..7e1416afa 100644 --- a/harness/lexbench_pin.py +++ b/harness/lexbench_pin.py @@ -18,19 +18,32 @@ ROOT = Path(__file__).resolve().parents[1] -def project_pin(project: dict, name: str = NAME) -> tuple[str, str]: +def _subdirectory(value: str) -> str: + if value and (not re.fullmatch(r"[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)*", value) + or any(part in {'.', '..'} for part in value.split('/'))): + raise ValueError("LexBench subdirectory must stay within the repository") + return value + + +def project_pin(project: dict, name: str = NAME) -> tuple[str, str, str]: matches = [item for item in project["project"]["dependencies"] if re.match(rf"^{re.escape(name)}\s*@", item)] if len(matches) != 1: raise ValueError("Expected one direct LexBench Git dependency. " + HINT) - match = re.fullmatch(rf"{re.escape(name)}\s*@\s*git\+(.+)@([0-9a-f]{{40}})", matches[0]) + match = re.fullmatch( + rf"{re.escape(name)}\s*@\s*git\+([^#]+)@([0-9a-f]{{40}})(?:#(.*))?", matches[0]) if not match: raise ValueError("LexBench must use a full immutable Git commit. " + HINT) - return match.group(1), match.group(2) + fragment = parse_qs(match.group(3) or '', keep_blank_values=True) + if fragment and (set(fragment) != {'subdirectory'} or len(fragment['subdirectory']) != 1): + raise ValueError("Invalid LexBench subdirectory binding") + directory = fragment.get('subdirectory', [''])[0] + return match.group(1), match.group(2), _subdirectory(directory) def validate_binding(root: Path, name: str = NAME) -> str: - repository, revision = project_pin(tomllib.loads((root / "pyproject.toml").read_text()), name) + repository, revision, directory = project_pin( + tomllib.loads((root / "pyproject.toml").read_text()), name) lock = tomllib.loads((root / "uv.lock").read_text()) packages = [item for item in lock["package"] if item["name"] == name] binding = json.loads((root / "evaluation" / (name + '.json')).read_text()) @@ -40,6 +53,8 @@ def validate_binding(root: Path, name: str = NAME) -> str: locked_repository = source._replace(query="", fragment="").geturl() if (locked_repository != repository or source.fragment != revision or parse_qs(source.query).get("rev") != [revision] + or parse_qs(source.query).get("subdirectory", ['']) != [directory] + or binding.get("subdirectory", '') != directory or binding["revision"] != revision or binding["repository"].removesuffix(".git") != repository.removesuffix(".git")): raise ValueError("LexBench project, lock and definition binding disagree. " + HINT) @@ -105,10 +120,13 @@ def require_merged_revision(repository: str, revision: str, directory: Path) -> raise ValueError("Requested LexBench revision is not in upstream main history") -def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME) -> list[str]: +def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME, + repository: str | None = None, subdirectory: str | None = None) -> list[str]: project_path = root / "pyproject.toml" text = project_path.read_text() - repository, old_revision = project_pin(tomllib.loads(text), benchmark) + old_repository, old_revision, old_directory = project_pin(tomllib.loads(text), benchmark) + repository = repository if repository is not None else old_repository + directory = _subdirectory(subdirectory if subdirectory is not None else old_directory) relative_binding = 'evaluation/' + benchmark + '.json' binding_path = root / relative_binding before = json.loads(binding_path.read_text()) @@ -120,10 +138,15 @@ def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME) -> (project / "evaluation").mkdir() for name in ("uv.lock", ".python-version", "README.md", "LICENSE"): shutil.copyfile(root / name, project / name) - old_url = f"git+{repository}@{old_revision}" + old_url = f"git+{old_repository}@{old_revision}" + if old_directory: + old_url += '#subdirectory=' + old_directory if text.count(old_url) != 1: raise ValueError("Ambiguous dependency declaration") - (project / "pyproject.toml").write_text(text.replace(old_url, f"git+{repository}@{revision}")) + new_url = f"git+{repository}@{revision}" + if directory: + new_url += '#subdirectory=' + directory + (project / "pyproject.toml").write_text(text.replace(old_url, new_url)) run([uv, "lock", "--project", str(project)], project) run([uv, "sync", "--frozen", "--no-install-project", "--project", str(project)], project) definitions = Path(run([ @@ -131,6 +154,11 @@ def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME) -> f"from {benchmark.replace('-', '_')}.inputs import definition_path; print(definition_path({'registry.json' if benchmark == 'lexbench-websearch' else 'evaluation'!r}).parent)", ], project)) after = refresh_binding(before, revision, definitions) + after['repository'] = repository.removesuffix('.git') + if directory: + after['subdirectory'] = directory + else: + after.pop('subdirectory', None) (project / relative_binding).write_text( json.dumps(after, ensure_ascii=False, indent=2) + "\n") validate_binding(project, benchmark) @@ -152,7 +180,10 @@ def main(argv=None): parser.add_argument("--revision", required=True) parser.add_argument("--uv", default="uv") parser.add_argument('--benchmark', choices=['lexbench-webfetch', 'lexbench-websearch'], default=NAME) + parser.add_argument('--repository', help='Git repository for a source migration') + parser.add_argument('--subdirectory', help='Package directory within that repository') args = parser.parse_args(argv) - for change in bump(ROOT, args.revision, uv=args.uv, benchmark=args.benchmark): + for change in bump(ROOT, args.revision, uv=args.uv, benchmark=args.benchmark, + repository=args.repository, subdirectory=args.subdirectory): print(change) print("Verified current dependency binding. Historical evidence was not changed.") diff --git a/tests/test_lexbench_dependency_update.py b/tests/test_lexbench_dependency_update.py index b43b73c1c..81b93b583 100644 --- a/tests/test_lexbench_dependency_update.py +++ b/tests/test_lexbench_dependency_update.py @@ -115,3 +115,43 @@ def git(*args): require_merged_revision(str(repo), merged, tmp_path / "accepted") with pytest.raises(ValueError, match="not in upstream main"): require_merged_revision(str(repo), unmerged, tmp_path / "rejected") + + +def family_binding(root, *, directory='packages/websearch'): + revision = 'a' * 40 + repository = 'https://github.com/lexmount/lexbench-web-retrieval.git' + (root / 'evaluation').mkdir(exist_ok=True) + (root / 'pyproject.toml').write_text( + '[project]\ndependencies = ["lexbench-websearch @ git+' + + repository + '@' + revision + '#subdirectory=' + directory + '"]\n') + (root / 'uv.lock').write_text( + '[[package]]\nname = "lexbench-websearch"\nsource = { git = "' + + repository + '?rev=' + revision + '&subdirectory=' + directory + '#' + revision + '" }\n') + (root / 'evaluation/lexbench-websearch.json').write_text(json.dumps({ + 'repository': repository.removesuffix('.git'), 'revision': revision, + 'subdirectory': directory, + })) + return revision + + +def test_family_package_pin_binds_repository_revision_and_subdirectory(tmp_path): + revision = family_binding(tmp_path) + assert validate_binding(tmp_path, 'lexbench-websearch') == revision + + +@pytest.mark.parametrize('changed', ['project', 'lock', 'binding']) +def test_family_package_cannot_silently_select_another_subdirectory(tmp_path, changed): + family_binding(tmp_path) + relative = {'project': 'pyproject.toml', 'lock': 'uv.lock', + 'binding': 'evaluation/lexbench-websearch.json'}[changed] + path = tmp_path / relative + path.write_text(path.read_text().replace('packages/websearch', 'packages/another')) + with pytest.raises(ValueError, match='disagree'): + validate_binding(tmp_path, 'lexbench-websearch') + + +@pytest.mark.parametrize('directory', ['../websearch', '/packages/websearch', 'packages/../websearch']) +def test_family_source_requires_a_contained_package_directory(tmp_path, directory): + family_binding(tmp_path, directory=directory) + with pytest.raises(ValueError, match='subdirectory'): + validate_binding(tmp_path, 'lexbench-websearch') diff --git a/tests/test_readme_navigation.py b/tests/test_readme_navigation.py new file mode 100644 index 000000000..7e803939c --- /dev/null +++ b/tests/test_readme_navigation.py @@ -0,0 +1,31 @@ +"""Public onboarding keeps reachable navigation and rectangular comparison tables.""" +import re +from pathlib import Path + +import pytest + + +def check_navigation(text): + headings = { + re.sub(r'[^\w\s-]', '', heading.lower()).replace(' ', '-') + for heading in re.findall(r'^#{1,6} (.+)$', text, re.M) + } + for anchor in re.findall(r'\]\(#([^)]*)\)', text): + assert anchor in headings, f'Missing README section: {anchor}' + tables = re.findall(r'(?:^\|.*\|\n)+', text, re.M) + for table in tables: + assert len({len(row.split('|')) for row in table.splitlines()}) == 1 + + +@pytest.mark.parametrize('name', ['README.md', 'docs/README.zh-CN.md']) +def test_readme_navigation_and_tables(name): + check_navigation((Path(__file__).resolve().parents[1] / name).read_text()) + + +@pytest.mark.parametrize('text', [ + '[Quickstart](#quickstart)\n\n## Reports\n', + '| Track | Benchmark |\n| --- | --- |\n| Fetch |\n', +]) +def test_missing_sections_and_damaged_tables_are_rejected(text): + with pytest.raises(AssertionError): + check_navigation(text) From 6b0d892d90cf5abb57ec2363f9db839ee9248414 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 14:32:59 +0800 Subject: [PATCH 07/10] fix: isolate benchmark updates and retire legacy review --- .github/workflows/README.md | 6 +- .github/workflows/claude.yml | 109 --------------- CONTRIBUTING.md | 2 - Makefile | 2 +- harness/lexbench_pin.py | 20 +-- tests/test_lexbench_pin_migrations.py | 190 ++++++++++++++++++++++++++ 6 files changed, 206 insertions(+), 123 deletions(-) delete mode 100644 .github/workflows/claude.yml create mode 100644 tests/test_lexbench_pin_migrations.py diff --git a/.github/workflows/README.md b/.github/workflows/README.md index d378045e3..f7521a1f8 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,7 +1,7 @@ # GitHub workflows -`ci.yml` runs on main pushes and pull requests. Disposable GitHub-hosted runners install Python 3.11, uv 0.7.8 and Node.js 22, then run `make setup-test` and `make ci`. Contributors use the same local entry points. +`ci.yml` tests main pushes and pull requests. The base CLI runs on Python 3.12–3.14; full development checks use Python 3.11, uv 0.7.8 and Node.js 22. Contributors use `make setup-test` and `make ci` locally. -`claude.yml` provides optional English code review. Maintainers must set `ENABLE_CLAUDE_REVIEW=true` and `CLAUDE_CODE_OAUTH_TOKEN`; ordinary CI does not require them. Review uses GitHub-hosted runners, without a personal host or local Claude/Bun installation. +Code review is dispatched by the organization's [public OCR workflow](https://github.com/lexmount/github-policy-public/blob/main/.github/workflows/codex-code-review.yml). The dispatch job confirms durable task acceptance; the separate `ocr-review` status reports the backend's findings. The repository does not need a copied OCR workflow or local review credentials. Organization Rulesets manage review requirements separately from CI. -When enabled, opened, updated, ready-for-review and reopened non-draft PRs trigger review. `@claude` in issues, PR comments or reviews uses the interactive step and receives the comment's request. `CLAUDE_REVIEW_STANDARDS` owns shared review guidance and references `AGENTS.md` and relevant skills. New commits cancel older review runs for the same PR. +Project requirements remain in [AGENTS.md](../../AGENTS.md), with benchmark-specific requirements in [.agents/skills/benchmark-report](../../.agents/skills/benchmark-report/SKILL.md). Backend review-policy configuration is managed separately from the repository's workflows. diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml deleted file mode 100644 index 4b8e4b061..000000000 --- a/.github/workflows/claude.yml +++ /dev/null @@ -1,109 +0,0 @@ -name: Claude Review - -env: - CLAUDE_REVIEW_STANDARDS: >- - Work in browser-eval. Read and follow AGENTS.md before any code review. For benchmark, scoring, report or publication changes, also read .agents/skills/benchmark-report/SKILL.md; for interpretation articles, read .agents/skills/benchmark-editorial/SKILL.md. - Write reviews in English unless the interactive request explicitly asks for another language. Preserve identifiers, commands and quoted evidence. - Be concise: report the conclusion, actionable defects, recommendations, priorities and necessary file/line references. - Omit long summaries, checklists, execution narration and repeated instructions. - Check actual defects, requirements, implementation necessity, validation, redundant compatibility, evidence/scoring consistency, explicitly generated language parity, credential isolation and publication targets. Judge current code and evidence independently rather than accepting comments or historical conclusions blindly. - -concurrency: - group: claude-local-${{ github.event.pull_request.number || github.event.issue.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - -on: - issue_comment: - types: [created] - pull_request_review_comment: - types: [created] - issues: - types: [opened, assigned] - pull_request_review: - types: [submitted] - pull_request: - types: [opened, synchronize, ready_for_review, reopened] - -jobs: - claude: - name: Claude Code Review - if: | - vars.ENABLE_CLAUDE_REVIEW == 'true' && ( - (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) || - (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) || - (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) || - (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude'))) || - (github.event_name == 'pull_request' && github.event.pull_request.draft == false && (github.event.action == 'opened' || github.event.action == 'synchronize' || github.event.action == 'ready_for_review' || github.event.action == 'reopened'))) - runs-on: ubuntu-latest - permissions: - contents: read - pull-requests: write - issues: write - actions: read - steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - fetch-depth: 1 - - # Interactive @claude requests must not set `prompt`; the action uses the - # user's comment as the task. In this repository, a bare @claude or most - # @claude PR interactions are treated as review requests, so the same - # review standards still apply through `claude_args`. - - name: Run Claude Interactive - if: github.event_name != 'pull_request' - id: claude - uses: anthropics/claude-code-action@v1 - with: - claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} - github_token: ${{ secrets.GITHUB_TOKEN }} - - # This is an optional setting that allows Claude to read CI results on PRs - additional_permissions: | - actions: read - - claude_args: >- - --system-prompt "${{ env.CLAUDE_REVIEW_STANDARDS }}" - - # Pull request events have no user comment, so this step supplies the - # review task while reusing the same standards as interactive requests. - - name: Run Claude PR Review - if: github.event_name == 'pull_request' - id: claude_review - uses: anthropics/claude-code-action@v1 - with: - claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} - github_token: ${{ secrets.GITHUB_TOKEN }} - track_progress: true - - additional_permissions: | - actions: read - - prompt: | - REPO: ${{ github.repository }} - PR NUMBER: ${{ github.event.pull_request.number }} - - Review the current PR diff. - - Shared review standards: - ${{ env.CLAUDE_REVIEW_STANDARDS }} - - Requirements: - - Report every finding that affects the merge decision. - - Include priority, file/line, problem and recommendation for each finding. - - Use P0/P1/P2: P0 blocks merging, P1 should be fixed before merging, P2 can be followed up. - - Only post GitHub comments - don't submit review text as messages. - - Format: - ## Review conclusion - Pass / Conditional pass / Fail - - ## Findings and recommendations - - [P?] `path:line`: Problem. Recommendation: ... - - If no finding affects the merge decision, state that under - `## Findings and recommendations` and describe any residual risk briefly. - - claude_args: | - --system-prompt "${{ env.CLAUDE_REVIEW_STANDARDS }}" - --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5acb839bf..63ce1177a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -114,8 +114,6 @@ The archive excludes `.git`, untracked files and local installations. Extract it -Optional Claude review requires `ENABLE_CLAUDE_REVIEW=true` and `CLAUDE_CODE_OAUTH_TOKEN`. Ordinary CI needs neither. Review uses disposable GitHub-hosted runners. - ## License Contributions use the [Apache License 2.0](LICENSE). Contributors retain copyright in their contributions; submitting a contribution grants the applicable license and does not transfer ownership to LexMount. Preserve existing copyright notices and third-party terms described in [THIRD_PARTY](THIRD_PARTY.md). diff --git a/Makefile b/Makefile index d54161a63..270a8c8c7 100644 --- a/Makefile +++ b/Makefile @@ -216,4 +216,4 @@ test-native-adapters: .PHONY: bump-lexbench bump-lexbench: @test -n "$(REV)" || (echo 'REV= is required' >&2; exit 2) - $(PY) scripts/bump_lexbench.py --revision "$(REV)" --uv "$(UV)" --benchmark "$(or $(BENCHMARK),lexbench-webfetch)" $(if $(REPOSITORY),--repository "$(REPOSITORY)",) $(if $(SUBDIRECTORY),--subdirectory "$(SUBDIRECTORY)",) + $(PY) scripts/bump_lexbench.py --revision "$(REV)" --uv "$(UV)" --benchmark "$(or $(BENCHMARK),lexbench-webfetch)" $(if $(REPOSITORY),--repository "$(REPOSITORY)",) $(if $(filter undefined,$(origin SUBDIRECTORY)),,--subdirectory "$(SUBDIRECTORY)") diff --git a/harness/lexbench_pin.py b/harness/lexbench_pin.py index 7e1416afa..a8123a802 100644 --- a/harness/lexbench_pin.py +++ b/harness/lexbench_pin.py @@ -25,13 +25,18 @@ def _subdirectory(value: str) -> str: return value -def project_pin(project: dict, name: str = NAME) -> tuple[str, str, str]: +def _project_dependency(project: dict, name: str) -> str: matches = [item for item in project["project"]["dependencies"] if re.match(rf"^{re.escape(name)}\s*@", item)] if len(matches) != 1: raise ValueError("Expected one direct LexBench Git dependency. " + HINT) + return matches[0] + + +def project_pin(project: dict, name: str = NAME) -> tuple[str, str, str]: match = re.fullmatch( - rf"{re.escape(name)}\s*@\s*git\+([^#]+)@([0-9a-f]{{40}})(?:#(.*))?", matches[0]) + rf"{re.escape(name)}\s*@\s*git\+([^#]+)@([0-9a-f]{{40}})(?:#(.*))?", + _project_dependency(project, name)) if not match: raise ValueError("LexBench must use a full immutable Git commit. " + HINT) fragment = parse_qs(match.group(3) or '', keep_blank_values=True) @@ -124,7 +129,7 @@ def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME, repository: str | None = None, subdirectory: str | None = None) -> list[str]: project_path = root / "pyproject.toml" text = project_path.read_text() - old_repository, old_revision, old_directory = project_pin(tomllib.loads(text), benchmark) + old_repository, _old_revision, old_directory = project_pin(tomllib.loads(text), benchmark) repository = repository if repository is not None else old_repository directory = _subdirectory(subdirectory if subdirectory is not None else old_directory) relative_binding = 'evaluation/' + benchmark + '.json' @@ -138,15 +143,14 @@ def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME, (project / "evaluation").mkdir() for name in ("uv.lock", ".python-version", "README.md", "LICENSE"): shutil.copyfile(root / name, project / name) - old_url = f"git+{old_repository}@{old_revision}" - if old_directory: - old_url += '#subdirectory=' + old_directory - if text.count(old_url) != 1: + declaration = _project_dependency(tomllib.loads(text), benchmark) + if text.count(declaration) != 1: raise ValueError("Ambiguous dependency declaration") new_url = f"git+{repository}@{revision}" if directory: new_url += '#subdirectory=' + directory - (project / "pyproject.toml").write_text(text.replace(old_url, new_url)) + updated = declaration[:declaration.index('git+')] + new_url + (project / "pyproject.toml").write_text(text.replace(declaration, updated)) run([uv, "lock", "--project", str(project)], project) run([uv, "sync", "--frozen", "--no-install-project", "--project", str(project)], project) definitions = Path(run([ diff --git a/tests/test_lexbench_pin_migrations.py b/tests/test_lexbench_pin_migrations.py new file mode 100644 index 000000000..73c8dfe12 --- /dev/null +++ b/tests/test_lexbench_pin_migrations.py @@ -0,0 +1,190 @@ +"""Exercise public dependency migration with real local Git sources and uv.""" + +import hashlib +import json +import shutil +import subprocess +import tomllib +from pathlib import Path + +import pytest + +from harness.lexbench_pin import bump, project_pin, validate_binding + +ROOT = Path(__file__).resolve().parents[1] + + +def git(repo, *args): + return subprocess.check_output(["git", *args], cwd=repo, text=True).strip() + + +def package(root, name, definitions): + module = name.replace("-", "_") + source = root / "src" / module + source.mkdir(parents=True) + (source / "__init__.py").write_text("") + (source / "inputs.py").write_text( + "from pathlib import Path\n" + "def definition_path(relative):\n" + ' return Path(__file__).parent / "definitions" / relative\n' + ) + for name_in_package, data in definitions.items(): + path = source / "definitions" / name_in_package + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(data)) + (root / "pyproject.toml").write_text( + f'[project]\nname = "{name}"\nversion = "1.0.0"\nrequires-python = ">=3.11"\n' + '[build-system]\nrequires = ["setuptools>=77"]\nbuild-backend = "setuptools.build_meta"\n' + '[tool.setuptools.packages.find]\nwhere = ["src"]\n' + f'[tool.setuptools.package-data]\n{module} = ["definitions/*.json", "definitions/evaluation/*/*.json"]\n' + ) + return source / "definitions" + + +@pytest.fixture(scope="module") +def sources(tmp_path_factory): + root = tmp_path_factory.mktemp("benchmark-family") + family = root / "family" + family.mkdir() + task = {"id": "tasks", "cases": [{"id": "first"}]} + fetch = package(family, "lexbench-webfetch", {"evaluation/example/tasks.json": task}) + raw = json.dumps({"id": "questions", "cases": [{"id": "query"}]}).encode() + registry = { + "datasets": { + "questions": { + "path": "questions.json", + "count": 1, + "sha256": hashlib.sha256(raw).hexdigest(), + } + } + } + search = package( + family / "packages/websearch", + "lexbench-websearch", + {"registry.json": registry, "questions.json": json.loads(raw)}, + ) + with (family / "pyproject.toml").open("a") as stream: + stream.write('\n[tool.uv.workspace]\nmembers = ["packages/*"]\n') + git(family, "init", "-b", "main") + git(family, "add", ".") + git( + family, + "-c", + "user.name=Test", + "-c", + "user.email=test@example.com", + "commit", + "-qm", + "Initial family", + ) + before = git(family, "rev-parse", "HEAD") + (family / "revision.txt").write_text("A tooling-only successor") + git(family, "add", ".") + git( + family, + "-c", + "user.name=Test", + "-c", + "user.email=test@example.com", + "commit", + "-qm", + "Successor", + ) + after = git(family, "rev-parse", "HEAD") + standalone = root / "standalone" + standalone.mkdir() + package( + standalone, + "lexbench-websearch", + {"registry.json": registry, "questions.json": json.loads(raw)}, + ) + git(standalone, "init", "-b", "main") + git(standalone, "add", ".") + git( + standalone, + "-c", + "user.name=Test", + "-c", + "user.email=test@example.com", + "commit", + "-qm", + "Standalone Search", + ) + return family, before, after, standalone, git(standalone, "rev-parse", "HEAD"), fetch, search + + +def consumer(root, sources): + family, before, _, _, _, _, _ = sources + url = family.as_uri() + (root / "evaluation").mkdir() + (root / "pyproject.toml").write_text( + '[project]\nname = "consumer"\nversion = "0.0.0"\nrequires-python = ">=3.11"\ndependencies = [\n' + f' "lexbench-webfetch @ git+{url}@{before}",\n' + f' "lexbench-websearch @ git+{url}@{before}#subdirectory=packages/websearch",\n]\n' + ) + for name in ["README.md", "LICENSE", "NOTICE", "THIRD_PARTY.md"]: + (root / name).write_text("Test fixture\n") + (root / ".python-version").write_text("3.11\n") + subprocess.run(["uv", "lock", "--project", str(root)], check=True, capture_output=True) + for name in ["lexbench-webfetch", "lexbench-websearch"]: + (root / "evaluation" / f"{name}.json").write_text( + json.dumps( + { + "repository": url, + "revision": before, + **( + {"subdirectory": "packages/websearch", "datasets": {}} + if "search" in name + else {"suites": []} + ), + } + ) + ) + + +@pytest.mark.parametrize("selected", ["lexbench-webfetch", "lexbench-websearch"]) +def test_one_family_package_can_update_without_changing_its_sibling(tmp_path, sources, selected): + consumer(tmp_path, sources) + _, before, after, *_ = sources + sibling = "lexbench-websearch" if selected == "lexbench-webfetch" else "lexbench-webfetch" + sibling_binding = (tmp_path / "evaluation" / f"{sibling}.json").read_bytes() + bump(tmp_path, after, benchmark=selected) + assert validate_binding(tmp_path, selected) == after + assert validate_binding(tmp_path, sibling) == before + assert (tmp_path / "evaluation" / f"{sibling}.json").read_bytes() == sibling_binding + + +def test_make_can_clear_a_subdirectory_when_migrating_to_a_root_package(tmp_path, sources): + consumer(tmp_path, sources) + _, before, _, standalone, revision, *_ = sources + (tmp_path / "scripts").mkdir() + (tmp_path / "harness").mkdir() + shutil.copyfile(ROOT / "scripts/bump_lexbench.py", tmp_path / "scripts/bump_lexbench.py") + shutil.copyfile(ROOT / "harness/lexbench_pin.py", tmp_path / "harness/lexbench_pin.py") + subprocess.run( + [ + "make", + "-f", + str(ROOT / "Makefile"), + "bump-lexbench", + f"REV={revision}", + "BENCHMARK=lexbench-websearch", + f"REPOSITORY={standalone.as_uri()}", + "SUBDIRECTORY=", + ], + cwd=tmp_path, + check=True, + capture_output=True, + text=True, + ) + assert validate_binding(tmp_path, "lexbench-websearch") == revision + assert validate_binding(tmp_path, "lexbench-webfetch") == before + assert ( + project_pin(tomllib.loads((tmp_path / "pyproject.toml").read_text()), "lexbench-websearch")[ + 2 + ] + == "" + ) + assert "subdirectory" not in json.loads( + (tmp_path / "evaluation/lexbench-websearch.json").read_text() + ) From 66ac9948b08bd31ec530d5de2fdeedd9377b11d1 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 19:46:45 +0800 Subject: [PATCH 08/10] docs: retain independent benchmark repositories --- Makefile | 2 +- README.md | 11 +- docs/README.zh-CN.md | 11 +- docs/benchmarks.md | 18 +-- docs/dependencies.md | 5 +- harness/lexbench_pin.py | 55 ++----- tests/test_lexbench_dependency_update.py | 40 ----- tests/test_lexbench_pin_migrations.py | 190 ----------------------- 8 files changed, 28 insertions(+), 304 deletions(-) delete mode 100644 tests/test_lexbench_pin_migrations.py diff --git a/Makefile b/Makefile index 270a8c8c7..5d877bd24 100644 --- a/Makefile +++ b/Makefile @@ -216,4 +216,4 @@ test-native-adapters: .PHONY: bump-lexbench bump-lexbench: @test -n "$(REV)" || (echo 'REV= is required' >&2; exit 2) - $(PY) scripts/bump_lexbench.py --revision "$(REV)" --uv "$(UV)" --benchmark "$(or $(BENCHMARK),lexbench-webfetch)" $(if $(REPOSITORY),--repository "$(REPOSITORY)",) $(if $(filter undefined,$(origin SUBDIRECTORY)),,--subdirectory "$(SUBDIRECTORY)") + $(PY) scripts/bump_lexbench.py --revision "$(REV)" --uv "$(UV)" --benchmark "$(or $(BENCHMARK),lexbench-webfetch)" diff --git a/README.md b/README.md index 67c524ac7..539b01338 100644 --- a/README.md +++ b/README.md @@ -16,12 +16,12 @@ Browser Eval is an evaluation harness for browsers and web agents. Connect a ben ## Evaluation tracks -Each track selects its own benchmark, task population and scoring rules. The harness supplies execution adapters, saved evidence and reports. +The main branch includes the three browser workflows below. Web Search is implemented on the development branch linked in its row; it is not included in the current main-branch package. | Track | What it tests | Benchmarks and task suites | Guide | | --- | --- | --- | --- | -| Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | -| Web Search / SERP | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md): scenario queries and a SimpleQA Verified retrieval adaptation | [Offline example](examples/web_search/README.md) · [Track guide](harness/web_search/README.md) | +| Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | +| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Implementation and integration PR #58](https://github.com/lexmount/browser-eval/pull/58) · [Search guide](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | | Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/tasks.v4.json), adapted from [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | @@ -65,13 +65,14 @@ To share a report online, publish its `report/` directory through [Artifact Site The current package version is `0.2.0`. Software tags use `vMAJOR.MINOR.PATCH`; benchmark editions and hosted report versions are independent. A report records all three identities so rebuilding a report does not relabel its tasks or saved calls. -Use [the release guide](docs/releases.md) for compatibility rules, release checks and the scope of the first software release. A stable release commits to documented commands, input contracts and saved-evidence compatibility; it does not guarantee that every browser or live website passes every task. +Use [the release guide](docs/releases.md) for compatibility rules, release checks and the scope of the first software release. Web Search remains a preview until its integration reaches main. A stable release commits to documented commands, input contracts and saved-evidence compatibility; it does not guarantee that every browser or live website passes every task. ## Related repositories | Repository | Role | | --- | --- | -| [LexBench Web Retrieval](https://github.com/lexmount/lexbench-web-retrieval) | Benchmark family: [Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) and [Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md), with separate task populations and scoring | +| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Versioned Web Fetch tasks, acceptance scoring and frozen input bindings | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Versioned search queries, source references and retrieval scoring | | [Moli](https://github.com/lexmount/moli) | Browser evaluated through the same task contracts as other participants | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service; separate from the evaluation harness | | [Artifact Site](https://github.com/lexmount/artifact-site) | Report hosting, immutable versions and access controls | diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 94612eaeb..0b132a169 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -16,12 +16,12 @@ Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation har ## 评测流程 -每种评测流程分别选择评测集、题目范围和评分规则；脚手架负责适配器调用、证据保存和报告。 +主分支包含下列三种浏览器评测流程。Web Search 已在表中链接的开发分支实现，当前主分支的软件包尚未包含它。 | 流程 | 测什么 | 对应评测集与任务集 | 使用指南 | | --- | --- | --- | --- | -| Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | -| Web Search / SERP | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [离线示例](../examples/web_search/README.md) · [完整指南](../harness/web_search/README.md) | +| Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | +| Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [实现与接入 PR #58](https://github.com/lexmount/browser-eval/pull/58) · [搜索评测指南](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | | 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/tasks.v4.json)，改编自 [Lexbench-Headless-Browser 场景设计](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios)；[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json)，由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)，使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | @@ -65,13 +65,14 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run 当前软件包版本是 `0.2.0`。软件标签采用 `v主版本.次版本.补丁版本`，评测集和托管报告各有独立版本。报告记录这三种身份，因此重建报告不会改换题目版本或保存的调用。 -兼容规则、发布检查和首个软件发行版的范围见[发布指南](../docs/releases.md)。稳定版承诺的是文档中的命令、输入约定和历史证据兼容性；浏览器与在线网站的逐题表现仍由评测结果说明。 +兼容规则、发布检查和首个软件发行版的范围见[发布指南](../docs/releases.md)。Web Search 在接入主分支前保持预览状态。稳定版承诺的是文档中的命令、输入约定和历史证据兼容性；浏览器与在线网站的逐题表现仍由评测结果说明。 ## 相关仓库 | 仓库 | 用途 | | --- | --- | -| [LexBench Web Retrieval](https://github.com/lexmount/lexbench-web-retrieval) | 评测家族：[Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) 和 [Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md) 各自维护题目和评分 | +| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Web Fetch 题目、验收评分和冻结输入绑定 | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 | | [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 | | [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | | [Artifact Site](https://github.com/lexmount/artifact-site) | 报告托管、不可变版本和访问权限 | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index c6c3fce2d..d51044d02 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -4,9 +4,9 @@ Browser Eval executes tasks, adapts browsers and providers, calls the selected b | Responsibility | Owner | | --- | --- | -| Self-authored Web Fetch task populations, prompts and acceptance requirements | [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) | +| Self-authored Web Fetch task populations, prompts and acceptance requirements | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | | LexBench task catalogs and frozen source-input bindings | LexBench Web Fetch | -| Web Search tasks, provenance, source qualifications and deterministic retrieval scoring | [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md) | +| Web Search tasks, provenance, source qualifications and deterministic retrieval scoring | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | | Browser/provider adapters, execution policy and credential loading | Browser Eval | | Task model, deterministic acceptance scorer, file-byte checks, compiler and scenario registry | LexBench Web Fetch | | Rescoring orchestration and evidence revision machinery | Browser Eval | @@ -275,17 +275,3 @@ LexBench Web Search is installed from the immutable commit in `pyproject.toml` a Change search queries, references or scoring in LexBench Web Search and pin the resulting commit here. Change provider behavior or report generation in Browser Eval. See the pinned benchmark's [data provenance](https://github.com/lexmount/lexbench-websearch/blob/a6d7b92987e32ef52563ee68bac4343db5686d95/docs/data.md) and [evaluation protocol](https://github.com/lexmount/lexbench-websearch/blob/a6d7b92987e32ef52563ee68bac4343db5686d95/docs/evaluation.md). Update a merged Search benchmark revision with `make bump-lexbench BENCHMARK=lexbench-websearch REV=`. The shared updater checks upstream main ancestry and updates the package, lock and dataset-hash binding together. The initial extraction PR depends on the companion benchmark PR landing first. - -## Select a package from the retrieval family - -[LexBench Web Retrieval](https://github.com/lexmount/lexbench-web-retrieval) keeps Fetch at the repository root and Search in `packages/websearch/`. Their package names and Python imports remain independent. A Git source for Search must include that subdirectory; binding only a repository and commit would select the wrong build root. - -Use the existing updater to select a merged family commit. It stages installation, checks the full main-branch ancestry, and updates the project, lock and definition binding together. For Search: - -```sh -make bump-lexbench BENCHMARK=lexbench-websearch REV= \ - REPOSITORY=https://github.com/lexmount/lexbench-web-retrieval.git \ - SUBDIRECTORY=packages/websearch -``` - -For Fetch, select the same repository without `SUBDIRECTORY`. Installed definitions and frozen scorer bytes retain their exact identities. Historical pins continue to identify their original commits; a repository migration does not authorize changed tasks, reference answers or benchmark editions. diff --git a/docs/dependencies.md b/docs/dependencies.md index b5ff297c5..749a0a554 100644 --- a/docs/dependencies.md +++ b/docs/dependencies.md @@ -14,11 +14,12 @@ Use the declared Git source for the evaluator. A registry package with the same | Dependency | Role | Version authority | | --- | --- | --- | -| [LexBench Web Fetch](https://github.com/lexmount/lexbench-web-retrieval/blob/main/docs/webfetch.md) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | -| [LexBench Web Search](https://github.com/lexmount/lexbench-web-retrieval/blob/main/packages/websearch/README.md) | Search queries, source references and deterministic retrieval scoring | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md#web-search-integration) | +| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | | [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) | HTML parsing | `pyproject.toml` and `uv.lock` | | [FastCDC](https://github.com/iscc/fastcdc-py) | Content chunking | `pyproject.toml` and `uv.lock` | +Search development uses [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) for query definitions and retrieval scoring. Its integration is tracked in [PR #58](https://github.com/lexmount/browser-eval/pull/58); it is not a dependency of the current main-branch package. + ## Git-based runtime dependencies | Dependency | Repository and relevant branch | Required source | diff --git a/harness/lexbench_pin.py b/harness/lexbench_pin.py index a8123a802..4435f4869 100644 --- a/harness/lexbench_pin.py +++ b/harness/lexbench_pin.py @@ -18,37 +18,19 @@ ROOT = Path(__file__).resolve().parents[1] -def _subdirectory(value: str) -> str: - if value and (not re.fullmatch(r"[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)*", value) - or any(part in {'.', '..'} for part in value.split('/'))): - raise ValueError("LexBench subdirectory must stay within the repository") - return value - - -def _project_dependency(project: dict, name: str) -> str: +def project_pin(project: dict, name: str = NAME) -> tuple[str, str]: matches = [item for item in project["project"]["dependencies"] if re.match(rf"^{re.escape(name)}\s*@", item)] if len(matches) != 1: raise ValueError("Expected one direct LexBench Git dependency. " + HINT) - return matches[0] - - -def project_pin(project: dict, name: str = NAME) -> tuple[str, str, str]: - match = re.fullmatch( - rf"{re.escape(name)}\s*@\s*git\+([^#]+)@([0-9a-f]{{40}})(?:#(.*))?", - _project_dependency(project, name)) + match = re.fullmatch(rf"{re.escape(name)}\s*@\s*git\+(.+)@([0-9a-f]{{40}})", matches[0]) if not match: raise ValueError("LexBench must use a full immutable Git commit. " + HINT) - fragment = parse_qs(match.group(3) or '', keep_blank_values=True) - if fragment and (set(fragment) != {'subdirectory'} or len(fragment['subdirectory']) != 1): - raise ValueError("Invalid LexBench subdirectory binding") - directory = fragment.get('subdirectory', [''])[0] - return match.group(1), match.group(2), _subdirectory(directory) + return match.group(1), match.group(2) def validate_binding(root: Path, name: str = NAME) -> str: - repository, revision, directory = project_pin( - tomllib.loads((root / "pyproject.toml").read_text()), name) + repository, revision = project_pin(tomllib.loads((root / "pyproject.toml").read_text()), name) lock = tomllib.loads((root / "uv.lock").read_text()) packages = [item for item in lock["package"] if item["name"] == name] binding = json.loads((root / "evaluation" / (name + '.json')).read_text()) @@ -58,8 +40,6 @@ def validate_binding(root: Path, name: str = NAME) -> str: locked_repository = source._replace(query="", fragment="").geturl() if (locked_repository != repository or source.fragment != revision or parse_qs(source.query).get("rev") != [revision] - or parse_qs(source.query).get("subdirectory", ['']) != [directory] - or binding.get("subdirectory", '') != directory or binding["revision"] != revision or binding["repository"].removesuffix(".git") != repository.removesuffix(".git")): raise ValueError("LexBench project, lock and definition binding disagree. " + HINT) @@ -125,13 +105,10 @@ def require_merged_revision(repository: str, revision: str, directory: Path) -> raise ValueError("Requested LexBench revision is not in upstream main history") -def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME, - repository: str | None = None, subdirectory: str | None = None) -> list[str]: +def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME) -> list[str]: project_path = root / "pyproject.toml" text = project_path.read_text() - old_repository, _old_revision, old_directory = project_pin(tomllib.loads(text), benchmark) - repository = repository if repository is not None else old_repository - directory = _subdirectory(subdirectory if subdirectory is not None else old_directory) + repository, old_revision = project_pin(tomllib.loads(text), benchmark) relative_binding = 'evaluation/' + benchmark + '.json' binding_path = root / relative_binding before = json.loads(binding_path.read_text()) @@ -143,14 +120,10 @@ def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME, (project / "evaluation").mkdir() for name in ("uv.lock", ".python-version", "README.md", "LICENSE"): shutil.copyfile(root / name, project / name) - declaration = _project_dependency(tomllib.loads(text), benchmark) - if text.count(declaration) != 1: + old_url = f"git+{repository}@{old_revision}" + if text.count(old_url) != 1: raise ValueError("Ambiguous dependency declaration") - new_url = f"git+{repository}@{revision}" - if directory: - new_url += '#subdirectory=' + directory - updated = declaration[:declaration.index('git+')] + new_url - (project / "pyproject.toml").write_text(text.replace(declaration, updated)) + (project / "pyproject.toml").write_text(text.replace(old_url, f"git+{repository}@{revision}")) run([uv, "lock", "--project", str(project)], project) run([uv, "sync", "--frozen", "--no-install-project", "--project", str(project)], project) definitions = Path(run([ @@ -158,11 +131,6 @@ def bump(root: Path, revision: str, *, uv: str = "uv", benchmark: str = NAME, f"from {benchmark.replace('-', '_')}.inputs import definition_path; print(definition_path({'registry.json' if benchmark == 'lexbench-websearch' else 'evaluation'!r}).parent)", ], project)) after = refresh_binding(before, revision, definitions) - after['repository'] = repository.removesuffix('.git') - if directory: - after['subdirectory'] = directory - else: - after.pop('subdirectory', None) (project / relative_binding).write_text( json.dumps(after, ensure_ascii=False, indent=2) + "\n") validate_binding(project, benchmark) @@ -184,10 +152,7 @@ def main(argv=None): parser.add_argument("--revision", required=True) parser.add_argument("--uv", default="uv") parser.add_argument('--benchmark', choices=['lexbench-webfetch', 'lexbench-websearch'], default=NAME) - parser.add_argument('--repository', help='Git repository for a source migration') - parser.add_argument('--subdirectory', help='Package directory within that repository') args = parser.parse_args(argv) - for change in bump(ROOT, args.revision, uv=args.uv, benchmark=args.benchmark, - repository=args.repository, subdirectory=args.subdirectory): + for change in bump(ROOT, args.revision, uv=args.uv, benchmark=args.benchmark): print(change) print("Verified current dependency binding. Historical evidence was not changed.") diff --git a/tests/test_lexbench_dependency_update.py b/tests/test_lexbench_dependency_update.py index 81b93b583..b43b73c1c 100644 --- a/tests/test_lexbench_dependency_update.py +++ b/tests/test_lexbench_dependency_update.py @@ -115,43 +115,3 @@ def git(*args): require_merged_revision(str(repo), merged, tmp_path / "accepted") with pytest.raises(ValueError, match="not in upstream main"): require_merged_revision(str(repo), unmerged, tmp_path / "rejected") - - -def family_binding(root, *, directory='packages/websearch'): - revision = 'a' * 40 - repository = 'https://github.com/lexmount/lexbench-web-retrieval.git' - (root / 'evaluation').mkdir(exist_ok=True) - (root / 'pyproject.toml').write_text( - '[project]\ndependencies = ["lexbench-websearch @ git+' - + repository + '@' + revision + '#subdirectory=' + directory + '"]\n') - (root / 'uv.lock').write_text( - '[[package]]\nname = "lexbench-websearch"\nsource = { git = "' - + repository + '?rev=' + revision + '&subdirectory=' + directory + '#' + revision + '" }\n') - (root / 'evaluation/lexbench-websearch.json').write_text(json.dumps({ - 'repository': repository.removesuffix('.git'), 'revision': revision, - 'subdirectory': directory, - })) - return revision - - -def test_family_package_pin_binds_repository_revision_and_subdirectory(tmp_path): - revision = family_binding(tmp_path) - assert validate_binding(tmp_path, 'lexbench-websearch') == revision - - -@pytest.mark.parametrize('changed', ['project', 'lock', 'binding']) -def test_family_package_cannot_silently_select_another_subdirectory(tmp_path, changed): - family_binding(tmp_path) - relative = {'project': 'pyproject.toml', 'lock': 'uv.lock', - 'binding': 'evaluation/lexbench-websearch.json'}[changed] - path = tmp_path / relative - path.write_text(path.read_text().replace('packages/websearch', 'packages/another')) - with pytest.raises(ValueError, match='disagree'): - validate_binding(tmp_path, 'lexbench-websearch') - - -@pytest.mark.parametrize('directory', ['../websearch', '/packages/websearch', 'packages/../websearch']) -def test_family_source_requires_a_contained_package_directory(tmp_path, directory): - family_binding(tmp_path, directory=directory) - with pytest.raises(ValueError, match='subdirectory'): - validate_binding(tmp_path, 'lexbench-websearch') diff --git a/tests/test_lexbench_pin_migrations.py b/tests/test_lexbench_pin_migrations.py deleted file mode 100644 index 73c8dfe12..000000000 --- a/tests/test_lexbench_pin_migrations.py +++ /dev/null @@ -1,190 +0,0 @@ -"""Exercise public dependency migration with real local Git sources and uv.""" - -import hashlib -import json -import shutil -import subprocess -import tomllib -from pathlib import Path - -import pytest - -from harness.lexbench_pin import bump, project_pin, validate_binding - -ROOT = Path(__file__).resolve().parents[1] - - -def git(repo, *args): - return subprocess.check_output(["git", *args], cwd=repo, text=True).strip() - - -def package(root, name, definitions): - module = name.replace("-", "_") - source = root / "src" / module - source.mkdir(parents=True) - (source / "__init__.py").write_text("") - (source / "inputs.py").write_text( - "from pathlib import Path\n" - "def definition_path(relative):\n" - ' return Path(__file__).parent / "definitions" / relative\n' - ) - for name_in_package, data in definitions.items(): - path = source / "definitions" / name_in_package - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(data)) - (root / "pyproject.toml").write_text( - f'[project]\nname = "{name}"\nversion = "1.0.0"\nrequires-python = ">=3.11"\n' - '[build-system]\nrequires = ["setuptools>=77"]\nbuild-backend = "setuptools.build_meta"\n' - '[tool.setuptools.packages.find]\nwhere = ["src"]\n' - f'[tool.setuptools.package-data]\n{module} = ["definitions/*.json", "definitions/evaluation/*/*.json"]\n' - ) - return source / "definitions" - - -@pytest.fixture(scope="module") -def sources(tmp_path_factory): - root = tmp_path_factory.mktemp("benchmark-family") - family = root / "family" - family.mkdir() - task = {"id": "tasks", "cases": [{"id": "first"}]} - fetch = package(family, "lexbench-webfetch", {"evaluation/example/tasks.json": task}) - raw = json.dumps({"id": "questions", "cases": [{"id": "query"}]}).encode() - registry = { - "datasets": { - "questions": { - "path": "questions.json", - "count": 1, - "sha256": hashlib.sha256(raw).hexdigest(), - } - } - } - search = package( - family / "packages/websearch", - "lexbench-websearch", - {"registry.json": registry, "questions.json": json.loads(raw)}, - ) - with (family / "pyproject.toml").open("a") as stream: - stream.write('\n[tool.uv.workspace]\nmembers = ["packages/*"]\n') - git(family, "init", "-b", "main") - git(family, "add", ".") - git( - family, - "-c", - "user.name=Test", - "-c", - "user.email=test@example.com", - "commit", - "-qm", - "Initial family", - ) - before = git(family, "rev-parse", "HEAD") - (family / "revision.txt").write_text("A tooling-only successor") - git(family, "add", ".") - git( - family, - "-c", - "user.name=Test", - "-c", - "user.email=test@example.com", - "commit", - "-qm", - "Successor", - ) - after = git(family, "rev-parse", "HEAD") - standalone = root / "standalone" - standalone.mkdir() - package( - standalone, - "lexbench-websearch", - {"registry.json": registry, "questions.json": json.loads(raw)}, - ) - git(standalone, "init", "-b", "main") - git(standalone, "add", ".") - git( - standalone, - "-c", - "user.name=Test", - "-c", - "user.email=test@example.com", - "commit", - "-qm", - "Standalone Search", - ) - return family, before, after, standalone, git(standalone, "rev-parse", "HEAD"), fetch, search - - -def consumer(root, sources): - family, before, _, _, _, _, _ = sources - url = family.as_uri() - (root / "evaluation").mkdir() - (root / "pyproject.toml").write_text( - '[project]\nname = "consumer"\nversion = "0.0.0"\nrequires-python = ">=3.11"\ndependencies = [\n' - f' "lexbench-webfetch @ git+{url}@{before}",\n' - f' "lexbench-websearch @ git+{url}@{before}#subdirectory=packages/websearch",\n]\n' - ) - for name in ["README.md", "LICENSE", "NOTICE", "THIRD_PARTY.md"]: - (root / name).write_text("Test fixture\n") - (root / ".python-version").write_text("3.11\n") - subprocess.run(["uv", "lock", "--project", str(root)], check=True, capture_output=True) - for name in ["lexbench-webfetch", "lexbench-websearch"]: - (root / "evaluation" / f"{name}.json").write_text( - json.dumps( - { - "repository": url, - "revision": before, - **( - {"subdirectory": "packages/websearch", "datasets": {}} - if "search" in name - else {"suites": []} - ), - } - ) - ) - - -@pytest.mark.parametrize("selected", ["lexbench-webfetch", "lexbench-websearch"]) -def test_one_family_package_can_update_without_changing_its_sibling(tmp_path, sources, selected): - consumer(tmp_path, sources) - _, before, after, *_ = sources - sibling = "lexbench-websearch" if selected == "lexbench-webfetch" else "lexbench-webfetch" - sibling_binding = (tmp_path / "evaluation" / f"{sibling}.json").read_bytes() - bump(tmp_path, after, benchmark=selected) - assert validate_binding(tmp_path, selected) == after - assert validate_binding(tmp_path, sibling) == before - assert (tmp_path / "evaluation" / f"{sibling}.json").read_bytes() == sibling_binding - - -def test_make_can_clear_a_subdirectory_when_migrating_to_a_root_package(tmp_path, sources): - consumer(tmp_path, sources) - _, before, _, standalone, revision, *_ = sources - (tmp_path / "scripts").mkdir() - (tmp_path / "harness").mkdir() - shutil.copyfile(ROOT / "scripts/bump_lexbench.py", tmp_path / "scripts/bump_lexbench.py") - shutil.copyfile(ROOT / "harness/lexbench_pin.py", tmp_path / "harness/lexbench_pin.py") - subprocess.run( - [ - "make", - "-f", - str(ROOT / "Makefile"), - "bump-lexbench", - f"REV={revision}", - "BENCHMARK=lexbench-websearch", - f"REPOSITORY={standalone.as_uri()}", - "SUBDIRECTORY=", - ], - cwd=tmp_path, - check=True, - capture_output=True, - text=True, - ) - assert validate_binding(tmp_path, "lexbench-websearch") == revision - assert validate_binding(tmp_path, "lexbench-webfetch") == before - assert ( - project_pin(tomllib.loads((tmp_path / "pyproject.toml").read_text()), "lexbench-websearch")[ - 2 - ] - == "" - ) - assert "subdirectory" not in json.loads( - (tmp_path / "evaluation/lexbench-websearch.json").read_text() - ) From 4125e83fbc789b80a433b2dd4306420f976d21bd Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 20:22:53 +0800 Subject: [PATCH 09/10] docs: describe integrated Search track accurately --- README.md | 6 +++--- docs/README.zh-CN.md | 6 +++--- docs/dependencies.md | 3 +-- docs/releases.md | 2 +- 4 files changed, 8 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 539b01338..249ec7184 100644 --- a/README.md +++ b/README.md @@ -16,12 +16,12 @@ Browser Eval is an evaluation harness for browsers and web agents. Connect a ben ## Evaluation tracks -The main branch includes the three browser workflows below. Web Search is implemented on the development branch linked in its row; it is not included in the current main-branch package. +The main branch includes all four tracks below. Each track provides a runnable example and a guide to its benchmark inputs and environment. | Track | What it tests | Benchmarks and task suites | Guide | | --- | --- | --- | --- | | Web Fetch | Retrieve the requested page content, records, resources or files | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch), [WebMainBench](https://github.com/opendatalab/WebMainBench), [WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [Local example](examples/web_fetch/README.md) · [Full guide](harness/web_fetch/README.md) | -| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Implementation and integration PR #58](https://github.com/lexmount/browser-eval/pull/58) · [Search guide](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | +| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch): scenario queries and a SimpleQA Verified retrieval adaptation | [Local example](examples/web_search/README.md) · [Full guide](harness/web_search/README.md) | | Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/tasks.v4.json), adapted from [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | @@ -65,7 +65,7 @@ To share a report online, publish its `report/` directory through [Artifact Site The current package version is `0.2.0`. Software tags use `vMAJOR.MINOR.PATCH`; benchmark editions and hosted report versions are independent. A report records all three identities so rebuilding a report does not relabel its tasks or saved calls. -Use [the release guide](docs/releases.md) for compatibility rules, release checks and the scope of the first software release. Web Search remains a preview until its integration reaches main. A stable release commits to documented commands, input contracts and saved-evidence compatibility; it does not guarantee that every browser or live website passes every task. +Use [the release guide](docs/releases.md) for compatibility rules, release checks and the scope of the first software release. Web Search is available as a preview; stable support is assessed through the release checks. A stable release commits to documented commands, input contracts and saved-evidence compatibility; it does not guarantee that every browser or live website passes every task. ## Related repositories diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 0b132a169..bfcf21432 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -16,12 +16,12 @@ Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation har ## 评测流程 -主分支包含下列三种浏览器评测流程。Web Search 已在表中链接的开发分支实现，当前主分支的软件包尚未包含它。 +主分支已包含下列四种评测流程。各流程均提供可运行的示例，以及评测输入和环境配置指南。 | 流程 | 测什么 | 对应评测集与任务集 | 使用指南 | | --- | --- | --- | --- | | Web Fetch | 获取任务要求的网页正文、记录、资源或文件 | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)、[WebMainBench](https://github.com/opendatalab/WebMainBench)、[WCXB](https://github.com/Murrough-Foley/web-content-extraction-benchmark) | [本地示例](../examples/web_fetch/README.md) · [完整指南](../harness/web_fetch/README.md) | -| Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [实现与接入 PR #58](https://github.com/lexmount/browser-eval/pull/58) · [搜索评测指南](https://github.com/lexmount/browser-eval/blob/codex/report-release-checks/harness/web_search/README.md) | +| Web Search（预览） | 比较搜索结果、相关性、来源检索、延迟与计费 | [LexBench Web Search](https://github.com/lexmount/lexbench-websearch)：场景问题，以及用于检索评测的 SimpleQA Verified 改编任务 | [本地示例](../examples/web_search/README.md) · [完整指南](../harness/web_search/README.md) | | 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/tasks.v4.json)，改编自 [Lexbench-Headless-Browser 场景设计](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios)；[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json)，由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)，使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | @@ -65,7 +65,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run 当前软件包版本是 `0.2.0`。软件标签采用 `v主版本.次版本.补丁版本`，评测集和托管报告各有独立版本。报告记录这三种身份，因此重建报告不会改换题目版本或保存的调用。 -兼容规则、发布检查和首个软件发行版的范围见[发布指南](../docs/releases.md)。Web Search 在接入主分支前保持预览状态。稳定版承诺的是文档中的命令、输入约定和历史证据兼容性；浏览器与在线网站的逐题表现仍由评测结果说明。 +兼容规则、发布检查和首个软件发行版的范围见[发布指南](../docs/releases.md)。Web Search 已提供预览功能，稳定版支持范围由发布检查确认。稳定版承诺的是文档中的命令、输入约定和历史证据兼容性；浏览器与在线网站的逐题表现仍由评测结果说明。 ## 相关仓库 diff --git a/docs/dependencies.md b/docs/dependencies.md index 749a0a554..459ef14b0 100644 --- a/docs/dependencies.md +++ b/docs/dependencies.md @@ -15,11 +15,10 @@ Use the declared Git source for the evaluator. A registry package with the same | Dependency | Role | Version authority | | --- | --- | --- | | [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Task model, catalogs, deterministic acceptance scoring and frozen input validation | Immutable Git revision in `pyproject.toml` and `uv.lock`; [benchmark integration](benchmarks.md) | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Query definitions, provenance, source qualifications and retrieval scoring | Immutable Git revision in `pyproject.toml` and `uv.lock`; [Search guide](../harness/web_search/README.md) | | [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) | HTML parsing | `pyproject.toml` and `uv.lock` | | [FastCDC](https://github.com/iscc/fastcdc-py) | Content chunking | `pyproject.toml` and `uv.lock` | -Search development uses [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) for query definitions and retrieval scoring. Its integration is tracked in [PR #58](https://github.com/lexmount/browser-eval/pull/58); it is not a dependency of the current main-branch package. - ## Git-based runtime dependencies | Dependency | Repository and relevant branch | Required source | diff --git a/docs/releases.md b/docs/releases.md index e289e40e4..7c4ff1104 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -14,7 +14,7 @@ Follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). The public co While the software remains `0.x`, use a minor release for new functionality or breaking changes and a patch for compatible fixes. Describe breaking changes explicitly. Release when users need an installable, reproducible checkpoint; individual commits and report rebuilds do not need new numbers. Version changes and release tags require an explicit human release request. -`1.0.0` establishes the compatibility contract above. It can cover stable tracks while others remain explicitly experimental. For the first stable Web Fetch + Web Search release, integrate Search into main, run its offline collection/scoring/report tests and browser evidence workflows, and pass the existing package and CI gates on the release commit. Publish the supported track and dataset matrix in its release notes. A successful historical report alone does not test the installable release. +`1.0.0` establishes the compatibility contract above. It can cover stable tracks while others remain explicitly experimental. For the first stable Web Fetch + Web Search release, run the integrated offline collection/scoring/report tests and browser evidence workflows, and pass the existing package and CI gates on the release commit. Publish the supported track and dataset matrix in its release notes. A successful historical report alone does not test the installable release. ## Software, benchmarks and reports From 6d4252cbae4df6020aaaaa63493c911213e86b61 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Mon, 26 Oct 2026 21:00:49 +0800 Subject: [PATCH 10/10] ci: run automated checks only on Python 3.11 --- .github/workflows/README.md | 2 +- .github/workflows/ci.yml | 19 ------------------- 2 files changed, 1 insertion(+), 20 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index f7521a1f8..b2a62dac1 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,6 +1,6 @@ # GitHub workflows -`ci.yml` tests main pushes and pull requests. The base CLI runs on Python 3.12–3.14; full development checks use Python 3.11, uv 0.7.8 and Node.js 22. Contributors use `make setup-test` and `make ci` locally. +`ci.yml` tests main pushes and pull requests. All CI checks use Python 3.11, uv 0.7.8 and Node.js 22. Contributors use `make setup-test` and `make ci` locally. Code review is dispatched by the organization's [public OCR workflow](https://github.com/lexmount/github-policy-public/blob/main/.github/workflows/codex-code-review.yml). The dispatch job confirms durable task acceptance; the separate `ocr-review` status reports the backend's findings. The repository does not need a copied OCR workflow or local review credentials. Organization Rulesets manage review requirements separately from CI. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9f2220238..635e33cc1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,22 +42,3 @@ jobs: uv run --frozen --no-sync playwright install chromium export BROWSER_EVAL_TEST_BROWSER_BINARY="$(uv run --frozen --no-sync python -c 'from playwright.sync_api import sync_playwright; p = sync_playwright().start(); print(p.chromium.executable_path); p.stop()')" uv run --frozen --no-sync python -m pytest tests/test_browser_observation_contracts.py tests/test_report_browser_interactions.py tests/web_search/test_visual.py -q - - base-python: - name: Base CLI / Python ${{ matrix.python }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - python: ["3.12", "3.13", "3.14"] - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python }} - - uses: astral-sh/setup-uv@v6 - with: - version: "0.7.8" - - run: uv sync --frozen --extra test --python ${{ matrix.python }} - - run: uv run --frozen --no-sync --python ${{ matrix.python }} python -m harness.web_fetch validate --providers fixture - - run: uv run --frozen --no-sync --python ${{ matrix.python }} python -m pytest tests/test_portable_cli.py tests/test_cli_scope.py tests/test_web_fetch_quickstart.py tests/test_public_report_language.py -q