diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 11ae445..82d4724 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,13 +1,17 @@ name: ci -on: [push, pull_request] +on: + push: + branches: [main] + pull_request: permissions: contents: read jobs: check: runs-on: ubuntu-latest strategy: + fail-fast: false matrix: - python: ["3.11", "3.12", "3.13", "3.14"] + python: ["3.11"] steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 @@ -17,17 +21,5 @@ jobs: with: version: "0.7.8" - run: uv sync --frozen --extra test --python "$(command -v python)" - - run: uv run --frozen --extra test python -m pytest tests -q - - run: uv run --frozen lexbench-webfetch validate - - run: uv build --sdist - - run: uv build dist/*.tar.gz --wheel - - name: Validate an isolated wheel installation - run: | - uv venv /tmp/lexbench-wheel --python "$(command -v python)" - for lexbench_wheel in dist/*.whl; do - uv pip install --python /tmp/lexbench-wheel/bin/python "${lexbench_wheel}[test]" - done - cp -R tests /tmp/lexbench-wheel-tests - cd /tmp - /tmp/lexbench-wheel/bin/lexbench-webfetch validate - uv run --no-project --no-sync --python /tmp/lexbench-wheel/bin/python python -m pytest /tmp/lexbench-wheel-tests -q + - run: make check BENCH_PYTHON="$(command -v python)" + - run: make package-check BENCH_PYTHON="$(command -v python)" diff --git a/Makefile b/Makefile index 99bc9dd..be9ebec 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,24 @@ -.PHONY: check -check: - uv run --frozen lexbench-webfetch validate - uv run --frozen --extra test python -m pytest tests -q +UV ?= uv +BENCH_PYTHON ?= $(shell cat .python-version) +PY := $(UV) run --frozen --extra test --python "$(BENCH_PYTHON)" python + +.PHONY: check test package-check +check: test + $(PY) -m lexbench_webfetch.cli validate + +test: + $(PY) -m pytest tests -q + +package-check: + @set -eu; \ + dist="$$(mktemp -d)"; run="$$(mktemp -d)"; \ + trap 'rm -rf "$$dist" "$$run"' EXIT; \ + $(UV) build --sdist --python "$(BENCH_PYTHON)" --out-dir "$$dist"; \ + for source in "$$dist"/*.tar.gz; do $(UV) build "$$source" --wheel --python "$(BENCH_PYTHON)" --out-dir "$$dist"; done; \ + $(UV) venv --python "$(BENCH_PYTHON)" "$$run/venv"; \ + set -- "$$dist"/lexbench_webfetch-*.whl; \ + $(UV) pip install --python "$$run/venv/bin/python" "$${1}[test]"; \ + cp -R tests "$$run/tests"; \ + cd "$$run"; \ + $(UV) run --no-project --no-sync --python "$$run/venv/bin/python" python -m pytest tests -q; \ + "$$run/venv/bin/lexbench-webfetch" validate diff --git a/README.md b/README.md index 6fd5894..4b8bb6a 100644 --- a/README.md +++ b/README.md @@ -8,11 +8,13 @@ [![License: Apache 2.0](https://img.shields.io/badge/code-Apache--2.0-51735b)](LICENSE) [![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](pyproject.toml) -[Tasks](#task-suites) · [Quickstart](#quickstart) · [Scoring](#score-an-answer) · [Contributing](CONTRIBUTING.md) +[Tasks](#task-suites) · [Quickstart](#quickstart) · [Scoring](#score-an-answer) · [Versions](#versions) · [Related projects](#related-repositories) · [Contributing](CONTRIBUTING.md) -LexBench Web Fetch provides tasks and scoring tools for web retrieval: page text, records with usable links, resources and files. Use it with your own runner or [Browser Eval](https://github.com/lexmount/browser-eval). +LexBench Web Fetch defines 550 Web Fetch tasks and scores returned page text, records with usable links, resources and files. Use the tasks with your own runner or [Browser Eval](https://github.com/lexmount/browser-eval) to compare browsers, extraction tools and hosted services. + +This repository owns the task definitions, acceptance rules and frozen input bindings. Browser Eval owns provider calls, timing, evidence and comparison reports. ## Task suites @@ -61,13 +63,28 @@ uv run --frozen lexbench-webfetch score --task examples/task.json --response exa For real evaluations, use the complete task from a verified dataset. Add `--metadata metadata.json` for structured response fields, or `--file /absolute/path/file` for delivered files. Scoring requirements and result reporting are described in the [evaluation protocol](docs/protocol.md). -## Versions and execution +## Run browsers and build reports + +Follow the [Browser Eval integration guide](https://github.com/lexmount/browser-eval/blob/main/docs/benchmarks.md) to run exported tasks against browsers or services and build comparison reports. Provider accounts, API keys and measured outputs belong to the execution environment. The commands above validate and score saved inputs without calling providers. + +## Versions + +The current full catalog is benchmark edition `2.0.1` (`bench-v2.0.1`), with 550 entries. The Python tooling package is `0.3.0`. These versions identify different things: the edition fixes tasks, references and scoring; the package versions the commands and Python implementation. + +Run `uv run --frozen lexbench-webfetch version` to verify the installed catalog. Use `release` to freeze the exact selected tasks and scorer in `benchmark-release.json`, and `compare-releases` to check whether results can be compared directly, need rescoring or belong to separate populations. Installing a newer package leaves an existing report's edition unchanged. See [versioning](docs/versions.md) for commands and change rules. -Run `uv run --frozen lexbench-webfetch version` to inspect the installed catalog. Tags use `bench-vMAJOR.MINOR.PATCH`. Each report freezes the selected tasks and scoring source in `benchmark-release.json`; a package upgrade leaves that selection unchanged. +## Related repositories -Use `release` to freeze an edition and `compare-releases` to check comparability. See [versioning](docs/versions.md) for commands and major/minor/patch rules. +| Repository | Role | +| --- | --- | +| [Browser Eval](https://github.com/lexmount/browser-eval) | Browser/provider execution, saved evidence and comparison reports | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Search queries, source references and retrieval scoring | +| [Moli](https://github.com/lexmount/moli) | Browser that can be evaluated with these tasks | +| [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service | +| [Artifact Site](https://github.com/lexmount/artifact-site) | Hosting, versioning and sharing of generated reports | +| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Protocol, automation-driver and web-platform benchmarks | -To execute browser or service calls and build reports, follow the [Browser Eval integration guide](https://github.com/lexmount/browser-eval/blob/main/docs/benchmarks.md). +[Third-party notices](THIRD_PARTY.md) list scoring dependencies and source rights. WebMainBench, WCXB and WebArena are separate benchmarks; they are not part of this 550-entry catalog. ## Contribute @@ -77,7 +94,7 @@ Read [CONTRIBUTING](CONTRIBUTING.md) before changing task requirements or scorin ## Cite LexBench Web Fetch -Use this citation or [CITATION.cff](CITATION.cff), and record the benchmark edition and suites used. +Use this citation or [CITATION.cff](CITATION.cff), and record the benchmark edition, suites and code revision used. The citation below identifies the tooling package; the catalog edition is `2.0.1`. ```bibtex @software{lexbench_webfetch_2026, diff --git a/THIRD_PARTY.md b/THIRD_PARTY.md index e3156af..8738a6e 100644 --- a/THIRD_PARTY.md +++ b/THIRD_PARTY.md @@ -5,3 +5,16 @@ The Apache-2.0 license covers project-owned code, documentation and task require Original website text, document files, media, screenshots and frozen reference captures retain their source terms and are not included here. Public accessibility and a source URL do not grant redistribution permission. Task definitions retain source URLs and compact acceptance requirements; full historical reference inputs must be obtained separately under an appropriate authorization basis. WebMainBench, WCXB and WebArena are separate projects and are not redistributed as LexBench Web Fetch suites. Cite their owners and follow their terms when using them through an execution harness. + +## Scoring dependencies + +Exact versions and resolved hashes are in `pyproject.toml` and `uv.lock`. + +| Library | Use | License | +| --- | --- | --- | +| [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) | HTML parsing | MIT | +| [Soup Sieve](https://github.com/facelessuser/soupsieve) | CSS selectors | MIT | +| [APTED](https://github.com/JoaoFelipe/apted) | Optional tree-edit scoring | MIT | +| [RapidFuzz](https://github.com/rapidfuzz/RapidFuzz) | Optional text matching | MIT | + +These libraries retain their own licenses. Installing the `tree` extra enables the optional scoring dependencies. diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index 57a7178..afbb4e0 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -8,11 +8,13 @@ [![License: Apache 2.0](https://img.shields.io/badge/code-Apache--2.0-51735b)](../LICENSE) [![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](../pyproject.toml) -[任务集](#任务集) · [快速开始](#快速开始) · [评分](#给响应评分) · [参与贡献](../CONTRIBUTING.md) +[任务集](#任务集) · [快速开始](#快速开始) · [评分](#给响应评分) · [版本](#版本) · [相关仓库](#相关仓库) · [参与贡献](../CONTRIBUTING.md) -LexBench Web Fetch 提供网页获取任务和评分工具，评测正文、带有效链接的记录、资源和文件。可以使用自己的运行器，也可以接入 [Browser Eval](https://github.com/lexmount/browser-eval)。 +LexBench Web Fetch 定义了 550 条网页获取任务，评测返回的正文、带有效链接的记录、资源和文件。可以用自己的运行器，也可以接入 [Browser Eval](https://github.com/lexmount/browser-eval)，比较浏览器、提取工具和在线服务。 + +本仓库维护题目、验收规则和冻结输入绑定。Browser Eval 负责服务调用、计时、证据保存和对比报告。 ## 任务集 @@ -61,13 +63,28 @@ uv run --frozen lexbench-webfetch score --task examples/task.json --response exa 实际评测应使用经过验证的数据集中的完整任务。结构化响应字段通过 `--metadata metadata.json` 传入，交付文件通过 `--file /absolute/path/file` 传入。评分要求和结果发布规则见[评测协议](../docs/protocol.md)。 -## 版本与运行 +## 运行浏览器并生成报告 + +按 [Browser Eval 接入指南](https://github.com/lexmount/browser-eval/blob/main/docs/benchmarks.md)执行导出的任务，调用浏览器或在线服务，生成对比报告。服务账号、API 密钥和实测输出由运行环境管理。上面的命令只检查输入、给已保存的响应评分，不调用服务。 + +## 版本 + +当前完整题库的评测版本为 `2.0.1`（`bench-v2.0.1`），共 550 条题目；Python 工具包版本为 `0.3.0`。评测版本固定题目、参考输入和评分规则，工具包版本标识命令与 Python 实现。 + +运行 `uv run --frozen lexbench-webfetch version` 验证安装的题库。用 `release` 将实际选用的题目和评分器固定在 `benchmark-release.json` 中，用 `compare-releases` 判断结果能直接比较、需要重算评分，还是属于不同题目范围。升级包不会替换已有报告的评测版本。命令和版本变更规则见[版本说明](../docs/versions.md)。 -运行 `uv run --frozen lexbench-webfetch version` 查看安装的题库版本。标签格式为 `bench-v主版本.次版本.补丁版本`。每份报告通过 `benchmark-release.json` 固定所选题目和评分源码，升级包不会替换这份选择。 +## 相关仓库 -用 `release` 固定评测版本，用 `compare-releases` 检查结果是否可比。命令和主版本、次版本、补丁版本规则见[版本说明](../docs/versions.md)。 +| 仓库 | 用途 | +| --- | --- | +| [Browser Eval](https://github.com/lexmount/browser-eval) | 运行浏览器与服务、保存证据、生成对比报告 | +| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 | +| [Moli](https://github.com/lexmount/moli) | 可使用本题库评测的浏览器 | +| [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 | +| [Artifact Site](https://github.com/lexmount/artifact-site) | 报告托管、版本管理和分享 | +| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 协议、自动化驱动和 Web 平台评测 | -执行浏览器或服务调用并生成报告，请按 [Browser Eval 接入指南](https://github.com/lexmount/browser-eval/blob/main/docs/benchmarks.md)操作。 +评分依赖和来源权利见[第三方说明](../THIRD_PARTY.md)。WebMainBench、WCXB 和 WebArena 是独立评测集，不计入这里的 550 条题目。 ## 参与贡献 @@ -77,7 +94,7 @@ uv run --frozen lexbench-webfetch score --task examples/task.json --response exa ## 引用 LexBench Web Fetch -使用以下引用或 [CITATION.cff](../CITATION.cff)，并注明评测版本和任务集。 +使用以下引用或 [CITATION.cff](../CITATION.cff)，并注明评测版本、任务集和代码版本。以下引用标识工具包版本，题库评测版本为 `2.0.1`。 ```bibtex @software{lexbench_webfetch_2026, diff --git a/docs/versions.md b/docs/versions.md index 01ccb6e..43c9d28 100644 --- a/docs/versions.md +++ b/docs/versions.md @@ -30,6 +30,15 @@ population and request differences; maintainers still review whether a rule change preserves the intended task. Existing annotation revisions retain their independent source evidence and executable correction controls. +## Package versions and release approval + +Package versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html): compatible fixes use a patch, compatible CLI or Python API additions use a minor, and incompatible public interfaces use a major. While a package remains `0.x`, use a minor for new functionality or breaking changes and a patch for compatible fixes. Document migrations explicitly. + +The package version, benchmark edition and hosted report version are independent. Web Fetch benchmark tags use `bench-vMAJOR.MINOR.PATCH`; Web Search benchmark tags use `bench-MAJOR.MINOR.PATCH`. Both repositories use `vMAJOR.MINOR.PATCH` for tooling-only software releases. Browser Eval uses software tags, since it owns execution and reporting rather than a benchmark catalog. A change to task population can require a major benchmark edition while its software support remains backward compatible. Each repository keeps its own version number and release schedule. + +Version changes and tags require an explicit human release request. Documentation edits, routine fixes, synchronization and report rebuilds do not automatically bump versions or rewrite a frozen catalog. Release from a reviewed, merged commit after code, catalog and installed-package checks pass. Preserve all existing tags and exact bindings; a readable tag is not a substitute for input or scorer hashes. + + ## Freeze an edition `lexbench-webfetch version` verifies the installed catalog's complete suite list,