[assassin808]

What

The Bench part of the v0.8 platform (B1–B3). A v0.8 study is a folder with study.json and protocol.json, read by one engine, hsenv. Bench publishes a released protocol as a community revision and checks it with the pinned engine. Bench never interprets or executes protocol files itself.

• B1, pin the engine. requirements.txt pins hsenv v0.8.0 by commit SHA (fd38302574260b5876cfeaa6741540c5163ba662). This is the only place the pin is written: the sync-studies-index, research-catalog, maintain-community and validate-studies workflows read that line from it. scripts/research/effects_core.py now re-exports hsenv.effects_core, so Bench no longer keeps its own copy.

• B2, validate protocol releases. scripts/research/protocols.py checks a release:

  • schema
  • public-path allowlist
  • visibility: public and a signed HIGH FIDELITY review
  • no release_hold
  • protocol hash binding and a current checks report
  • effects and capabilities
  • manifest digest

  validate_assets sends protocol revisions through these checks. validate-studies.yml gains a validate-protocols job. scripts/check_contributors.py is one contributor rule for legacy index.json and protocol study.json.

• B3, research_index. build_catalog adds a protocol block to the paper entry: counts and ids only, never protocol text. replayable and humanBenchmark are computed from the protocol. studies_index.json and research_index.json stay byte-identical for current studies. tests/fixtures/protocol_release/research-index-protocol.json is the shared contract fixture for the web reader.

Details are in docs/v080-protocol.md.

Why

Bench, the web runner and the desktop must judge a study with the same engine. Pinning hsenv by SHA means one reviewed engine version decides every verdict, and nothing in a contribution is ever run.

Protocol proposals need a maintainer

On the community proposal path (maintain-community.yml → revisions.integrate), a proposal that holds protocol.json is never accepted. It always returns needs_review with requiresMaintainer: true. This holds whatever the review says, including an LLM agent's "accept". It also holds whatever the release says about itself (fidelity.signed_by, a PASS review/checks-latest.json). No agent review is spent on these proposals. A protocol release is published only through a maintainer release PR, and the validate-protocols job checks that PR.

Tests

• Fresh venv (Python 3.13), pip install -r requirements.txt, which installs hsenv from GitHub at the pinned SHA. Then pytest tests -o addopts='': 250 passed, 1 failed.
• The one failure is tests/test_genius_integration.py::test_hosted_live_contract_preserves_context_and_uses_openrouter. It fails the same way on main under Python 3.13, and passes under 3.12. The test compares id() of backend objects, and 3.13 reuses ids after garbage collection. This PR does not touch it.
• The shared fixture's checks-latest.json and the other fixtures pass unchanged with the tagged engine.
• New and changed tests:
  • tests/test_hsenv_pin.py: the pin is a 40-hex SHA and the only one; workflows read it from requirements.txt; the installed version matches the tag. The effects golden covers all 42 community revisions with effects.json and matches the pre-hsenv Bench copy exactly. Protocol readers never call eval/exec outside the whitelisted hsenv.expr.
  • tests/test_protocol_release.py: release rules, the maintainer-only rule for proposals, and the protocol block in research_index.
  • tests/test_check_contributors.py.
• Test data is an invented CC0 fixture (example2026lamps), not a real paper.

🤖 Generated with Claude Code