Repository navigation
docs: clarify Browser Eval tracks and release boundaries - #67
Conversation
cf454ae to
32b0314
Compare
lanyue-llk
changed the title
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). |
![lexmount-ocr[bot]](https://avatars.githubusercontent.com/u/277637559?s=60&v=4){kind=small}
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Reviewed commit 32b0314.
The PR updates benchmark discovery and extends the existing updater to select a package within a shared Git repository. Keeping source selection in that updater is the right scope; no report-instance-specific branch or duplicated runner/renderer was added.
Two updater issues need correction:
- [P2] The new inline finding below: once Fetch and Search share a repository and revision, a Fetch update fails because its URL is also a prefix of the Search dependency URL. This was reproduced with a real local Git repository, uv lock/sync and the ancestry check, without mocking.
- [P2] The existing empty SUBDIRECTORY finding is valid. make -n confirms that SUBDIRECTORY= drops the argument, so the public Make target cannot clear a previously selected package directory. Distinguish an undefined variable from a defined empty value and cover subdirectory-to-root migration.
Validation: dependency updates, README navigation, dependency boundaries, report software identities and the Search suite passed: 529 tests, 1 skipped and 45 subtests, including the browser-dependent cases. The shared-repository fixture installed both packages successfully before the updater rejected the valid state. Add behavior tests for independent updates with both packages pinned to one commit, plus the directory-clearing transition. No additional actionable regression or report-instance coupling was confirmed.
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Re-reviewed 6b0d892. No additional actionable finding.
Both earlier updater defects are fixed. Real Git/uv integration with the actual family packages now updates Fetch and Search independently when both start at the same repository and commit; the sibling pin and binding stay intact. The public Make command also clears SUBDIRECTORY and installs a root package successfully.
Dependency migration, README navigation, source boundaries, software-version recording and Search regression checks passed: 532 tests, 1 skipped and 45 subtests, including browser-dependent scenarios. GitHub CI and the current OCR status passed. The workflow removal leaves organization OCR as the documented review owner. The change continues to reuse the existing updater and introduces no report-instance coupling. The companion family PR still has the separately reported quantity-prefix scoring regression; consumer pin migration remains a separate change after that PR lands.
lanyue-llk
changed the title
There was a problem hiding this comment.
✅ Re-review complete · 66ac994
No findings in this incremental review.
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
Reviewed the complete current diff at 66ac994. One P2 documentation defect is described inline: Search is already included in the target main branch, so the new development-only description and links are stale.
The PR now retains independent Fetch/Search repositories and makes no runtime or dependency-updater changes relative to its base. No report-instance coupling was introduced. Relevant dependency, README, report-version and Search checks passed locally: 522 tests, 1 skipped and 45 subtests, including browser interactions. Current CI and OCR statuses are successful.
There was a problem hiding this comment.
ℹ️ Review skipped · 4125e83
No reviewable content was selected for this run.
There was a problem hiding this comment.
✅ Re-review complete · 4125e83
No findings in this incremental review.
There was a problem hiding this comment.
✅ Re-review complete · 6d4252c
No findings in this incremental review.
Summary
Describe Browser Eval as the execution and reporting harness, map each evaluation track to its benchmark authorities and restore related-project links in both READMEs. Attribute the curated replay scenario designs to Lexbench-Headless-Browser while distinguishing adapted public-page trajectories from the full upstream population.
Keep LexBench Web Fetch and LexBench Web Search as independent repositories and packages. Existing immutable consumer pins and historical evidence are unchanged. Document software releases separately from benchmark editions and report builds, and clarify contributor copyright under Apache 2.0. No version bump or tag is included.
Remove the obsolete repository-local Claude workflow and its setup instructions. Organization OCR remains separate.
Python CI runs only on 3.11. Existing test and package-validation steps remain unchanged.
Validation