Repository navigation
fix: redesign benchmark integrity foundation - #2
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SKILL.mdVerification
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -v— PASS, 27 tests in 34.197spython3 -B scripts/verify_no_copied_skills.py— PASS, 4 manifests and 32 unique skillspython3 -B scripts/run_benchmark.py --help— PASS from committed scorer blobgit diff --check— PASS before commitpython3 -B scripts/aggregate_stability.py— expected exit 1 with the retired-window noticeKnown merge blocker
Do not merge this draft yet. Independent review reproduced a remaining publication race in
scripts/benchmark_scorer.py::atomic_publish: another directory can replace the just-created public bundle betweenbundle.mkdir()and the firstbundle.lstat(). The replacement can then be accepted as the reservation, overwritten, and markedCOMPLETE.A fresh bounded follow-up should replace pathname reservation with complete staging plus an atomic no-replace directory rename (
renamex_np(..., RENAME_EXCL)on macOS andrenameat2(..., RENAME_NOREPLACE)on Linux), fail closed elsewhere, add creation-to-publish race regressions, and receive a fresh independent test-critic review.CI status
GitHub's
validatejob currently fails in the unchangedpython3 scripts/refresh_external_skills.py --checkstep because upstream pins have moved for 39 external skill entries and many configured raw-content paths now return HTTP 404. This branch does not change the external-skill manifests. The focused unit, manifest-structure, and committed-bootstrap checks above pass locally; the existing external-refresh drift remains separate from the publication-race blocker.