Skip to content

fix: redesign benchmark integrity foundation - #2

Merged
AlexU-A merged 2 commits into
mainfrom
codex/benchmark-output-integrity
Sep 19, 2026
Merged

AlexU-A merged 2 commits into
mainfrom
codex/benchmark-output-integrity

Conversation

@AlexU-A

@AlexU-A AlexU-A commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • migrate all 24 React, Next.js, and React Native fixtures to strict schema v2 with explicit public context and private adjudication
  • add five intentional-clean controls and correct unsupported framework gold claims
  • expose complete public rule definitions while limiting the candidate treatment to inline committed SKILL.md
  • execute the scorer from exact committed bytes and strengthen capture, schema, metric, and publication-substitution tests
  • retain descriptive-only boundaries: this includes no provider captures, performance result, full-skill claim, or statistical-superiority claim

Verification

  • PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -v — PASS, 27 tests in 34.197s
  • python3 -B scripts/verify_no_copied_skills.py — PASS, 4 manifests and 32 unique skills
  • python3 -B scripts/run_benchmark.py --help — PASS from committed scorer blob
  • git diff --check — PASS before commit
  • python3 -B scripts/aggregate_stability.py — expected exit 1 with the retired-window notice

Known merge blocker

Do not merge this draft yet. Independent review reproduced a remaining publication race in scripts/benchmark_scorer.py::atomic_publish: another directory can replace the just-created public bundle between bundle.mkdir() and the first bundle.lstat(). The replacement can then be accepted as the reservation, overwritten, and marked COMPLETE.

A fresh bounded follow-up should replace pathname reservation with complete staging plus an atomic no-replace directory rename (renamex_np(..., RENAME_EXCL) on macOS and renameat2(..., RENAME_NOREPLACE) on Linux), fail closed elsewhere, add creation-to-publish race regressions, and receive a fresh independent test-critic review.

CI status

GitHub's validate job currently fails in the unchanged python3 scripts/refresh_external_skills.py --check step because upstream pins have moved for 39 external skill entries and many configured raw-content paths now return HTTP 404. This branch does not change the external-skill manifests. The focused unit, manifest-structure, and committed-bootstrap checks above pass locally; the existing external-refresh drift remains separate from the publication-race blocker.

@AlexU-A
AlexU-A marked this pull request as ready for review September 19, 2026 15:36
@AlexU-A
AlexU-A merged commit aa54119 into main Sep 19, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant