Skip to content

fix(ci): verify the published pages instead of pushing them to main #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bkd-dotcom merged 4 commits into from
Sep 9, 2026
Merged

fix(ci): verify the published pages instead of pushing them to main #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
55a7af2
fix(ci): run the publish gates even when the push fails
bkd-dotcom Sep 9, 2026
63a670e
docs: refresh the leaderboard for signetry-core 0.8.0
bkd-dotcom Sep 9, 2026
2e4319b
fix(ci): verify the published pages instead of pushing them to main
bkd-dotcom Sep 9, 2026
75fd873
fix(ci): check out the PR, and refuse a tree without the recipe
bkd-dotcom Sep 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 80 additions & 38 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,25 @@
name: benchmark

# Regenerates docs/BENCHMARK.md — the detection head-to-head against the other
# scanners — and publishes it as a release artifact, so the published numbers always
# match the shipped engine.
# Checks that docs/BENCHMARK.{md,json} — the detection head-to-head against the other
# scanners — is current.
#
# This workflow owns docs/BENCHMARK.{md,json} and nothing else; the governance
# leaderboard is owned by leaderboard.yml. Both push to main, so both sit in the
# `published-numbers` concurrency group.
# This workflow does not publish. It regenerates the table with `make benchmark` (the same
# recipe a contributor runs) and fails if the committed table differs. Refreshing is a
# pull request.
#
# It used to push the refreshed table straight to `main`, which branch protection rejects:
# every run that actually had a change to publish failed at the push, and the failure came
# before the parity gate below, so a detection regression would have read as a git error.
# Runs only looked green because the table happened to be unchanged and the push step took
# its "no change to publish" exit. Verifying instead of pushing removes both failure modes.
#
# It runs on a schedule as well as on releases: the detection corpus and the competing
# scanners both change between tags, and a table only refreshed at release time can be
# months out of date while still reading as current.
#
# This workflow owns docs/BENCHMARK.{md,json} and nothing else; the governance leaderboard
# is owned by leaderboard.yml. Nothing is pushed by either, so they no longer share a
# concurrency group.
on:
schedule:
# Mondays 06:47 UTC. Off the hour on purpose (scheduled jobs queued at :00 are
Expand All @@ -23,66 +32,99 @@ on:
paths:
- "signetry_eval/corpus/**"
- "signetry_eval/detection.py"
# The table is produced by the pinned signetry-core's scanner, so the pin is an input
# to it: bumping it without refreshing is exactly the staleness caught here.
- "pyproject.toml"
- "Makefile"
# Verify on the pull request too, so a change to any input above cannot merge while
# leaving the committed table stale. Only safe to run on PRs because the token below is
# read-only and this job needs no secrets.
pull_request:
paths:
- "signetry_eval/corpus/**"
- "signetry_eval/detection.py"
- "pyproject.toml"
- "Makefile"
workflow_dispatch:

# Read-only: nothing here writes to the repository any more.
permissions:
contents: write
contents: read

concurrency:
group: published-numbers
cancel-in-progress: false
# Only stops a superseded commit's verification from running alongside the commit that
# replaced it — there is no longer a push to serialise against leaderboard.yml.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
# Never the incoming ref: on a release event that is the tag, which
# leaves us in detached HEAD and turns the push below into a tag write.
ref: ${{ github.event.repository.default_branch || 'main' }}
# Verify the ref that triggered us. On a pull request that is `refs/pull/N/merge`
# — the merge result, which is what actually lands — because verifying the default
# branch would say nothing about the change under review. The one exception is a
# release: its ref is the tag, which is not where the table is committed.
#
# The *release* case is the true branch on purpose. The obvious spelling,
# `event_name == 'pull_request' && '' || default_branch`, is a trap: Actions' `&&`
# and `||` yield operand values rather than booleans, so a falsy true branch ('')
# collapses to the false branch and *every* event checks out the default branch.
# This workflow shipped that bug for one run: PR #39 verified `main` instead of
# itself. Both branches here are non-empty strings.
ref: ${{ github.event_name == 'release' && (github.event.repository.default_branch || 'main') || github.ref }}
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Install
run: pip install -e ".[dev]"
- name: Require the signetry-core detection engine

- name: Regenerate the head-to-head table
id: regen
run: |
# This benchmark has no meaning without the SAST engine, and an absent scanner
# must never be published as a zero. Fail rather than emit a table.
python -c "from signetry_core import scan_repository" || {
echo "::error::Installed signetry-core lacks scan_repository; pin a signetry-core with the detection engine."
# `make benchmark` is the single recipe, shared with contributors: the workflow
# no longer inlines the commands, so the generated header and the command that
# actually writes the file cannot drift apart.
#
# The recipe refuses to run without the signetry-core detection engine. That
# matters more here than anywhere else: this benchmark has no meaning without the
# scanner, and an absent scanner must never be published as a zero.
# The checked-out tree must be the one that carries the recipe. `make benchmark`
# without a Makefile does not necessarily fail: `benchmark` can name a real path in
# this repository, in which case make treats the target as already up to date and
# exits 0 — "Nothing to be done for 'benchmark'". A regeneration step that silently
# regenerates nothing is the same class of bug as the push step that silently
# exited 0 on "no change to publish", so check before trusting it.
test -f Makefile || {
echo "::error::No Makefile in the checked-out tree — regeneration would silently do nothing. Wrong ref?"
exit 1
}
- name: Regenerate the head-to-head table
run: |
mkdir -p docs
{
echo "<!-- Generated by .github/workflows/benchmark.yml — do not edit by hand. -->"
echo ""
signetry-eval corpus --markdown
} > docs/BENCHMARK.md
signetry-eval corpus --json > docs/BENCHMARK.json
make benchmark

- uses: actions/upload-artifact@v7
with:
name: detection-benchmark
path: |
docs/BENCHMARK.md
docs/BENCHMARK.json
if-no-files-found: error
- name: Commit the refreshed benchmark

- name: Are the committed numbers current?
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add docs/BENCHMARK.md docs/BENCHMARK.json
git diff --cached --quiet && { echo "no change to publish"; exit 0; }
git commit -m "docs: refresh the detection head-to-head benchmark"
# leaderboard.yml pushes to the same branch. The concurrency group serializes
# us; rebase anyway rather than force or swallow the failure.
git pull --rebase --autostash origin "${{ github.event.repository.default_branch || 'main' }}"
git push origin "HEAD:${{ github.event.repository.default_branch || 'main' }}"
# Diff-only: the regeneration above already wrote into the working tree.
make check-benchmark || {
echo "::error::docs/BENCHMARK.{md,json} are stale. Run 'make benchmark', commit the result, and open a pull request."
exit 1
}

- name: Gate on parity (Signetry must keep 100% recall, 0 false positives)
# Must run even when the staleness check failed — see the same guard in
# leaderboard.yml. Without it a stale table aborts the job before the regression is
# judged, and a real detection regression would surface as a docs error.
if: ${{ !cancelled() && steps.regen.outcome == 'success' }}
run: |
# Last, deliberately: the regression is published above whatever this says,
# then this step fails the run so it cannot be missed.
# Last, deliberately: staleness is reported above whatever this says, then this
# step fails the run so a regression cannot be missed.
signetry-eval corpus --min-recall 1.0 --max-fp 0
145 changes: 84 additions & 61 deletions .github/workflows/leaderboard.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,26 @@
name: leaderboard

# Regenerates docs/LEADERBOARD.md — the two-axis governance leaderboard, including
# third-party submissions from leaderboard/entries/.
# Checks that docs/LEADERBOARD.{md,json} — the two-axis governance leaderboard, including
# third-party submissions from leaderboard/entries/ — is current.
#
# This workflow does not publish. It regenerates the page with `make leaderboard` (the
# same recipe a contributor runs and the one printed on the page itself) and fails if the
# committed page differs. Refreshing is a pull request.
#
# It used to push the refreshed page straight to `main`. `main` requires a pull request,
# so every run that actually had a change to publish was rejected by branch protection —
# the page went stale rather than current, and because the push failed before the gate
# below, a broken defense would have read as a git error. Verifying instead of pushing
# removes both failure modes and keeps the numbers arriving the way every other change
# does, which is the argument this repository exists to make.
#
# Why a schedule and not just releases: the governance axis takes submissions, so the
# page changes when someone else's PR lands, not only when we cut a tag. A leaderboard
# whose numbers are as old as the last release is a changelog.
#
# This workflow owns docs/LEADERBOARD.{md,json} and nothing else. The detection
# head-to-head (docs/BENCHMARK.md) is owned by benchmark.yml — one generator per file,
# so every "Generated by ..." header names the workflow that actually wrote it. Both
# share the `published-numbers` concurrency group because both push to main.
# head-to-head (docs/BENCHMARK.md) is owned by benchmark.yml — one generator per file, so
# every "Generated by ..." header names the recipe that actually wrote it.
on:
schedule:
# Mondays 07:23 UTC. Off the hour on purpose: scheduled jobs queued at :00 are
Expand All @@ -25,70 +35,81 @@ on:
- "signetry_eval/leaderboard.py"
- "signetry_eval/report.py"
- "signetry_eval/scenarios/**"
# The live row is labelled with the pinned signetry-core, so the pin is an input to
# the page: bumping it without refreshing is exactly the staleness caught here.
- "pyproject.toml"
- "Makefile"
# Verify on the pull request too, so a change to any input above cannot merge while
# leaving the committed page stale. Only safe to run on PRs because the token below is
# read-only and this job needs no secrets.
pull_request:
paths:
- "leaderboard/entries/**"
- "signetry_eval/leaderboard.py"
- "signetry_eval/report.py"
- "signetry_eval/scenarios/**"
- "pyproject.toml"
- "Makefile"
workflow_dispatch:

# Read-only: nothing here writes to the repository any more.
permissions:
contents: write
contents: read

concurrency:
group: published-numbers
cancel-in-progress: false
# Nothing is pushed any more, so there is no cross-workflow write to serialise against
# benchmark.yml; this only stops a superseded commit's verification from running
# alongside the commit that replaced it.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
leaderboard:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
# Never the incoming ref: on a release event that is the tag, which
# leaves us in detached HEAD and turns the push below into a tag write.
ref: ${{ github.event.repository.default_branch || 'main' }}
# Verify the ref that triggered us. On a pull request that is `refs/pull/N/merge`
# — the merge result, which is what actually lands — because verifying the default
# branch would say nothing about the change under review. The one exception is a
# release: its ref is the tag, which is not where the page is committed.
#
# The *release* case is the true branch on purpose. The obvious spelling,
# `event_name == 'pull_request' && '' || default_branch`, is a trap: Actions' `&&`
# and `||` yield operand values rather than booleans, so a falsy true branch ('')
# collapses to the false branch and *every* event checks out the default branch.
# This workflow shipped that bug for one run: PR #39 verified `main` instead of
# itself. Both branches here are non-empty strings.
ref: ${{ github.event_name == 'release' && (github.event.repository.default_branch || 'main') || github.ref }}
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Install
run: pip install -e ".[dev]"

- name: Does this build have the detection engine?
id: engine
run: |
# Axis 2 needs signetry-core's SAST engine. If it is absent we publish axis 1
# alone and say so on the page. We do not print an empty table, and we do not
# score a scanner that did not run as zero.
if python -c "from signetry_core import scan_repository" 2>/dev/null; then
echo "available=true" >> "$GITHUB_OUTPUT"
else
echo "available=false" >> "$GITHUB_OUTPUT"
echo "::warning::signetry-core lacks scan_repository; publishing the governance axis only."
fi

- name: Regenerate the leaderboard
id: regen
run: |
mkdir -p docs
# Record WHICH signetry-core produced the live row. A leaderboard that says
# "measured live in CI" without naming the version measured cannot be
# reproduced by a reader, which is the one thing it exists to support.
# Read from the installed package, never hardcoded — a stale label would be
# worse than the null it replaces. Left unset if it cannot be read, so the
# row degrades to "not measured" rather than to a guess.
CORE_VERSION="$(python -c 'import signetry_core; print(signetry_core.__version__)' 2>/dev/null || true)"
if [ -n "$CORE_VERSION" ]; then
VERSION_ARG="--version $CORE_VERSION"
echo "live row labelled signetry-core $CORE_VERSION"
else
VERSION_ARG=""
echo "::warning::could not read signetry_core.__version__; the live row will be unlabelled."
fi

# Runs the adversarial suite for the live signetry-core row. A failing defense
# still publishes — the row shows the failure. The gate below is what fails.
if [ "${{ steps.engine.outputs.available }}" = "true" ]; then
signetry-eval leaderboard --with-detection $VERSION_ARG > docs/LEADERBOARD.md
signetry-eval leaderboard --with-detection $VERSION_ARG --json > docs/LEADERBOARD.json
else
signetry-eval leaderboard $VERSION_ARG > docs/LEADERBOARD.md
signetry-eval leaderboard $VERSION_ARG --json > docs/LEADERBOARD.json
fi
# `make leaderboard` is the single recipe, shared with contributors. CI does not
# inline the commands: when it did, the workflow and the regeneration
# instructions printed on the page drifted, and the page ended up documenting a
# command that produced a different page.
#
# The recipe reads the version label from the installed signetry-core and fails
# outright if the SAST engine is absent, rather than regenerating an axis-1-only
# page that would diff against the committed two-axis page and read as merely
# stale when the real problem is the install.
# The checked-out tree must be the one that carries the recipe. `make leaderboard`
# without a Makefile does not necessarily fail: `leaderboard` can name a real path in
# this repository, in which case make treats the target as already up to date and
# exits 0 — "Nothing to be done for 'leaderboard'". A regeneration step that silently
# regenerates nothing is the same class of bug as the push step that silently
# exited 0 on "no change to publish", so check before trusting it.
test -f Makefile || {
echo "::error::No Makefile in the checked-out tree — regeneration would silently do nothing. Wrong ref?"
exit 1
}
make leaderboard

- uses: actions/upload-artifact@v7
with:
Expand All @@ -98,23 +119,25 @@ jobs:
docs/LEADERBOARD.json
if-no-files-found: error

- name: Commit the refreshed page
- name: Are the committed numbers current?
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add docs/LEADERBOARD.md docs/LEADERBOARD.json
git diff --cached --quiet && { echo "no change to publish"; exit 0; }
git commit -m "docs: refresh the governance leaderboard"
# benchmark.yml pushes to the same branch on release. The concurrency group
# serializes us, but rebase anyway rather than force or fail.
git pull --rebase --autostash origin "${{ github.event.repository.default_branch || 'main' }}"
git push origin "HEAD:${{ github.event.repository.default_branch || 'main' }}"
# Diff-only: the regeneration above already wrote into the working tree.
make check-leaderboard || {
echo "::error::docs/LEADERBOARD.{md,json} are stale. Run 'make leaderboard', commit the result, and open a pull request."
exit 1
}

- name: Gate — every governed defense must hold
# Must run even when the staleness check failed. Ordering this step last is
# deliberate, but `if:` is what makes the ordering safe: a failing step above
# aborts the job before the gate, so a broken defense would surface as a stale-page
# error and the defense itself would never be judged. Skipped only when
# regeneration did not produce numbers to judge in the first place.
if: ${{ !cancelled() && steps.regen.outcome == 'success' }}
run: |
# Last, deliberately: the page is published above whatever this says, then this
# Last, deliberately: staleness is reported above whatever this says, then this
# step fails the run so a broken defense is impossible to miss.
signetry-eval run > /dev/null || {
echo "::error::An adversarial scenario's defense did not hold. The leaderboard was published with the failing row."
echo "::error::An adversarial scenario's defense did not hold."
exit 1
}
Loading