Skip to content

Triage: catalogue EvoRiskBench, PAGE and STEER-Bench (#196, #194, #198) - #222

Open
emmanuelgjr wants to merge 1 commit into
mainfrom
triage/benchmarks-evorisk-page-steer
Open

emmanuelgjr wants to merge 1 commit into
mainfrom
triage/benchmarks-evorisk-page-steer

Conversation

@emmanuelgjr

Copy link
Copy Markdown
Contributor

Three rows for evals/EXTERNAL_BENCHMARKS.md, from the watcher triage of the issues below, plus a precedent note in docs/TRIAGE_RULES.md rule 3.

Closes #196
Closes #194
Closes #198

The rows

Benchmark Paper Scale, as published Release Disclosure in row OWASP entry (DRAFT)
EvoRiskBench arXiv:2610.03153 450 adversarial tasks, 6 scenarios, 9 model-harness configs not released: the paper says it will be released "after completion of artifact safety and reproducibility checks" — ASI02 · ASI05
PAGE arXiv:2610.03434 not stated in the abstract not released: the arXiv record links none the authors' own guardrail (Persona Guardrail) is evaluated on it LLM01 · LLM03
STEER-Bench arXiv:2610.03089 101 tasks, 9 domains not released: the arXiv record links none the authors' own defense (COBRA) is evaluated on it LLM01 · ASI01

Every figure and quotation is taken from the paper's abstract, as the catalogue's header requires. Nothing has been re-run.

The precedent (maintainer ruling, 2026-10-09)

This PR adds the first catalogue rows for benchmarks that are a secondary contribution of a defense paper. Rule 3 now says:

  • a defense paper that also introduces a named benchmark gets a row for the benchmark;
  • the defense itself is noted;
  • the row names the authors' own defense evaluated on it;
  • a defense paper with no named benchmark is noted and closed, as before.

C4

DRAFT — SME review required. The OWASP entry column is a reading of each benchmark's stated scope, not a reviewed mapping, under the catalogue's existing DRAFT banner.

Checks (local)

  • validate.js: 0 errors / 92 warnings / 329 passed, the same before and after.
  • test:scripts: 137/137.
  • stats:check: current.
  • generate.js: no drift.
  • markdownlint: no new findings in either file. The 16 pre-existing MD060 findings come from a newer local linter than CI's v16.

The link check doesn't run on PRs. The three arXiv URLs resolved when checked today.

🤖 Generated with Claude Code

Three benchmark rows from the watcher triage of #196, #194 and #198.
None of the three has released its benchmark, so each row records it
as not released, quoting the paper or noting the arXiv record links
none. PAGE and STEER-Bench are introduced by defense papers; each row
names the authors' own defense evaluated on it, as the APort Vault
row does for its disclosure.

The OWASP entry column is DRAFT, as for every row in the catalogue.

docs/TRIAGE_RULES.md rule 3 records the maintainer ruling that a
defense paper introducing a named benchmark gets a row for the
benchmark, and one with no named benchmark is noted and closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment