AI verification. I test whether the numbers that decide what AI ships (benchmark scores, LLM-judge verdicts, rater labels, RL reward signals) can be trusted, and build the checks that catch them when they can't.
| Repo | Verifies | Result |
|---|---|---|
| edge-diag-agent | offline LLM answers on industrial drives | on 40 SINAMICS G120 fault codes, a 4-bit Qwen2.5-3B on 4 CPU cores matched the manual's meaning (0.94 word overlap) with the manuals and failed without them (0.06; it called 30 of 40 an encoder fault); a gate flags the 2 answers that stated the wrong acknowledgment |
| int8-flip-audit | INT8 models on edge NPUs | on 5,000 images, 10.4–24.8% of a Qualcomm NPU's answers differed from the PC INT8 check that passed the model, while accuracy moved by less than 2 points |
| verdict-gate (paper) | LLM-judge self-preference | on 8,794 real AI-shopping answers with names hidden, a Gemini judge picked Gemini 88–91% of the time vs. 38–40% for a third-family judge; showing the judge only the claims cut Gemini's bias from 50 to 24 points |
| rl-env-audit (paper) | RL environment graders | 60 public environments scanned: 19 certified, 2 graders paid replies with no correct answer; both fixed to zero, honest answers unchanged |
| reward-audit (paper) | RL reward signals | GRPO with an LLM judge: reward 0.55 → 0.73 while true accuracy 0.22 → 0.30; auditing 1 in 8 outputs gives a provably unbiased reward (0.22 vs. a true 0.23) |
| rater-audit | rating pipelines | on real ratings from 123 people, a 3-rater majority matched the expert label only 63% of the time |
| score-audit | LLM-judge verdicts | false "incorrect" verdicts cut from 42 to 5 when the judge must quote the false sentence |
| judge-ceiling | judge leaderboards | 39 published scores reproduced exactly; one stated threshold moves the mean by 38% |
| leaderboard-audit | model rankings | the model ranked 1st on ground truth falls to 5th depending only on which judge grades |
| asr-tier-audit (paper) | accuracy claims | scoring choices flip the verdict in 5 of 6 languages; an anytime-valid test settles the claim |
| runopsy | agent runs | finds the step where an agent run started going wrong, then replays it to confirm |
| longhaul-bench | long-horizon agents | plausible but wrong memory cost 19.7 points; nonsense labels changed nothing (single-seed pilot) |
Website · Google Scholar · ORCID · LinkedIn