Repository navigation
Interview eval harness (S0): simulated users, blind judge, paired test - #57
Merged
Merged
Conversation
|
Warning Review limit reached
This review includes 31 billable files and costs up to $7.75.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 43 minutes for your next included review. View limit details
Comment |
4 of 5 tasks
cleak
force-pushed
the
feat/interview-skill-rewrite
branch
from
October 11, 2026 22:05
97b40f7 to
e7682eb
Compare
cleak
force-pushed
the
feat/interview-eval-harness
branch
from
October 11, 2026 22:05
277e6e3 to
f4785e9
Compare
cleak
force-pushed
the
feat/interview-skill-rewrite
branch
from
October 13, 2026 04:08
e7682eb to
a029d8b
Compare
Stage S0 of the interview-challenge proposal: measure the skill before
and after changes instead of judging by feel.
- evals/interview/setup.sh builds a throwaway sandbox tempyr project
(its own git repo, fixture graph, structural-only index) with two skill
arms: baseline from a git ref (default origin/master), candidate from
the working tree or a ref.
- /tempyr-eval (a skill inside the sandbox) runs workflow.js through the
Workflow tool: per run, a fresh interviewer subagent per turn drives
the real tempyr MCP tools from its arm's SKILL.md, and a simulated-user
subagent answers from a persona card with hidden facts and a patience
budget. Messages are relayed verbatim; runs never commit.
- A blind judge per run grades against rubric.md and the scenario's
judge notes (inspecting the session via interview_show {full}); the
script aggregates metrics deterministically and the skill writes a
summary, transcripts, and blind pairs for the owner's comparison.
- Six scenarios: vague hobby idea, sound audit mandate (negative
control), self-answer bait, open idea with a hidden better framing,
contradiction with a fixture constraint, overlap with a fixture
feature.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- setup.sh builds the workspace outside the repo (default ~/.tempyr-eval/interview, --dir / $TEMPYR_EVAL_DIR): sandbox/ and harness/ are rebuilt, results/ is kept. Agents no longer inherit this repo's CLAUDE.md or skills, and scenario files sit outside the interviewer's working directory. The default baseline is the skill before the rewrite, found from history, so it stays right after merge. - The eval skill runs one workflow per arm; workflow.js also lowers the turn cap (with a log line) if a run would exceed the 1000-agent limit. - Interviewers aren't offered graph_context (it always embeds and the sandbox has no key), are told not to use the Skill tool, and don't see the scenario id. A run with no session id ends as `no_session` instead of restarting the interview every turn. - Aggregation selects scenarios by `scores` keys (adds `push`), and recall uses each scenario's declared planted-flaw count; the judge is told to list every planted flaw. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cleak
force-pushed
the
feat/interview-eval-harness
branch
from
October 13, 2026 04:11
f4785e9 to
644084a
Compare
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is S0 of
docs/interview-challenge-research.md. It measures the interview skill (#55 vs the current one) instead of judging it by feel. It's stacked on #55, and the candidate arm needs the tools added in #54.The whole harness runs inside one Claude Code session as subagents orchestrated by the Workflow tool. That means it uses the real skill, tools and MCP server, and bills against your subscription. There's no SDK and no API keys.
Pieces
evals/interview/setup.shbuildsevals/interview/.run/sandbox(gitignored):baseline(the skill at a ref, defaultorigin/master) andcandidate(working tree or a ref)tempyron PATH includes Interview: advance phases only on answers; add full-content reads #54/tempyr-eval: a skill that lives in the sandbox. It reads the scenarios, callsworkflow.jsvia the Workflow tool, and writesresults/<timestamp>/containing:summary.md: metrics against the proposal's provisional targetspairs/for the owner's 8-pair comparison, with formatting normalized and a separateKEY.mdworkflow.js: one run is an interviewer subagent and a simulated-user subagent taking turns.SKILL.mdand calls the realinterview_*tools; session state lives in the MCP server.rubric.mdand the scenario'sjudge.md, inspecting the session throughinterview_show {full}. The script aggregates the metrics deterministically.Scenarios D1–D6:
Each scenario's brain dump, persona and judge notes sit in separate files, so the interviewer, simulated user and judge each see only their own.
Test plan
setup.shbuilds the sandbox, validates the fixture graph (4 nodes, 2 edges) and builds the index. Works under Git Bash on Windows (ref:pathargs are protected from MSYS path conversion).interview startstays in Discovery, and one answer moves to Product (Interview: advance phases only on answers; add full-content reads #54). Filter-onlyjournal search --tagworks.workflow.jssyntax-checked with node, and run end-to-end with mockedagent()/pipeline(), which exercises the turn loop, gave-up handling and aggregation.cd evals/interview/.run/sandbox && claude, then/tempyr-eval). This needs a Claude Code session started in the sandbox, so it couldn't run from this session.🤖 Generated with Claude Code