Repository navigation
feat(score): score a finished agent session with a team scorer - #1298
Conversation
Testing: The description documents sufficient testing: a post-build real-CLI run at |
cbe8f76 to
77e9c5d
Compare
Both earlier findings remain unresolved; neither is raised as a new issue. Testing: The description documents sufficient post-build real-CLI verification at |
77e9c5d to
58b18c5
Compare
Earlier findings: Both are resolved: attribution now uses captured delivery provenance, and historical usage no longer depends on current files. Testing: The description provides sufficient post-build real-CLI verification. Older commit references and explicitly deferred provider/agent coverage are not blocking. I ran no PR code, builds, installations, or tests. |
58b18c5 to
1f395bb
Compare
Earlier findings: The original provenance and historical-file checks are resolved, as is the omission of skills delivered by the startup pull. The resume scenario above is distinct. Testing: The description documents sufficient post-build real-CLI verification. Older commit references and explicitly deferred provider/agent coverage are not blocking. I ran no PR code, builds, installations, or tests. Scope limitation: The shallow checkout has no merge base for the requested three-dot diff. This review uses the direct comparison of the two supplied commits. |
1f395bb to
280753a
Compare
Earlier findings: The resume/replacement-skill attribution issue remains unresolved: Testing: The description documents sufficient post-build real-CLI verification. Older commit references and explicitly deferred provider/agent coverage are not blocking. I read only the requested diff; I ran no PR code, builds, installations, or tests. |
280753a to
005f08c
Compare
|
Earlier findings still unresolved
Resolved: The original provenance and historical-file issues, new-revision startup-pull omission, incompatible-driver closure, and archived-pass-range regression remain resolved. No additional confident findings. Testing: The description documents sufficient post-build real-CLI verification. Older commit references are not blocking; explicitly deferred provider/agent coverage is acceptable. I read the requested three-dot diff and trusted base review criteria. I ran no PR code, builds, installations, or tests. |
005f08c to
49cdb78
Compare
|
Earlier Findings Still Unresolved
Resolved: The original provenance and historical-file issues, new-revision startup-pull omission, incompatible-driver closure, and archived-pass-range regression remain resolved. No additional confident findings. Testing: The description documents sufficient post-build real-CLI verification. Older commit references are not blocking, and explicitly deferred provider/agent coverage is acceptable. I read the requested three-dot diff and trusted base review criteria. I ran no PR code, builds, installations, or tests. |
49cdb78 to
9339c56
Compare
Earlier findings: Same-revision restoration and timestamp-based resume/replacement attribution are resolved. The skipped-read finding remains partially unresolved, as described above. The original provenance, historical-file, new-revision startup-pull, incompatible-driver, and archived-pass-range findings remain resolved. Testing: The description documents sufficient post-build real-CLI verification. Older commit references are not blocking, and explicitly deferred provider/agent coverage is acceptable. I read the specified three-dot diff and trusted base review criteria. I ran no PR code, builds, installations, or tests. |
|
Fixed in 9339c56, each folded into the slice that introduced it:
Checked with a real Claude Code session for the first and third, and the live |
|
On the three findings still listed at 5b4e493:
The last two are decided and will not change in this PR. |
|
I couldn’t review the current diff: the shallow checkout contains the base commit but lacks
Please provide the requested diff or make the head commit and merge-base history available locally. I ran no PR code, builds, installations, or tests. |
|
I couldn’t review the current diff: the shallow checkout lacks
Please provide the requested diff or make the head commit and merge-base history available locally. I read the trusted base review criteria and ran no PR code, builds, installations, or tests. |
6842f11 to
57953c6
Compare
|
Earlier Findings Still Unresolved
Resolved: Resumed workflow passes now filter usage by launch time, fixing inherited usage from earlier passes. All other previously resolved findings remain resolved. No additional confident findings. Testing: The description documents sufficient post-build real-CLI verification with concrete results. Older commit references are not blocking, and explicitly deferred provider/agent coverage is acceptable. I reviewed the specified three-dot diff against the trusted base criteria. I ran no PR code, builds, installations, or tests. |
57953c6 to
882de19
Compare
Code Erosion ReportInformational — never blocks the merge.
Scanned 431 files / 97050 SLOC · high-CC functions 761/8522 * On TypeScript, Rule hits (TS verbosity layer)Standalone
66 distinct lines flagged across |
|
Earlier Findings Still Unresolved
Resolved: Resumed workflow passes now filter usage by launch time. All other previously resolved findings remain resolved. No additional confident findings. Testing: The description documents sufficient post-build real-CLI verification with concrete results. Older commit references are not blocking; explicitly deferred provider/agent coverage is acceptable. I reviewed the requested three-dot diff against the trusted checked-out criteria. I ran no PR code, builds, installations, or tests. |
882de19 to
3c415e3
Compare
|
Earlier Findings Still Unresolved
Resolved: Resumed workflow passes filter usage by launch time. Other previously resolved findings remain resolved, including historical capture attribution, same-revision restoration, and code-mode Testing: The description documents sufficient post-build real-CLI verification with concrete results. Older commit references are not blocking; explicitly deferred provider/agent coverage is acceptable. I reviewed the requested three-dot diff against the trusted base criteria. I ran no PR code, builds, installations, or tests. |
teamai score --session <id> [--source <platform>] --scorer <name> judges a
finished agent session with the local judge and keeps one label per scorer
with source "session" and subject { platform, sessionId }. Without
--session, the agent session variable names the session when it names
exactly one; otherwise the command fails and never guesses. An id that
ends more than one session's id is refused with the ids it matches.
The label, its scorers and its reasoning belong to the scope the session
ran in, found from the session's directory as its hooks found it, so
scoring from another directory changes nothing.
The evidence is the redacted session with its subagents' transcripts,
fenced as untrusted data and cut once at 200,000 characters; the
transcript file the judge can read holds the same cut text. Redaction
also masks the values of the secrets the team declares in
env/secrets.yaml, in the scorer prompts too. The reasoning file starts
with the member's own redacted first prompt, not text the tool
injected. Codex session titles use the same typed text, so a first line
naming a file stays the title.
A label that did not pass prints the share hint workflow labels print,
with the scored session:
To share what went wrong, ask your agent to read <reasoning file> and
share it with the team (teamai contribute --session-id <id>).
teamai score --push publishes every pending label to scores/<member>/<YYYY-MM>.jsonl on teamai-reports, alone or after --session, through the label outbox: merged by key against origin, the newest scoredAt kept. With usage reporting off, or when the push fails, the labels stay pending for the next --push. An http team repo has no teamai-reports branch, so --push answers the unsupported_repo error and the labels stay here. teamai --dry-run score stays refused before anything runs. teamai doctor checks "Score labels are published" next to the learnings check, with the fix teamai score --push.
The session-start hook writes one capture per start, before the
background pull runs:
<dataHome>/sessions/captures/claude-code/<session>/<captureId>.json
{ captureId, platform, sessionId, capturedAt, sessionFile,
agent: { tool, model, team: { revision, localChanges }, project, driver: "session", fingerprint } }
The hook reads the checkout's git state once, capped at two seconds,
and hands it to both the capture and the dashboard's session_start
event. The tool version is added by the background pass, so the hook
never waits on --version. A resume writes another capture; a
compaction, or a session teamai launched (TEAMAI_LAUNCHED_BY), writes
none. When the session's own pull delivers a new revision, its captures
get teamChanged: true. Both rewrites take a per-session lock, so two
hook processes of one session keep each other's change. A capture that
fails is logged at debug level. The background pass removes the
captures of a session whose file has been gone for a day.
A session label takes its agent from the captures:
every capture the same (tool, model, effort, team) -> agent
same, one marked teamChanged -> agent without team and fingerprint
captures differ, or none -> no agent
model.reported is set only when the session's messages name one model.
--source can be left out when the captures name one platform for the id.
A pull writes the member's selection into the checkout's record when it
delivers:
lastPullByWorkspace[<checkout>].selection =
{ scope, primaryRole, additionalRoles, projects, subscribedTags,
excludedSkills, inheritUserScope }
The agent observation copies it into team.selection, with the record's
workspace children, never from the current config; the fingerprint
covers it. A record an older CLI wrote without it gives no team and no
fingerprint. Session captures and workflow passes share the builder.
backend member -> team.selection.primaryRole "backend", fingerprint A
frontend member -> team.selection.primaryRole "frontend", fingerprint B
teamai roles set, no pull -> the next capture keeps the last pull's selection
A session's pull that delivers the same revision for another selection,
or for other workspace children, marks its captures teamChanged, as a new
revision does.
A session label carries the team skills the session or its subagents
loaded:
used: { skills: ["deploy-notes", "review-notes"] }
They are found in the session's messages by the rules the skill-use
hooks apply to their events: a call to the tool's skill tool, a read of
a SKILL.md under a skill root, or a /name or Codex $name mention the
member typed, not one in text the tool injected, such as AGENTS.md.
A read counts on one line of a longer shell command too, and in a
Codex code-mode script, for each command it passes as a string:
sed -n '1,260p' ~/.agents/skills/review/SKILL.md\nrg -n gate notes.md
tools.exec_command({cmd:"cat ~/.agents/skills/review/SKILL.md"})
A skill is credited only when the session's captures recorded that
teamai delivered it, unchanged, when the session started or by the
session's own pull, and a read counts only when it read that delivered
copy. Today's files and team repo play no part:
team skill loaded, removed by a later pull -> still credited
team skill the session's own pull delivered -> credited
member's own skill with a team skill's name -> not credited
no capture for the session -> nothing credited
A driver's report names the team skills the pass loaded, and the
workflow label of that pass carries them, as a session label does:
teamai workflow report <run> --json '{..., "used": {"skills": ["review-notes"]}}'
label: { source: "workflow", subject: {...}, used: { skills: ["review-notes"] } }
Names only. A value not shaped like a skill name, such as a path, is
dropped, not rejected; a `used` that is not an object with a list of
strings is refused as invalid_report. A report without `used` means
nothing was observed, so its label has none; `"skills": []` means no
team skill was loaded. A `workflow:` step's label lists every skill its
child passes reported, and has none when none of them reported `used`.
The pass keeps `used` in the run file and in the published snapshot.
The run record does not carry it.
The local driver reports `used` for a prompt pass, so its workflow label names the team skills the pass or its subagents loaded, by the rules a session label applies: pass loads review-notes, delivered unchanged -> ["review-notes"] member's own skill with a team skill's name -> not credited a pull removes the skill while the pass runs -> still credited no pull record for the checkout -> no `used` At launch the driver reads the checkout's pull record once, for the agent observation and for the team skills the checkout holds as delivered. After the pass it reads the session the tool reported, subagents included. Those files stay on this machine; the report carries names only.
The scorer guide, in both languages, shows scoring a session from inside it and by id, what the command prints before and after the judge, --push and the usage-reporting opt-out, and where captures, labels and reasoning live. The share workflow lists a failed label's reasoning file as something it can turn into a learning. The scorer guide also says which team skills a workflow label names, how a driver reports them with `used` in `teamai workflow report`, and that a session's own pull at start counts. The workflow guide lists `used` with the other report fields.
3c415e3 to
0f21aba
Compare
|
Earlier Findings Still Unresolved
The description now acknowledges both limitations; neither is fixed. No additional confident findings. Resolved: Resumed workflow passes filter usage by launch time. Other previously resolved findings remain resolved, including historical capture attribution, same-revision restoration, and code-mode Testing: The description documents sufficient post-build real-CLI verification with concrete results. Older commit references and explicitly deferred provider/agent coverage are not blocking. I reviewed the exact requested three-dot diff against the trusted checked-out rules. I ran no PR code, builds, installations, or tests. |
jeff-r2026
left a comment
jeff-r2026
left a comment
There was a problem hiding this comment.
Reviewed (agent review: no blocking findings; P2 follow-ups: doctor's 'Score labels are published' check never clears on http team repos because LabelStore.unpublished() ignores usesBranchWorktree, and SessionCaptures.atStart has no time cap on the session-start path). Verified merged onto main: tsc, lint, build pass; full unit suite 15376 passed with only environment failures that reproduce on plain main; hooks-team / scope-isolation / scorer E2E 17/17. The rebased head 0f21aba has the identical tree to the verified merge. CI 18/18.
|
Follow-up: both [P2] findings, the skipped read after |
Closes #1178. Stacked on #1297.
teamai scorecan now judge a finished Claude Code, Codex, or OpenCode session with a team scorer. The judge runs on the member's machine, the reasoning stays there, and--pushpublishes only the label.What a team member sees
Inside a session,
teamai scorescores that session. Outside one,--sessionnames it:Scoring never guesses the session, and it has no dry run:
The label is the workflow label from #1297, with
source: "session":{"scorer":"quality","rubric":{"sourceHash":"f47a4a4a…","teamRevision":"6ae64ecd…"}, "label":"followed","score":1,"passed":true,"source":"session", "subject":{"platform":"claude-code","sessionId":"baa8107c…"}, "used":{"skills":["deploy-notes","review-notes"]}, "agent":{"v":1,"tool":{"name":"claude","version":"2.1.308"},"model":{"reported":"…"}, "team":{"revision":"2ab46400…","localChanges":false,"selection":{"primaryRole":"backend",…}}, "project":{"commit":"f666c037…","dirty":false},"driver":"session","fingerprint":"…"}, "judge":{"kind":"model","tool":"claude","model":{"requested":"haiku","reported":"claude-haiku-5-5","alias":"fast"},"isolation":"verified","access":"read","usage":{…}}, "scoredAt":"2026-10-23T02:45:21.940Z"}agentcomes from the captures the session-start hook writes, before that hook's pull starts. A resume writes another capture. A session started by a workflow driver or a judge writes none. The label keepsagentonly when every capture agrees. When the session's own pull delivered a new revision, it dropsteamandfingerprint. Scoring reads the capture's scope, soteamai score --sessionfrom another directory keepsagent. Captures of sessions whose file is gone are swept.team.selectionis the role, projects, and tags the last pull delivered for, kept in that checkout's pull record. Changing a role without pulling does not change it.used.skillslists the team skills the session or its subagents loaded through the skill tool, a read of theirSKILL.md, or a/nameor$namemention the member typed. Each one is checked against what the capture in effect when it loaded recorded as delivered, and a skill a resume's pull delivers counts from that resume on, so a member's own skill with a team skill's name is not listed, and a later pull that removes a skill does not change a rescore. Text Codex injects fromAGENTS.mddoes not count. A Codex command's paths are read from itsworkdir.used.skillstoo, with every driver. An external driver reportsused: {skills}with the pass, names only, and drops what is not a skill name. The local driver works it out from the pass's transcript since its launch and what the pull delivered, so a pass that resumes a session does not take an earlier pass's loads. A sub-workflow step's label carries the skills of its child passes.model: fastcaps what a long session costs. When a member's~entry ormodels switchturns the alias off,scoreprints oneℹline saying so, in the wordsdoctoruses.--pushprintsUsage reporting is off, so labels stay on this machine.and the labels stay pending. On a team repo without ateamai-reportsbranch it answersunsupported_repo.teamai doctorfailsScore labels are publishedwhile labels wait.--session-id <scored id>, socontributemarks the scored session. Codex session titles are the member's own first line, also when it names a file.docs/guide/scorers.mdand its Chinese page.skill-data/sharelists a failed label's reasoning file as something to share.skill-data/core/references/commands.mdis regenerated.Out of scope
Scoring part of a session, background scoring (#1179), scoring a subagent as its own subject, showing labels (#1182), and a spending budget. Assets delivered to the user scope (HOME) are not in
agent.How it was verified
npm run build,npx tsc --noEmit,npm run lint, and the tests each commit touchesnpx vitest runat the head (3c415e32), on #1297 atfe7e22b0fe7e22b0after #1280 mergednpm run test:e2e -- score-external-driver score-session workflow-records-reports-branch-1083 workflow-publish-resume hooks-team-e2e hooks-project-isolation-issue373 scope-isolation-issue85at the head (3c415e32)d148b2bb(same code as the head for these paths) and at12cd3f2c, afternpm run build, for this round's two fixes12cd3f2c(record below)12cd3f2c, on #1280 atac82dde5, afternpm run buildused.skillsfor sessions and workflow passes, all passed. Three runs shared with #1297, 191 checks, all passed.58b18c5f77e9c5d2, the judge modeljudge.model {requested, reported, alias}; oneℹline when a member's~entry turns the alias offcbe8f764, the session path end to endThe two review fixes, real CLI at
d148b2bband at12cd3f2c, afternpm run build.A skill a resume's pull delivers. Project install with
--agent claude. A real Claude Code 2.1.308claude -psession, where the member types/notes-r19bbefore the team has that skill. The team then publishes it. The session-start hook runs again withsource: resume, and its pull delivers the skill. The judge is real Claude Code.A Codex command's
workdir. User install. The run uses a Codex rollout copied from a real one, plus one code-mode calltools.exec_command({cmd:"cat .codex/skills/wayfinder/SKILL.md",workdir:"<home>"}), made from a session whose cwd is the app.The run at
12cd3f2cusedteamai init --scope project --agent claudeagainst a bare git team repo with two team skills, plus one skill of the member's own. Sessions are real Claude Code 2.1.308claude -pruns, and every judge is real Claude Code.team.selection. The pull delivered a new revision and marked the captureteamChanged.--sourceneeded. The printed first prompt is the member's own line.used.skillsnamesreview-notes(skill tool) anddeploy-notes(loaded only by a subagent), not the member's own skill. Because the team changed,agentkeeps noteamand no fingerprint. The judge ranverifiedwithreadaccess. A phrase only the judge wrote is in the local reasoning file.agent.team.selectionand a fingerprint.--pushteamai-reportswith no reasoning. The judge's phrase, a code word from the transcript, and the first prompt are nowhere on the branch.Earlier real-CLI runs on this branch, with the same code as the head for these paths, covered a typed
/review-notes check whether attempts < 3as the first prompt, a read of a teamSKILL.md, Codex rollouts copied from real ones where$review-notesappears only in the injectedAGENTS.mdblock, a native Codex session TeamAI had no dashboard events for, scoring from inside a session throughCLAUDE_CODE_SESSION_ID, the no-session and dry-run refusals, resume captures, a label with noagentwhen two captures differ in model, no capture from a--driver localstep, and two members on different roles with differentteam.selection.At
58b18c5f, real Claude Code sessions in a project install:Not run here: a live Codex or OpenCode session (the Codex checks use copied rollouts), CodeBuddy, and the
gitlabandgithubproviders. That coverage is left to CI. While this PR is stacked, CI runs only the review and the source scan here; Lint & Test and the E2E jobs run once it targetsmain.Known limits
used.skillscomes from what a session's commands say, not from what they did. Two cases follow from that, and both are kept on purpose:SKILL.mdread after&&counts even when the command before it fails, as infalse && cat .claude/skills/review/SKILL.md; true. Parsing the command line cannot tell whether the left side succeeded, and in real sessions it does (test -f .claude/skills/review/SKILL.md && cat .claude/skills/review/SKILL.md), so dropping reads after&&would lose real loads. A read after||, which runs only when the left side fails, does not count.tools.exec_commandread counts, including one in a branch that never runs, such asif (false) { … }. The rollout records the script, not which of its calls ran.Each trades a rare false positive for not missing real loads. Telling them apart needs the command's output as evidence of the read, which is a change to how reads are attributed and is not part of this PR.
Known follow-ups
Commits