Repository navigation
Conversation
The PR description documents sufficient testing, including a post-build real-CLI record at the head. No testing-record finding. I did not run, build, or install PR code. |
58893b8 to
92784ce
Compare
Both earlier findings are resolved: ownership follows transitive archived ancestry, and Codex preserves its configured model without a custom provider. The description documents sufficient post-build real-CLI testing. The older recorded commit is a note, not blocking. Review limitation: the shallow checkout cannot compute the requested merge base, so I inspected the endpoint diff. I did not run, build, or install PR code. |
92784ce to
6fb5549
Compare
All four earlier findings are resolved in the current diff. The description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed the requested three-dot diff only. I did not run, build, or install PR code. |
6fb5549 to
5f4825d
Compare
All previously reported findings are resolved; these are separate remaining issues. The description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed only the requested three-dot diff. I did not run, build, or install PR code. |
5f4825d to
3737a5c
Compare
These two latest findings remain unresolved. All findings from the earlier three passes are resolved. The description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed the requested three-dot diff only. I did not run, build, or install PR code. |
3737a5c to
8e85930
Compare
|
No new findings I’m confident warrant reporting. All earlier findings are resolved in the current diff, including rubric redaction and ownership selection across archived passes. The PR description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed only the requested three-dot diff. I did not run, build, or install PR code. |
8e85930 to
acbb6fd
Compare
All earlier findings are resolved in the current diff. The PR description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed the requested three-dot diff only. I did not run, build, or install PR code. |
acbb6fd to
df34d57
Compare
This latest earlier finding remains unresolved. All other previously reported findings are resolved; no additional findings I’m confident warrant reporting. The description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed only the requested three-dot diff. I did not run, build, or install PR code. |
df34d57 to
00a4fd5
Compare
|
No new findings I’m confident warrant reporting. All earlier findings are resolved in the current diff, including inherited The PR description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed only the requested three-dot diff against the checked-out base’s Code Review Rules. I did not run, build, or install PR code. |
|
Fixed in 00a4fd5 (folded into
|
00a4fd5 to
78a7973
Compare
|
No new findings I’m confident warrant reporting. All earlier findings are resolved in the current diff, including inherited The PR description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed only the requested three-dot diff against the trusted base’s Code Review Rules. I did not run, build, or install PR code. |
78a7973 to
147c3fc
Compare
All earlier findings are resolved in the current diff, including inherited The description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed only the requested three-dot diff against the trusted base’s Code Review Rules. I did not run, build, or install PR code. |
Scorers need these without importing src/workflow/. Move only, no behavior change: - canonicalJson: src/workflow/snapshot.ts -> src/utils/canonical-json.ts - NoModel and its classification: src/workflow/step-model.ts -> src/models/aliases.ts (noModelReason); step messages stay put - stopProcess: src/workflow/process-stop.ts -> src/tool-launch/ - tool names and families, stdout parsers and TokenUsage: src/workflow/local-tool-adapter.ts and types.ts -> src/tool-launch/ - teamRevision helpers: src/workflow/cache.ts -> src/utils/team-revision.ts - DriverSchema: src/workflow/step-state.ts -> src/driver-identity.ts
A scorer is a Markdown file with Warp-style frontmatter:
---
labels:
- { value: good, score: 1, description: Does what was asked. }
- { value: bad, score: 0, description: Misses the request. }
passingScore: 1
samplingRate: 25 # 0-100, default 25
model: fast # optional
isolation: verified # optional, verified | best-effort
---
Judge whether the change does what the step asked.
Workflow sync now syncs scorers first, so pull (both paths), push and
review read the same tree. Valid scorers go to their own cache under
<dataHome>/scorers/cache/. scorers/README.md is ignored; any other file
warns at pull and is a fail finding in teamai review --branch. Review
also runs its leak check on every file in scorers/, as it does for
rules: a credential or an internal address fails the branch.
Adds docs/guide/scorers.md and its zh-CN twin, checked by the docs
example test. The failure-code guard now scans src/workflow/ and
src/scorers/ recursively.
defaults:
scorers: { batch: true }
steps:
- id: implement
prompt: Implement the change.
scorers: [review-quality]
- id: review
prompt: Review the change.
scorers: { names: [review-quality], batch: false }
scorers: goes on prompt, prompt_file and workflow steps. Batching
resolves step > defaults.scorers.batch > the team's default > off.
A name with no valid scorers/<name>.md is a fail finding in teamai
review --branch. At pull it is a warning, the workflow stays cached and
the step runs without it. teamai doctor lists invalid scorer files and
unknown names, since hook pulls run silently.
Nothing is judged yet.
$ teamai score --next --json '{"driver":{"kind":"t3","machine":"m","workspace":"/w",
"can":{"judge":{"isolation":"verified","tools":["claude"]}}},"running":[]}'
{"data":{"lease":{"ref":"s5d1e07c2a9b4/a91c0d7e","scorer":"review-quality",
"subject":{"runId":"r3f9…","step":"review","pass":1},"evidence":"…","prompt":"…/prompt.md",
"answer":{"schema":{…}},"expiresAt":"…","submit":"teamai score --lease s5d1e07c2a9b4/a91c0d7e --submit --stdin"}}}
$ teamai score --lease s5d1e07c2a9b4/a91c0d7e --submit --stdin < answer.json
{"data":{"label":{"scorer":"review-quality","label":"thorough","score":1,"passed":true,…}}}
Open judge work is computed, never queued: succeeded passes of steps that
list a scorer, the scorers the run pinned at start, deterministic sampling,
and the attempts under <dataHome>/scores/attempts/. The engine only stamps
installId on the passes it settles and within on archived passes; scoring
never writes a run file, so the run settles without waiting.
A workflow pass belongs to the install that settled its first child
pass: a step's passes settle in pass order, and a pass whose commits
start where another's end settled after it.
Leases last 30 minutes and renew while the driver lists them in running;
the fourth lapse fails the entry lease_expired. Entries expire 14 days after
the run last changed. A label outside the scorer's list is refused and the
lease stays open. workflow report refuses a judge ref. A project whose data
still lives in <project>/.teamai is offered nothing, and doctor says why.
Evidence is a placeholder until it is built (gaps: evidence_not_built).
<dataHome>/scores/labels.jsonl every label on this machine
<dataHome>/scores/reasoning/<hash>.md the judge's reasoning per label, never published
<dataHome>/scores/pending/<labelId>.json outbox
scores/<member>/<YYYY-MM>.jsonl on teamai-reports
One ScoreLabel type and parser for workflow and session labels. The newest
scoredAt wins per (scorer, rubric.sourceHash, subject, judge.kind), merged
against origin so an older label from another clone never replaces a newer
one. A pending label is removed only while it still holds what was sent.
Submit publishes at once; pull retries what is left. With usage reporting
off, labels stay pending.
Workflow Runs shows {scorer, label, passed} per pass, a hand label over a
model label, and a scoring total kept apart from the run's tokens: every
attempt on this machine, labelled ones for a published run. The same run at
samplingRate 100 and 0 has the same stepCounts, iterations, failure and
work tokens.
```sh
teamai score --lease "$REF" --json '{"driver":{"kind":"t3","machine":"m1","workspace":"/work"}}'
```
Runs the tool the lease names in a confined profile and settles the
lease through the same submit path as any driver.
- Claude: restricted read tools, evidence as an appended system prompt
file, prompt on stdin, model-switch env passed with --settings.
Verified for 2.1.308.
- Codex: temporary home with auth and a permission profile that lets
its shell read only the evidence, no network, no writes, core env;
[agents] enabled = false, --ephemeral, hooks/plugins/apps/multi-agent
disabled. Verified for codex-cli 0.163.0.
- OpenCode: --standalone with isolated config denying bash, edit, web,
task and external directories, no project config, and its database
and XDG dirs in a temp dir; the judge session and its children are
deleted after. Verified for v2.1.2 when the temp dir is outside any
git or hg checkout, best-effort otherwise.
- CodeBuddy: its Claude-like flags (read tools, no MCP, no saved
session, no setting sources), prompt on stdin. Best-effort; a run
that prints "Authentication required" fails judge_tool_unavailable
whatever its exit code.
- Env comes from an allow-list with TEAMAI_LAUNCHED_BY=scorer; the hook
dispatcher exits at once for scorer launches.
- The lease records the chosen tool and model (scorer alias, or tool
default for a local default or switch); an unmapped or unresolved
alias closes the entry with judge_model_unavailable.
- Each tool's login is read from its own home (toolHome), so
claude-internal, tcodex and a home moved by tool-homes.json judge
with their own settings and auth. The judge tools are the launch
tools plus CodeBuddy.
- LocalWorkflowTool is now LaunchTool.
```yaml # scorers/scoring.yaml isolation: verified ``` ```text $ teamai doctor Scoring isolation: verified, set by scorers/scoring.yaml isolation. ``` The team's scorers/scoring.yaml, a scorer's isolation and scoring.isolation in the scope's config.yaml may each ask for verified; the strictest wins and the default is best-effort. A value teamai does not know counts as verified and warns. score --next leases work only to a driver whose can.judge.isolation meets it, and the local judge refuses a tool whose profile is below it. scorers/<ns>/scoring.yaml applies where <ns> is an active scorers namespace, as its scorers do, and replaces the root file's key; two active namespaces give the strictest. A scoring.yaml that does not parse stops scoring in that scope: score --next hands out nothing and a submit fails with scoring_settings_invalid. teamai review --branch checks every scoring.yaml in every namespace, and neither pull nor review takes it for a scorer.
```yaml
- id: review
prompt: Review the change.
scorers: { names: [review-quality, test-coverage], batch: true }
```
```json
{"driver":{"kind":"t3","machine":"m1","workspace":"/work"},"outcome":"success",
"answer":{"review-quality":{"label":"good","reasoning":"…"},"test-coverage":{"label":"thin","reasoning":"…"}}}
```
A step's scorers.batch, its workflow's defaults.scorers.batch and the
team's batch in scorers/scoring.yaml now act. Two active namespaces
whose scoring.yaml set batch differently stop scoring in that scope, as
one model alias in two active namespaces stops aliases. Scorers on the
same pass that resolve to the same tool and model share one lease, one
evidence dir and one prompt listing each rubric; the answer schema has
one label per scorer and each label records judge.batched: true. A
batch never mixes tools or models. Submit checks every label first: one
bad label refuses the whole answer and the lease stays open. Renewal,
hand-back, release and the local judge treat the batch as one lease.
Evidence redaction also masks the values of the secrets the team
declares in env/secrets.yaml, in the scorer prompts too. A label that
did not pass prints where its reasoning is and how to share it:
To share what went wrong, ask your agent to read <reasoning file> and
share it with the team (teamai contribute).
```console $ teamai score --run r3f9 --step review --scorer review-quality # judge here, print the label $ teamai score --run r3f9 --step review --pass 2 --scorer review-quality --queue # any driver's --next takes it $ teamai score --run r3f9 --step review --scorer quality --scorer tests --batch --queue ``` Sampling does not apply. `--pass` defaults to the step's latest succeeded pass. A scorer the run pinned judges with that rubric; any other scorer teamai pull cached judges with its current one. The command records `<dataHome>/scores/attempts/<entryId>/request.json` with the pinned rubric. Labels and settled attempts older than the request no longer close the entry, so a failed or expired entry opens again, and the request starts a new 14-day window. Without `--queue`, the local judge runs here with the first judge tool on PATH, preferring the tool that ran the pass. An unknown run, step, pass or scorer fails with the values that exist. Taking a lease also sweeps what no attempt reads any more: rubric files no attempt names, the request of an entry that closed, and the raw output of an attempt a newer one replaced. Labels are never swept. teamai doctor lists the judge tools on PATH and the isolation each reaches here, and teamai status lists the cached scorers: Judge tools here: claude 2.1.308 (Claude Code) is verified, codex codex-cli 0.163.0 is best-effort. scorers: 2 (review-quality, test-coverage)
147c3fc to
4d9175f
Compare
This latest earlier finding remains unresolved. All other previously reported findings are resolved; no additional findings I’m confident warrant reporting. The description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking. I reviewed only the requested three-dot diff against the trusted base’s Code Review Rules. I did not run, build, or install PR code. |
review --branch now reads the branch's models/ and fails: - an aliases file pull cannot load, in the words doctor uses; - an agent, step, defaults.model or scorer whose model alias the branch's team files map for none of the tools that would use it. Member overrides, model switches and installed tools are the reviewing machine's, so they play no part. A model name is never a finding. Each role and project the branch declares, and members with neither, get their own agents and aliases files, so each is checked on its own and a finding names the ones it is about. An agent's model resolves as pull resolves it, its tool_extras model first, and a step whose agent sets no model is checked for defaults.model, as a run uses it.
docs/guide/scorers.md (and zh-CN) reads top to bottom: a scorer file, a step with `scorers:`, a run and its judge, any driver with `can.judge`, on demand with `--run`, batching, isolation per tool, and where labels live. The workflow pages keep one example and point there. docs/dev/workflow-engine.md names every way a judge lease settles.
4d9175f to
6a167a3
Compare
|
Fixed in 6a167a3. Checked with the real CLI. A branch adds roles $ teamai review --branch teamai/push/alice/1 --json # 6a167a32
ok: true, model findings: 0
$ teamai review --branch teamai/push/alice/1 --json # 4d9175f0
fail scorers/ops/quality.md - Scorer "quality" asks for model alias "fast", which has no model for any tool in models/aliases.yaml (no role or project, role dev, project svc-a). … |
|
No new findings I’m confident warrant reporting. All earlier findings are resolved in the current diff, including scorer-model validation being limited to contexts that receive the scorer. The PR description documents sufficient post-build real-CLI testing. Older recorded commit IDs are a note, not blocking; no testing-record finding. I reviewed only the requested three-dot diff against the trusted base’s Code Review Rules. I did not run, build, or install PR code. |
Closes #1085. Stacked on #1280.
A team keeps a rubric in
scorers/<name>.md, and a workflow step lists it. After the run settles, any driver can take the judge work withteamai score --next, or a member can judge it locally. The label goes to its own store onteamai-reports. The judge's reasoning stays on the machine that judged.What a team member sees
A scorer in the team repo:
A step that lists it:
The run settles first. Judge work is taken separately:
teamai score --lease <ref>without--submitruns the judge on this machine in a confined profile instead.teamai score --run <id> --step review --scorer review-quality [--queue]judges a pass on demand, whatever sampling says.Where things live:
<dataHome>/scores/labels.jsonl<dataHome>/scores/reasoning/<labelVersion>.md<dataHome>/scores/pending/scores/<member>/<YYYY-MM>.jsonlonteamai-reportsscoredAtwins per scorer, rubric, subject and judge kind.Labels never enter the run or its record. Workflow Runs shows
{scorer, label, passed}per pass and a scoring total kept apart from the run's own tokens.What else is in this change
The evidence directory holds
prompt.md,diff.patch,repo/(tracked files at the judged commit), andtranscripts/with subagents. Every file is redacted, including values of the team's declared secrets inenv/secrets.yaml. Rubric, request and raw judge files that no attempt reads any more are swept with the evidence.A label the scorer does not list is refused, and the lease stays open.
workflow reportrefuses a judge lease.Isolation: the team's
scoring.isolation, a scorer'sisolation, and the member'sscoring.isolationin the local config can each ask forverified. The strictest wins.teamai doctorprints the level, what sets it, and each judge tool on this machine with whether it runsverified. The local judge finds each tool's login where the rest of TeamAI finds that tool's home.Local judge profiles: Claude Code, Codex and OpenCode reach
verifiedwithreadaccess on the versions measured. OpenCode reaches it only outside a git or hg checkout. Any other version, and CodeBuddy, isbest-effort.Batching: the scorers of one pass that resolve to the same tool and model share one lease and one judge call.
teamai pullcaches valid scorers and warns about the rest, andteamai statuslists them.teamai review --branchfails on a malformed scorer, an unknown name inscorers:, or a credential in a scorer's text, as it does for a rule.A failed label prints the same hint a failed session label prints: ask your agent to read the reasoning file and share it with the team.
A scorer's
modelis a team model alias frommodels/aliases.yaml, resolved per judge tool like a workflow step's, or a model name. The label records the judge the way a pass records its agent:teamai doctornotes when no installed judge tool maps a scorer's alias, or when a member's~entry ormodels switchmakes the tool default judge.teamai review --branchfails on a malformedmodels/aliases.yamland on an alias that no tool maps in an agent, a workflow step or a scorer, judged from the team files only, for each role and project the branch's manifests declare. Adefaults.modelbehind an agent with no model is checked, and an agent'stool_extras.<tool>.modelsatisfies that tool.Docs:
docs/guide/scorers.mdand its Chinese page, with pointers from the workflow pages.skill-data/core/references/commands.mdis regenerated.An older teamai does not know the
scorerskey. Update every member and runner before a team adds it to a step.Out of scope
scorers:in a role or project needs feat(push): support all namespaced resource types #1234. Until feat(push): support all namespaced resource types #1234 lands, that key is ignored with a warning, and a team without roles or projects gets every namespace.How it was verified
npm run build,npx tsc --noEmit,npm run lint, and the tests each commit touchesnpx vitest runat the head (6a167a32), on #1280 ate70a33394d9175f0,review --branchon a branch whoseopsscorer asks forfast, mapped only inmodels/ops/aliases.yaml4d9175f0: fails it forno role or project, role dev, project svc-a, which do not receive ite70a3339(main with #1265)ALL_SUPPORTED_TOOLS. The judge's tool check andreview --branch's alias check now useisAgentToolIdandAGENT_TOOLSfromtools.ts, as main's own callers do (same set:ToolNameisAgentToolId). #1265's two new doctor tests mockisSilentlike the other doctor tests here. Not run with the real CLI.npm run test:e2e -- score-external-driver workflow-publish-resumeat #1298's head, which carries this one78a79737, an earlier head, afternpm run buildscoring.yamlper namespace,doctornaming the file that set the isolation,review --branchwith inherited model aliases and namespaced workflows, declared secrets in a scorer prompt, on-demand failures6fb5549bdoctorjudge tools,statusscorers, the failed-label hint92784ceb, model aliases58893b8a, the scoring path end to endAt the head, a scorer with
model: fast(mapped tohaikufor Claude) was judged by real Claude. The stored and published label carriesjudge.model {requested: haiku, reported: claude-haiku-5-5, alias: fast}. An external driver that submits without a model still gets{requested, alias}. A scorer alias no tool maps makesdoctorandscore --runprint the same text, and failsreview --branchalong with a step alias:The
58893b8arun used isolated HOMEs, bare git team repos, and real tools:--driver localscore --nextthenscore --leasejudged it with real Claude: a listed label,verifiedisolation,readaccess, published without reasoning. A phrase only the judge wrote is in the local reasoning file and nowhere in the run dir or onteamai-reports. Scoring left the run file unchanged.can.judgeand settled it with--submit --stdin.workflow report <judge ref>exits 1 withjudge_lease.samplingRate: 0,score --nextis idle andscore --run … --step … --scorer …still judges and publishes.verifiedandread.review --branchscorers/ops/review-quality.md. When activation switches between identical root and namespace files at one revision, the cache, the run's pin, andisolation_unmetall name the file in use.Codex was checked on an earlier commit of this branch, whose judge profile is unchanged since. Codex CLI 0.163.0 judged four leases whose transcripts and rubrics told it to write files, read outside the evidence, use the network, and spawn a subagent. Every label was
verifiedwithread, and no planted secret appeared in any output. Codex attempted none of those actions, so I ran each probe throughcodex sandboxwith the profile teamai generates. Writes, reads outside the evidence, and network access were all denied.Not run here: CodeBuddy (not installed) and the
gitlabandgithubproviders. That coverage is left to CI. While this PR's base is #1280's branch, CI runs only the review and the source scan here; Lint & Test and the E2E jobs run once #1280 merges and this PR targetsmain.Known follow-ups
best-effortuntil a signed-in run proves its profile.Commits
Slice 0 moves code only. Each later slice works on its own.