Repository navigation
Return AgentResult from BC-Bench agent harnesses - #1009
Merged
Sun Haoran (haoranpb) merged 1 commit intoNov 3, 2026
Merged
Conversation
The Copilot, Claude, PR Review and BCal harness functions now return the core AgentResult instead of a (metrics, experiment) tuple, and pipelines read it through AgentRunner. Copilot and Claude also return their final message as the response, which was previously discarded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the AgentResult migration.
No open issues 🎉
What changed in this PR
This PR adopts the harness-neutral AgentResult introduced in #1008 across BC-Bench’s four agent harnesses and eight evaluation pipelines, preparing for the later Harness[T] migration.
Changes:
- Replace
(metrics, experiment)return tuples withAgentResultand updateAgentRunnerand pipeline consumers. - Preserve Copilot and Claude final messages in
AgentResult.response. - Update mocks and tests to use the new result contract.
| File | Description |
|---|---|
tests/test_pr_review_agent.py |
Assert metrics and experiment fields on the returned result. |
tests/test_nl2al_pipeline.py |
Return AgentResult from the test runner. |
tests/test_evaluate_pipeline.py |
Adapt pipeline stubs and runners to the new contract. |
tests/test_copilot_agent.py |
Verify final response preservation and experiment access. |
tests/test_bcal_agent_provider.py |
Check metrics through the returned result. |
src/bcbench/evaluate/testgeneration.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/nl2al.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/ext_request_triage.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/ext_request_implement.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/ext_request_advisor.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/dataquery.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/codereview.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/bugfix.py |
Read metrics and experiment from AgentResult. |
src/bcbench/evaluate/base.py |
Change the AgentRunner protocol to return AgentResult. |
src/bcbench/commands/evaluate.py |
Return an empty AgentResult for mock evaluation. |
src/bcbench/agent/pr_review/agent.py |
Wrap review metrics and experiment in AgentResult. |
src/bcbench/agent/copilot/agent.py |
Return response, metrics, and experiment in AgentResult. |
src/bcbench/agent/claude/agent.py |
Return response, metrics, and experiment in AgentResult. |
src/bcbench/agent/bcal/agent.py |
Wrap execution metrics and experiment in AgentResult. |
Review effort: Balanced
Contributor
Code coverage overviewLanguages: Python Python / code-coverage/pytest
View coverage of the 10 most impacted files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| class AgentRunner[E: BaseDatasetEntry](Protocol): | ||
| def __call__(self, context: EvaluationContext[E], /) -> tuple[AgentMetrics | None, ExperimentConfiguration | None]: ... | ||
| def __call__(self, context: EvaluationContext[E], /) -> AgentResult: ... |
Collaborator
Author
|
Folded into #1008 (its commit is now part of that branch). |
An error occurred while trying to automatically change base from
core/harness-02-contract
to
main
November 3, 2026 14:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1008.
First BC-Bench adoption step: the Copilot, Claude, PR Review and BCal harness functions return the core
AgentResultinstead of a(metrics, experiment)tuple, andAgentRunner/ the 8 pipelines read it. Copilot and Claude now also return their final message asresponse(previously discarded).Next in the stack: harness classes implementing
Harness[T](withidentity()replacing the per-harnessget_*_version()calls in the CLI), then BCal transcripts and bc-bench-internal adoption.