fix(evals): update Azure Monitor exporter and eval models - #10082
Conversation
Resolve patched OpenTelemetry core through exporter beta.44 without overrides. Replace Sonnet 4.6 and GPT-5.5 with Sonnet 5 and GPT-5.6 Terra. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 22 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Use GPT-5.6 Luna to judge the GPT-5.6 Terra scenario evals and document the resulting model assignments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a7f0e941-6668-4986-aec0-de499a4be9fa
9d43e91 to
c537d12
Compare
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 22 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
No approval-blocking issues were identified; the remaining documentation issue is a non-blocking nit.
Review effort: Lite (auto)
Findings: None
Note
Copilot is running an experiment and ran this review at Lite.
Resolved since last review (1)
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
Kristen Womack (kristenwomack)
left a comment
There was a problem hiding this comment.
LGTM
|
/check-enforcer override |

Fixes #10006
Summary
This PR removes a vulnerable nested copy of
@opentelemetry/core@2.0.0by upgrading Azure Monitor Exporter to1.0.0-beta.44. All consumers now resolve to patched core2.9.0, without overrides, new direct dependencies, or a Vally upgrade.It also replaces deprecated Claude Sonnet 4.6 with Claude Sonnet 5 and upgrades GPT-5.5 experiment selections to GPT-5.6 Terra. The five Terra scenario evals use GPT-5.6 Luna as their judge, including both experiment arms. The Sonnet starter eval uses Terra as its judge.
Behaviour at a glance
Experiment labels and baseline references are updated alongside the model selections.
1.0.0-beta.321.0.0-beta.442.0.0and2.9.02.9.0onlyclaude-sonnet-4.6claude-sonnet-5gpt-5.5gpt-5.6-terragpt-5.5gpt-5.6-lunagpt-5.5gpt-5.6-terraUsing a different judge avoids exact-model self-grading, but does not eliminate shared-family bias. Results before and after the model changes are not directly comparable.
Testing
Verified a clean dependency install on Node 24, confirmed every resolved OpenTelemetry core copy is patched, and checked Vally CLI startup. Checked exporter construction and shutdown with Vally's trace SDK without exporting spans.
Validated the model assignments and experiment baseline references.