Stop sending every agent deploy to a deprecated command - #10042
Open
Mohamed Hessien (m7md7sien) wants to merge 2 commits into
Open
Mohamed Hessien (m7md7sien) wants to merge 2 commits into
Mohamed Hessien (m7md7sien) wants to merge 2 commits into
Conversation
The note printed after `azd up` invited the reader to set up evaluations with `azd ai agent eval generate`. That surface is deprecated -- the evaluations extension owns it now -- so following the suggestion landed on a command whose first act is to say it should not be used. It also said it unconditionally. A project whose azure.yaml already declares an `azure.ai.eval` service is already set up, and being told to set it up reads as though the deploy did not notice what is in the file it just read. The lookup is best effort: a project azd cannot be asked about is treated as having no eval service, which is the state the note was written for.
|
Azure Pipelines: 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Copilot started reviewing on behalf of
Mohamed Hessien (m7md7sien)
September 15, 2026 14:43
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The note advertises an unpublished command, and the service-detection path lacks direct coverage.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Updates agent deployment guidance to use the evaluations extension and suppress redundant setup instructions.
Changes:
- Detects existing
azure.ai.evalservices. - Replaces the deprecated evaluation command.
- Adds deploy-note tests.
File summaries
| File | Description |
|---|---|
service_target_agent.go |
Updates evaluation guidance and service detection. |
service_target_agent_test.go |
Adapts artifact tests to the new parameter. |
deploy_note_eval_test.go |
Tests deploy-note messaging. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Mohamed Hessien (m7md7sien)
added a commit
to m7md7sien/azure-dev
that referenced
this pull request
Sep 15, 2026
The deploy note pointing at the deprecated `azd ai agent eval generate` was fixed on this branch, but it is a change to `azure.ai.agents`, which builds and releases on its own cadence. Carried here it could only ship when this PR ships, and the bug-bash build that packaged this extension did not package that one -- so the deprecated suggestion kept reaching testers with the fix sitting merged in a branch beside them. It now lives in Azure#10042, against main, where it can ship on its own. Nothing outside this extension is left in this PR except the registration the extension needs: its schema host, CODEOWNERS entry, lint workflow, release pipeline and spelling dictionary.
| @@ -0,0 +1,55 @@ | |||
| // Copyright (c) Microsoft Corporation. All rights reserved. | |||
Contributor
Author
There was a problem hiding this comment.
Blocking comment till azd eval extension is released
Narrowed to the one thing this needs to do. The note now names `azd ai eval init` instead of the deprecated `azd ai agent eval generate`, and says it on every deploy as it always did. The earlier version also read azure.yaml and suppressed the suggestion when an `azure.ai.eval` service was already declared. That meant this extension knowing about another one's service host, an extra project read on the deploy path, and a parameter threaded through deployArtifacts and its four callers -- all to withhold a line that is harmless to read twice. Gone.
Copilot started reviewing on behalf of
Mohamed Hessien (m7md7sien)
September 15, 2026 18:19
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The note still invites projects with an existing evaluation service to set one up.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #10043
What
The note printed after an agent
azd upinvited the reader to set up evaluations withazd ai agent eval generate.That surface is deprecated — the evaluations extension owns it now — so following the printed suggestion lands on a command whose first act is to tell you it should not be used. It now names
azd ai eval init.That is the whole change: one command name, on a line that is printed exactly as often as it was before.
Why it stops there
An earlier revision of this PR also suppressed the suggestion when
azure.yamlalready declared anazure.ai.evalservice. That has been dropped deliberately.It is cheap to do —
p.projectServicesis already populated on the deploy path, so it costs no extra call — but it buys very little: the line is harmless to read twice, and a reader who has already wired evaluations up is not misled by being shown the command that wires them up. Against that it added a parameter threaded throughdeployArtifactsand its four callers, and made this extension match on another extension's service host by value. Not worth it for a suppressed sentence.So the note is unconditional, as it has always been.
Sequencing — this is why the PR is not ready to merge
azd ai eval initis not installable yet. The evaluations extension is #9500, which is still open and does not add a registry entry, so a released user following this note today would hit an unknown command.This PR should merge after the evaluations extension is published and registered. There is a blocking comment on the branch tracking that, and it should not be resolved until the registry entry exists.
Tests
deploy_note_eval_test.go—TestDeployNoteNamesTheSupportedEvalCommandasserts the note namesazd ai eval init, no longer advertisesazd ai agent eval, and still carries the invocation link.Full
azure.ai.agentssuite passes. Two failures ininternal/synthesis(TestARMTemplate_MatchesBicepBuild,TestExistingProjectARMTemplate_MatchesBicepBuild) reproduce on pristinemainwith these changes stashed — stale checked-in ARM templates, unrelated.