Classify agent operations through existing telemetry - #10072
Open
JianW (v1212) wants to merge 3 commits into
Open
JianW (v1212) wants to merge 3 commits into
JianW (v1212) wants to merge 3 commits into
Conversation
|
Azure Pipelines: Successfully started running 3 pipeline(s). 20 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
github-actions
Bot
added
ext-agents
Contributor
Author
|
Implementation and local validation completed at 5efbea8. Existing telemetry.go/context contract, host fields/TelemetryService/exporter, dependencies, command return values and business control flow remain unchanged; only additive bounded classification markers and query/docs/tests were introduced. Full extension tests, vet, build, repeated marker/failure tests and core event-field contract tests passed. CI spelling issue was fixed using document-scoped KQL keywords; latest checks currently have no failures (platform/live jobs still running). Official-registry ingestion, query-engine execution and extension privacy review remain pending and are not claimed complete. Core azd init and pre-classification failures are explicitly unknown in the supplied query; this is observed command-level statistics, not exact per-agent outcomes. |
Contributor
Author
ZIP regression against the #9890 bug bash plan — 5efbea8Built a separate Windows ZIP from this clean PR revision (1.0.0-beta.15.pr.10072.5efbea80), installed the extracted artifact in isolated azd configuration, and ran the existing plan serially.
These runs disabled telemetry upload and used a nonofficial ZIP; they demonstrate functional regression coverage, not production ingestion or KQL validation. Fake-host telemetry tests passed three repeated runs. Actual PSTN calls and manual Portal/browser listening remain unexecuted. Current CI: 65 passed / 1 skipped. No blanket full-plan acceptance claim. |
JianW (v1212)
marked this pull request as ready for review
September 17, 2026 07:18
JianW (v1212)
requested review from
JeffreyCA,
Rick Winter (RickWinter),
Glenn Harper (glharper),
Marina He (hemarina),
Hui Miao (huimiu),
Zhijie Huang (hund030),
Wei Meng (m5i-work),
Richard Park (richardpark-msft),
tg-msft,
John Miller (therealjohn),
Travis Angevine (trangevi) and
Victor Vazquez (vhvb1989)
as code owners
September 17, 2026 07:18
|
Azure Pipelines: Successfully started running 3 pipeline(s). 20 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Legacy configuration precedence, cancellation counting, and the authoritative telemetry inventory need correction.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds bounded agent-operation classification to existing extension telemetry, with supporting tests, privacy documentation, and KQL analysis.
Changes:
- Classifies init, provision, and deploy operations.
- Reports deduplicated markers through
ext.usage. - Adds telemetry documentation, tests, and query fixtures.
File summaries
| File | Description |
|---|---|
docs/specs/metrics-audit/telemetry-schema.md |
Documents marker classification. |
docs/specs/metrics-audit/privacy-review-checklist.md |
Records privacy-review scope. |
docs/specs/metrics-audit/feature-telemetry-matrix.md |
Describes operation coverage. |
docs/reference/telemetry-data.md |
Adds the public event contract. |
cli/azd/extensions/azure.ai.agents/README.md |
Links operation statistics. |
cli/azd/extensions/azure.ai.agents/internal/telemetry/operation.go |
Implements bounded classification. |
cli/azd/extensions/azure.ai.agents/internal/telemetry/operation_test.go |
Tests classification vocabulary. |
cli/azd/extensions/azure.ai.agents/internal/cmd/telemetry_operation.go |
Collects and reports markers. |
cli/azd/extensions/azure.ai.agents/internal/cmd/telemetry_operation_test.go |
Tests reporting and deduplication. |
cli/azd/extensions/azure.ai.agents/internal/cmd/listen.go |
Instruments lifecycle hooks. |
cli/azd/extensions/azure.ai.agents/internal/cmd/init.go |
Tracks init intent. |
cli/azd/extensions/azure.ai.agents/internal/cmd/init_reuse_project_agent.go |
Retains reused project configuration. |
cli/azd/extensions/azure.ai.agents/internal/cmd/init_managed.go |
Classifies managed prompt init. |
cli/azd/extensions/azure.ai.agents/internal/cmd/init_from_code.go |
Classifies code-based init. |
cli/azd/extensions/azure.ai.agents/internal/cmd/init_from_code_reuse.go |
Classifies reused definitions. |
cli/azd/extensions/azure.ai.agents/internal/cmd/init_adopt.go |
Classifies adopted projects. |
cli/azd/extensions/azure.ai.agents/docs/operation-telemetry.md |
Defines analysis and KQL. |
cli/azd/extensions/azure.ai.agents/docs/operation-telemetry-fixture.kql |
Adds synthetic query data. |
cli/azd/cmd/telemetry_test.go |
Asserts reused host constants. |
Review details
- Files reviewed: 19/19 changed files
- Comments generated: 3
- Review effort level: Balanced (auto)
Note
Copilot is running an experiment and ran this review at Balanced.
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The supplied query mixes pre-rollout completions into marker coverage, and documentation has coverage-inventory gaps.
Review details
Suppressed comments (3)
Previously missed (3) — in code that hasn't changed since the last review.
cli/azd/extensions/azure.ai.agents/docs/operation-telemetry.md:81
- The sample does not apply the fixed extension/host cohort required above. After rollout, completions from older
azure.ai.agentsversions in this seven-day window have no marker and are counted asno_marker, skewing coverage and rates. Filter bothMarkersandCompletedto the same released cohort (usingextension.versionfor usage/extension spans andextension.installedfor core spans) before joining.
cli/azd/extensions/azure.ai.agents/README.md:47 - The README's event inventory at lines 26–28 still says the extension emits only agent-context and local-client-route events. Add this operation marker to that list so the new section does not leave the introductory inventory stale.
cli/azd/extensions/azure.ai.agents/docs/operation-telemetry.md:34 - Lifecycle classification also returns
unknownwhen the service kind exists only in the conventional on-diskagent.yaml/agent.yml, becauseoperationServiceClassexamines only the in-memory service properties. Include this common file-backed shape in the coverage limits so analysts do not treat those rows as classifiable agent traffic.
- Files reviewed: 19/19 changed files
- Comments generated: 0 new
- Review effort level: Balanced (auto)
Note
Copilot is running an experiment and ran this review at Balanced.
Contributor
azure.ai.agents PR buildNote This is an unsigned development build. Install it only if you trust this PR. Install the extension: azd ext install "https://azuresdkartifacts.z5.web.core.windows.net/azd/extensions/pr/10072/azure-ai-agents.zip"
|
Contributor
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent.operation.v1.<operation>.<category>.<telephony>values to existingextension.eventonext.usage. No new attributes, core span names, ingestion changes, dependencies or version bumps.agent.context.resolvedevents, fields and deduplication remain unchanged.Compatibility and scope
Business prompts, validation, deployment requests and original errors/results remain unchanged. New reporting uses the existing bounded best-effort reporter without retries, additional project/file/Azure reads or customer values. Provider registration remains safe without a client.
Categories: hosted, hosted_invocations_ws, prompt, workflow, voice_managed, voice_byom, voice_hosted_wrapper, unknown. Telephony records configured presence only, without identifiers or provider secrets.
Explicit limits
Validation at 5efbea8
go test ./... -count=1 -timeout 8m,go vet ./...,go fix -diff ./..., build and whitespace checks passed.Telemetry Change Checklist
Fields and events
Privacy
Testing and downstream
Tracking
Closes #10077. Related to #9395 and #9230, whose broader downstream/field work is outside this PR.
Latest review follow-up — 3a9c8e5
Reused the existing ServiceConfigProps property precedence; added ext.user.cancelled to the query/fixture; recorded the marker in the authoritative event-contract table. Full local extension tests, vet, build and spelling passed. New CI is required for this head. The previously executed bug bash ZIP remains pinned to 5efbea8, not silently replaced.