Proposal: Semantic Conventions for Code-Level Runtime Telemetry and Software Engineering Agent Loops

[anirudha]

Status: Draft for otel SIG discussion
Author: Anirudha (OpenSearch Observability maintainer; OTel GenAI SIG contributor)
Part A target: open-telemetry/semantic-conventions (code and metrics areas), Semantic Conventions SIG
Part B target: open-telemetry/semantic-conventions-genai, GenAI SemConv and Instrumentations WG
Related: gen_ai.evaluation.result event (Development), in-flight gen_ai.experiment.* and test.case.* / test.suite.run.id evaluation proposals, code.* stability (core v1.33.0), vcs.* / deployment.* / cicd.* registries (Development), OpenSearch Observability Stack "Shift-Left Observability for AI Agents" RFC (downstream consumer)

---

1. Summary

AI agents now write, review, deploy, and repair code, and they increasingly consume telemetry directly to do it. Two gaps in the conventions block a vendor-neutral version of this loop:

1. Part A. There is no standard way to express function-level runtime behavior as metrics. Runtime sensors that record every invocation cannot emit a span per call, so they aggregate client-side, and today each one invents a proprietary schema for invocation counts, durations, errors, and caller edges.
2. Part B. There is no guidance for telemetry produced by software engineering (SWE) agents: how an agent run is tied to the repository, commit, and pull request it acts on, how a remediation trace links to the production failure that triggered it, and how an evaluation of a deployed fix is expressed.

The proposal is deliberately small. Part A adds one attribute and three metric instruments to the core conventions, all Development. Part B adds no new operation values; it adds one normative composition and linking document, and extends the existing gen_ai.evaluation.result event with two conditionally required attributes drawn from the core registry.

2. Motivation

The detect, fix, and verify loop is converging across the industry: production runtime data flows to a coding agent through MCP, the agent checks a pull request against live behavior of the functions it touches, a detected issue becomes a root-caused pull request, and the deployed fix is verified against production. Every current implementation of this loop uses proprietary names for the same concepts. That blocks portability in exactly the way gen_ai.request.model was designed to prevent for inference: a dashboard, a CI gate, or an open-source backend built against one sensor's schema reads nothing from the next one.

The concepts themselves are not vendor-specific. "How often is this function called, how long does it take, how often does it fail, and who calls it" is as generic as an HTTP request duration. "This agent run acted on this pull request and its fix was verified in production" is expressible entirely with attributes that already exist in the registries, provided the conventions say how to compose them.

Downstream consumers are ready. Open-source observability backends, CI actions, and evaluation frameworks (DeepEval, Ragas, Strands, and framework instrumentations discussed in the GenAI SIG) need these names to interoperate without translation layers.

3. What exists and is reused

This proposal introduces the minimum missing surface and reuses everything else:

Area	Registry	Status	Role here
code.function.name, code.file.path, code.line.number, code.column.number, code.stacktrace	core	Stable (v1.33.0)	Identity of the code unit; code.function.name is fully qualified
vcs.repository.url.full, vcs.ref.head.name, vcs.ref.head.revision, vcs.change.id, vcs.change.state	core	Development	Repository, branch, commit, and pull request identity
deployment.environment.name, deployment.id, deployment.name, deployment.status	core	Development (environment.name widely adopted)	Deployment cohorts and markers
cicd.pipeline.name, cicd.pipeline.run.id	core	Development	CI provenance of gates and agent runs
gen_ai.operation.name (invoke_agent, plan, execute_tool, invoke_workflow), gen_ai.agent.*, gen_ai.tool.*, gen_ai.usage.*	genai	Development	The agent run itself; no new values needed
gen_ai.evaluation.result event (gen_ai.evaluation.name, .score.value, .score.label, .explanation, gen_ai.response.id, error.type)	genai	Development	Verification outcomes; extended in Part B
error.type, exception.*, service.*	core	Stable	Errors and identity

---

Part A: Code-level runtime telemetry (core semantic conventions)

A.1 New attribute: code.function.caller.name

Attribute	Type	Stability	Description	Examples
code.function.caller.name	string	Development	The fully qualified name of the immediate caller of the code unit identified by code.function.name, following the same format rules as code.function.name.	com.example.api.CheckoutController.post; checkout.applyDiscount

Rationale. On spans, the caller is derivable from parentage. On metrics there is no parent, and sensors that observe every invocation aggregate client-side precisely because a span per call is not viable. The caller attribute preserves the call-graph edge that makes function metrics actionable: an agent evaluating a change to applyDiscount needs to know that CheckoutController.post is the dominant caller before it can reason about blast radius. Emitters MAY also set the attribute on spans and log records when the caller is known without a parent span (for example, sampled in-process stack capture).

The format intentionally mirrors code.function.name (fully qualified, no arguments), so the two attributes join directly. Short, unqualified values such as handler or run are ambiguous; emitters that cannot produce a fully qualified caller SHOULD omit the attribute and convey the edge through backend-side relationship data instead.

A.2 New metric instruments

All Development. Resource identity (service.*, deployment.environment.name, vcs.ref.head.revision) arrives through resource attributes as usual and is not repeated per instrument.

Name	Instrument	Unit	Description
code.function.call.duration	Histogram	s	Duration of invocations of a code unit.
code.function.errors	Counter	{error}	Invocations of a code unit that ended in an error.
code.function.calls	Counter	{call}	Invocations of a code unit. Authoritative traffic count; see note below.

Attributes on all three instruments:

Attribute	Requirement level	Notes
code.function.name	Required	Fully qualified
code.function.caller.name	Recommended	Instrumentations MUST provide a way to disable it for cardinality control
code.file.path	Opt-In	Usually derivable from the function name; adds cardinality
error.type	Required on code.function.errors; Conditionally Required on code.function.call.duration if the invocation errored	Low-cardinality error class per the error.type definition

Notes.

• Duration observations MAY be sampled by the emitter: production sensors commonly keep exact invocation counts while sampling duration measurements. The count of code.function.call.duration therefore MAY be lower than code.function.calls and MUST NOT be used as an invocation count. Consumers use code.function.calls for traffic and the histogram for latency statistics only. Emitters that record every duration MAY document that the two agree.
• Bucket boundaries: instrumentations SHOULD default to the standard duration boundaries used by http.server.request.duration and MAY add sub-millisecond buckets, since in-process calls are commonly faster than network requests. Final boundaries are an open question for the metrics area approvers.

A.3 Cardinality considerations

Function-level metrics are bounded by codebase size, not by traffic, which distinguishes them from user- or URL-derived attributes. A large service has thousands of functions, not millions. The conventions text will state:

1. Instrumentations MUST provide configuration to limit instrumented functions (allowlist, package filters, or a maximum function count) and MUST document the default.
2. Instrumentations SHOULD exclude third-party library internals by default and SHOULD collapse anonymous and generated functions to their enclosing named unit.
3. code.function.caller.name multiplies series by observed fan-in; the Recommended level plus a mandatory disable switch reflects that. Collectors MAY aggregate the caller dimension away in transit.

A.4 Interaction with existing signals

• Spans. Unchanged. Where both exist, code.function.name joins metrics to spans and exemplars can attach trace context to individual histogram samples.
• Profiling signal. Complementary, not competing. Profiles are sampled, stack-centric, and optimized for CPU and memory attribution; they do not provide per-invocation counts, error accounting, or unsampled latency distributions. Profile frames and these metrics meet at the fully qualified function name.
• Exceptions. Non-normative guidance: exception span events and error logs SHOULD carry code.* of the throwing frame when known, which is what lets a backend join an error fingerprint to the function metrics proposed here.

A.5 YAML sketch

groups:
  - id: registry.code.caller
    type: attribute_group
    brief: Caller relationship for code units.
    attributes:
      - id: code.function.caller.name
        type: string
        stability: development
        brief: >
          Fully qualified name of the immediate caller of the code unit
          identified by `code.function.name`, using the same format.
        examples: ['com.example.api.CheckoutController.post']

  - id: metric.code.function.call.duration
    type: metric
    metric_name: code.function.call.duration
    stability: development
    brief: Duration of invocations of a code unit.
    instrument: histogram
    unit: "s"
    attributes:
      - ref: code.function.name
        requirement_level: required
      - ref: code.function.caller.name
        requirement_level: recommended
      - ref: code.file.path
        requirement_level: opt_in
      - ref: error.type
        requirement_level:
          conditionally_required: If the invocation ended in an error.

---

Part B: Software engineering agent loops (GenAI semantic conventions)

B.1 No new operations

gen_ai.operation.name already covers the agent run shape: invoke_agent for the run, plan for decomposition, execute_tool for actions such as opening a pull request, invoke_workflow for multi-agent recipes. This proposal adds no enum values. What is missing is the binding between those spans and the software artifacts they act on, and the linking pattern for the loop.

B.2 Composition guidance for SWE agents (new document docs/gen-ai/gen-ai-swe-agents.md)

Normative guidance, all attributes referenced from the core registry:

Scenario	Attributes	Level
Agent run operates on a repository	vcs.repository.url.full, vcs.ref.head.name, vcs.ref.head.revision on the invoke_agent span	Recommended
Agent creates or updates a change (pull request)	vcs.change.id, vcs.change.state on the execute_tool span performing the action, and on the invoke_agent span once known	Conditionally Required when a change is created or updated
Agent reasons about or edits specific code	code.function.name and code.file.path on the relevant span	Recommended when a single code unit is the subject
Agent run executes inside CI	cicd.pipeline.name, cicd.pipeline.run.id	Recommended
Agent verifies a deployment	deployment.id, deployment.environment.name	Conditionally Required when a specific deployment is the subject

This is composition, not invention: the same pattern by which GenAI spans already reuse server.address and error.type.

B.3 Span links for the detect, fix, verify loop

The loop crosses traces by construction: the failure happens in the application's trace, the remediation happens in the agent's trace, the verification happens after a later deployment. The document in B.2 will specify:

1. A remediation invoke_agent span SHOULD carry a span link to the span containing the triggering exception or failure signal, when trace context for it is available.
2. A verification gen_ai.evaluation.result event SHOULD be parented to the remediation invoke_agent span when emitted by the same system, or SHOULD carry the correlation attributes in B.4 when emitted by a separate verification system.
3. When only a log record or an error fingerprint is available rather than trace context, instrumentations SHOULD record the fingerprint as an attribute on the remediation span using existing mechanisms rather than fabricating trace context.

Whether span links in this loop warrant a typed link attribute is left as an open question rather than proposed here.

B.4 Extension to the gen_ai.evaluation.result event: evaluating deployed changes

Today the event correlates to a single completion through parenting or gen_ai.response.id. Verification of an agent-produced fix evaluates a deployed change, not a completion. Proposed additions to the event's attribute table, both referencing the core registry:

Attribute	Requirement level	Description
vcs.change.id	Conditionally Required when the evaluation subject is a code change	The change (pull request) whose effect is being evaluated
deployment.id	Conditionally Required when the evaluation subject is a deployed change	The deployment in which the change was evaluated

Example: a post-deploy verifier emits gen_ai.evaluation.result with gen_ai.evaluation.name = production_verification, gen_ai.evaluation.score.label = pass, vcs.change.id = 4812, deployment.id = d-2291, parented to (or linked with) the remediation agent trace. This composes with the in-flight offline evaluation work (gen_ai.experiment.*, test.case.id, test.suite.run.id): those identify the subject in an offline experiment, these identify it in production.

B.5 Worked example

Trace 1 (application, production)
  span: POST /checkout                      status=Error
    event: exception  exception.type=TypeError
                      code.function.name=checkout.applyDiscount

Trace 2 (remediation agent)
  span: invoke_agent  gen_ai.agent.name=remediation
                      vcs.repository.url.full=https://github.com/acme/checkout
                      link -> Trace 1 exception span
    span: plan
    span: execute_tool  gen_ai.tool.name=create_pull_request
                        vcs.change.id=4812  vcs.change.state=open

Deployment marker: deployment.id=d-2291  vcs.change.id=4812

Event (verification): gen_ai.evaluation.result
    gen_ai.evaluation.name=production_verification
    gen_ai.evaluation.score.label=pass
    vcs.change.id=4812  deployment.id=d-2291
    parent: Trace 2 invoke_agent

---

4. Prior art and downstream consumers

Commercial runtime code sensors, error monitors, and AI debugging agents have each shipped this loop with proprietary schemas in the last year: function tables keyed by internal identifiers, SQL surfaces for coding agents, and bespoke pull request annotations. The concepts have therefore been production-validated at scale; only the names are fragmented. On the consuming side, the OpenSearch Observability Stack's shift-left RFC implements these conventions as proposed and commits to migrating to whatever names the SIGs accept, and the evaluation-framework integrations discussed in the GenAI SIG are natural emitters of the B.4 extension.

5. Privacy and security considerations

Function names and file paths reveal internal software structure and SHOULD be treated as internal telemetry; this is already true of code.* on spans and stack traces, and the caller attribute adds relationship information of the same sensitivity class. None of the Part A instruments carry payloads or arguments. In Part B, gen_ai.evaluation.explanation retains its existing content warnings; the added attributes are identifiers only.

6. Stability and migration

Everything proposed is Development. Part A is additive to the core registry and the metrics conventions; no existing attribute changes, so no schema-file transformations are needed. Part B is additive to a repository that is itself entirely Development and unreleased, which is the right window to land loop semantics before instrumentations fragment.

7. Alternatives considered

Alternative	Decision
Emit a span per function invocation	Rejected: volume makes it non-viable for every-invocation observation, which is the defining property of the use case
Rely on the profiling signal for function-level data	Rejected as a substitute: sampled and resource-centric, no invocation or error accounting; documented as complementary
Encode the caller as a metric name suffix or exemplar	Rejected: breaks aggregation and tooling; exemplars link samples to traces, they do not define series dimensions
Vendor namespaces (x.sensor.*) with backend translation	Rejected: recreates the pre-convention state this proposal exists to end
A new gen_ai event for agent actions on code	Not proposed: execute_tool spans plus core vcs.* attributes already express the action; a new event would duplicate them

8. Process and asks

1. File Part A as an issue plus PR against open-telemetry/semantic-conventions (code area for A.1, metrics area for A.2), and Part B as an issue plus PR against open-telemetry/semantic-conventions-genai with the new SWE-agent document and the event extension.
2. Discuss Part A in the Semantic Conventions SIG and Part B in the GenAI SemConv and Instrumentations WG; the author will bring both to the next scheduled meetings and requests area approver feedback on the instrument set, bucket boundaries, and cardinality language.
3. Prototype evidence will accompany the PRs: an OTLP-emitting reference instrumentation for the Part A metrics and a remediation-loop example emitting the Part B composition, both runnable against an open-source backend.

9. Open questions

1. Bucket boundary defaults for code.function.call.duration, and whether sub-millisecond boundaries should be normative.
2. Whether code.function.caller.name belongs in the stable code.* group's namespace as proposed or in a separate caller-relationship group covering future edges (callee, module).
3. Whether span links in the detect, fix, verify loop need a typed link attribute, or whether prose guidance is sufficient.
4. Whether the B.4 subject correlation generalizes into an explicit evaluation-subject concept alongside gen_ai.response.id, in coordination with the gen_ai.experiment.* proposal.
5. Whether deployment.id semantics need tightening (uniqueness scope) before being load-bearing for verification.
6. Whether the enclosing entry point (flow) belongs on the Part A instruments as a new code.flow.name and code.flow.type pair or through reuse of http.route, messaging.destination.name, and rpc.method per flow type. Regression attribution ("which function is responsible for this endpoint's slowdown") requires the dimension in some form.
7. Attribute naming for distinguishing caught from escaped errors on code.function.errors, in coordination with the current status of exception.escaped.