Skip to content

fix(anthropic): trace each call once when the SDK already traces it natively - #507

Merged
nik13 merged 7 commits into
devfrom
fix/anthropic-python-native-ownership
Nov 2, 2026
Merged

nik13 merged 7 commits into
devfrom
fix/anthropic-python-native-ownership

Conversation

@nik13

@nik13 nik13 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Anthropic SDK 1.14.0 and later emit their own OpenTelemetry span for every API call. When that span reaches the same tracer provider AnthropicInstrumentor exports to, each messages.create was recorded twice: the SDK's anthropic.messages.create span and the instrumentor's Messages span. This happens when the client is given open_telemetry={"tracer_provider": provider} or when the Future AGI provider is the global provider (register(set_global_tracer_provider=True)).

  • When the SDK's span would go to the instrumentor's own provider, the sync and async Messages.create and Messages.count_tokens wrappers keep the instrumentor's span and the SDK skips its own for that call. Each call is recorded once, and the span keeps what Future AGI reads: using_session / using_attributes context (session.id, user.id, metadata, tag.tags, prompt template), gen_ai.span.kind, input.value / output.value, and TraceConfig / FI_HIDE_*.
  • "Same provider" is read at call time from the client's resolved public open_telemetry option and compared by identity with the provider passed to instrument() (or the global provider when none was passed). The application's client and its with_options() copies keep their own option.
  • How the SDK's span is turned off: the original method runs unchanged on the application's client, through any other wrappers on Messages.create (wrapt patches from other instrumentations, APM agents or guardrails installed before this one still run), so lazily loaded client state such as AnthropicVertex Google credentials stays on that client and is loaded once. While the call runs, the SDK's span start (anthropic._base_client.start_api_call_span, whose contract is that None means "no span, the call goes on") returns None for that client's tracer, set through a ContextVar. OpenTelemetry gives clients with the same tracer provider the same tracer, so SDK calls to that provider made inside the instrumented call have no SDK span either; calls whose SDK spans go to another provider keep theirs. instrument() installs this hook and uninstrument() removes it. Present with the same contract in every SDK from 1.14.0 to 2.3.0 (all run, see Tests).
  • When native tracing is disabled (open_telemetry=False or ANTHROPIC_OPEN_TELEMETRY=false), sends to another provider, or has no provider (the documented non-global register() quickstart), nothing changes: each provider gets the call once.
  • If the option can't be read, or an SDK that traces natively has no start_api_call_span to hook, the instrumentor still traces the call; the SDK's span may then be a second one, and a one-time warning says so and points to open_telemetry=False.
  • The first per-call switch logs one INFO message per instrumentor.
  • Sent requests are identical with and without the instrumentor, apart from trace-context headers (Anthropic / AsyncAnthropic). AnthropicVertex / AsyncAnthropicVertex are tested with fake load_auth / refresh_auth: credentials load and refresh once for three calls and stay on the application's client. Bedrock and Foundry clients use the same request() path and were not run separately.
  • Review round 2: the previous revision ran each call on a per-call with_options(open_telemetry=False) copy and rebound the SDK function to it. That reloaded and refreshed Vertex credentials on every call (the copy received them) and skipped wrapt patches installed on Messages.create before the instrumentor. Both are fixed by the hook above; the copy is gone.
  • Limit: a non-Messages SDK call to the same tracer provider made from inside an instrumented call (for example by a wrapper that runs inside it, on any client) also has no SDK span. Calls whose SDK spans go to another provider, and nested Messages.create calls (which this instrumentor traces), are unaffected.
  • Completions, Protect, messages.stream() and async messages.count_tokens (which the instrumentor has never wrapped, so they keep the SDK's span without Future AGI context) and SDKs before 1.14.0 are unchanged. No version, changelog or release-manifest edits; release-please owns the bump.

Behavior to note: users who pass the Future AGI provider to the client and call AnthropicInstrumentor().instrument() get the instrumentor's Messages span, not anthropic.messages.create, for create / count_tokens (also with content_mode="content"). For those calls the SDK doesn't add its trace-context headers, as before SDK 1.14.0. Without the instrumentor, the SDK's native spans are untouched.

Design choice (review round 1): an earlier revision of this PR left these calls to the SDK's span, which dropped the Future AGI context attributes and input/output for global-provider users. The alternatives were to disclose that loss, or to add context attributes to non-traceAI spans at the fi_instrumentation provider level (a separate package release, and it still would not restore input/output or FI_HIDE_*). Keeping the instrumentor's span restores exactly what these users had before SDK 1.14.0, without the duplicate.

Reproduction (real SDK, synthetic transport, using_attributes(session_id, user_id, metadata, tags))

Scenario dev This PR
Non-global register() (docs quickstart) Messages Messages
FI provider is global anthropic.messages.create (no context) + Messages Messages with context
FI provider passed to client anthropic.messages.create (no context) + Messages Messages with context
Another vendor is global, FI non-global Messages (native goes to the other provider) unchanged
open_telemetry=False, FI global Messages Messages

Same results on anthropic 1.14.0 and 2.3.0.

Tests

  • tests/test_native_ownership.py: the scenario matrix for sync/async create, raw stream=True and count_tokens; Future AGI context (session.id, user.id, metadata, tag.tags, gen_ai.span.kind, input.value) on the single span for the global and client-passed provider; FI_HIDE_INPUTS / FI_HIDE_OUTPUTS applied; disabled by option and by environment; sampling out the instrumentor span does not bring back the SDK's; application clients and with_options() copies keep their option and parent context; partial streams and errors recorded once; messages.stream() keeps the SDK's span; an SDK without the span hook keeps the instrumentor and warns once; uninstrument() restores the SDK's span start; one-time INFO notice; the instrumented call sends the same method, URL, headers (including auth, default and extra headers) and body as the untraced client, sync and async; AnthropicVertex sync and async load Google credentials once and keep them on the application's client; a wrapt patch installed on Messages.create / AsyncMessages.create / Messages.count_tokens before the instrumentor runs once per call, shared provider or not; a call whose SDK span goes to another provider, made inside an instrumented call, keeps that span. The Vertex, wrapt and nested-call tests fail on the previous revision (8 failed) and pass now.
  • tests/test_native_otlp.py: register() provider -> OTLP/HTTP -> loopback harness Receiver at /tracer/v1/traces with X-Api-Key/X-Secret-Key; native route: one span with gen_ai.operation.name=chat, model, cache-inclusive usage and content in content mode, no content in metadata-only mode; instrumented route: one Messages span with session, user, span kind, input and output.
  • Package suite: anthropic 2.3.0 (latest) and 1.14.0, 173 passed / 2 skipped; 1.15.0, 1.16.0, 1.17.0, 2.0.0, 2.1.0 and 2.2.0, 173 passed / 2 skipped each; 0.41.0 (declared floor), 89 passed with the native tests skipped; locked 0.57.1, 93 passed. python/tests/test_framework_anthropic.py passes on 0.41.0, 0.57.1, 1.14.0 and 2.3.0.
  • anthropic-python-tests.yml runs 0.41.0 (Python 3.10, httpx<0.28), 1.14.0 (3.10) and 2.3.0 (3.12) with outbound network disabled.
  • release_plan.py --check passes. All local runs used an OS sandbox denying outbound network, and synthetic credentials. No live Anthropic or Future AGI calls.

The Python docs change (native route, and how the instrumentor and native tracing combine) only uses released packages and is in a separate docs PR (future-agi/docs#1081).

🤖 Generated with Claude Code

…atively

Anthropic SDK 1.14.0 and later emit their own span for every API call. When
that span reaches the tracer provider AnthropicInstrumentor exports to (the
client's open_telemetry tracer_provider, or a shared global provider), the
instrumentor added a second span for the same call, with a different input
token count (3 vs the SDK's cache-inclusive 12).

The Messages create (sync and async) and count_tokens wrappers now leave the
call to the SDK only in that case, read from the client's resolved public
open_telemetry option at call time. When native tracing is disabled, goes to
another provider, or has no provider, the instrumentor keeps its span, so its
provider never loses the call. Sampling decisions are never used as a fallback
signal. A one-time warning explains that TraceConfig and FI_HIDE_* do not apply
to the SDK's spans and how to keep traceAI spans (open_telemetry=False).

Adds offline ownership and loopback OTLP tests, and a CI matrix for anthropic
0.41.0 (declared floor), 1.14.0 (first native release) and 2.3.0 (latest).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 6 commits November 1, 2026 15:29
…ative-ownership

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the SDK's native span would reach the instrumentor's provider, the
instrumentor deferred to it, and calls lost session.id, user.id, metadata,
tag.tags, gen_ai.span.kind, input/output values and TraceConfig hiding.

The instrumentor now keeps its span and runs that one call on a
with_options(open_telemetry=False) copy of the client, so each call is
still recorded once. The application's client is not changed. If the copy
cannot be made, the instrumentor still traces the call and warns once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compares method, URL, headers and body sent by a client given the
instrumentor's provider, with and without the instrumentor, sync and
async. Notes in the README that async count_tokens keeps the SDK's span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous revision ran each shared-provider Messages call on a per-call
client copy with open_telemetry=False and rebound the SDK function to it.
That made AnthropicVertex load and refresh Google credentials on every call
(the copy, not the application's client, received them) and skipped any
wrapt patch installed on Messages.create before the instrumentor.

Now the original method runs unchanged on the application's client, through
every other wrapper. While it runs, the SDK's span start
(anthropic._base_client.start_api_call_span, which returns None for "no
span") returns None for that client's tracer only, so calls on other clients
inside it keep their spans. The hook is installed by instrument() and removed
by uninstrument(); an SDK without it keeps the instrumentor span, may record
a second span, and warns once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenTelemetry gives every client with the same tracer provider the same
tracer, so while an instrumented Messages call runs, the hook turns off SDK
spans for any call to that provider, not only the call's own client. Calls
whose SDK spans go to another provider keep them. This corrects the previous
commit message, which said calls on other clients keep their spans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13
nik13 merged commit 5ae2b9b into dev Nov 2, 2026
25 of 34 checks passed
@nik13
nik13 deleted the fix/anthropic-python-native-ownership branch November 2, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant