Repository navigation
fix(anthropic): trace each call once when the SDK already traces it natively - #507
Merged
Merged
Conversation
…atively Anthropic SDK 1.14.0 and later emit their own span for every API call. When that span reaches the tracer provider AnthropicInstrumentor exports to (the client's open_telemetry tracer_provider, or a shared global provider), the instrumentor added a second span for the same call, with a different input token count (3 vs the SDK's cache-inclusive 12). The Messages create (sync and async) and count_tokens wrappers now leave the call to the SDK only in that case, read from the client's resolved public open_telemetry option at call time. When native tracing is disabled, goes to another provider, or has no provider, the instrumentor keeps its span, so its provider never loses the call. Sampling decisions are never used as a fallback signal. A one-time warning explains that TraceConfig and FI_HIDE_* do not apply to the SDK's spans and how to keep traceAI spans (open_telemetry=False). Adds offline ownership and loopback OTLP tests, and a CI matrix for anthropic 0.41.0 (declared floor), 1.14.0 (first native release) and 2.3.0 (latest). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ative-ownership Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the SDK's native span would reach the instrumentor's provider, the instrumentor deferred to it, and calls lost session.id, user.id, metadata, tag.tags, gen_ai.span.kind, input/output values and TraceConfig hiding. The instrumentor now keeps its span and runs that one call on a with_options(open_telemetry=False) copy of the client, so each call is still recorded once. The application's client is not changed. If the copy cannot be made, the instrumentor still traces the call and warns once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compares method, URL, headers and body sent by a client given the instrumentor's provider, with and without the instrumentor, sync and async. Notes in the README that async count_tokens keeps the SDK's span. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous revision ran each shared-provider Messages call on a per-call client copy with open_telemetry=False and rebound the SDK function to it. That made AnthropicVertex load and refresh Google credentials on every call (the copy, not the application's client, received them) and skipped any wrapt patch installed on Messages.create before the instrumentor. Now the original method runs unchanged on the application's client, through every other wrapper. While it runs, the SDK's span start (anthropic._base_client.start_api_call_span, which returns None for "no span") returns None for that client's tracer only, so calls on other clients inside it keep their spans. The hook is installed by instrument() and removed by uninstrument(); an SDK without it keeps the instrumentor span, may record a second span, and warns once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenTelemetry gives every client with the same tracer provider the same tracer, so while an instrumented Messages call runs, the hook turns off SDK spans for any call to that provider, not only the call's own client. Calls whose SDK spans go to another provider keep them. This corrects the previous commit message, which said calls on other clients keep their spans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Anthropic SDK 1.14.0 and later emit their own OpenTelemetry span for every API call. When that span reaches the same tracer provider
AnthropicInstrumentorexports to, eachmessages.createwas recorded twice: the SDK'santhropic.messages.createspan and the instrumentor'sMessagesspan. This happens when the client is givenopen_telemetry={"tracer_provider": provider}or when the Future AGI provider is the global provider (register(set_global_tracer_provider=True)).Messages.createandMessages.count_tokenswrappers keep the instrumentor's span and the SDK skips its own for that call. Each call is recorded once, and the span keeps what Future AGI reads:using_session/using_attributescontext (session.id,user.id,metadata,tag.tags, prompt template),gen_ai.span.kind,input.value/output.value, andTraceConfig/FI_HIDE_*.open_telemetryoption and compared by identity with the provider passed toinstrument()(or the global provider when none was passed). The application's client and itswith_options()copies keep their own option.Messages.create(wrapt patches from other instrumentations, APM agents or guardrails installed before this one still run), so lazily loaded client state such as AnthropicVertex Google credentials stays on that client and is loaded once. While the call runs, the SDK's span start (anthropic._base_client.start_api_call_span, whose contract is thatNonemeans "no span, the call goes on") returnsNonefor that client's tracer, set through aContextVar. OpenTelemetry gives clients with the same tracer provider the same tracer, so SDK calls to that provider made inside the instrumented call have no SDK span either; calls whose SDK spans go to another provider keep theirs.instrument()installs this hook anduninstrument()removes it. Present with the same contract in every SDK from 1.14.0 to 2.3.0 (all run, see Tests).open_telemetry=FalseorANTHROPIC_OPEN_TELEMETRY=false), sends to another provider, or has no provider (the documented non-globalregister()quickstart), nothing changes: each provider gets the call once.start_api_call_spanto hook, the instrumentor still traces the call; the SDK's span may then be a second one, and a one-time warning says so and points toopen_telemetry=False.Anthropic/AsyncAnthropic).AnthropicVertex/AsyncAnthropicVertexare tested with fakeload_auth/refresh_auth: credentials load and refresh once for three calls and stay on the application's client. Bedrock and Foundry clients use the samerequest()path and were not run separately.with_options(open_telemetry=False)copy and rebound the SDK function to it. That reloaded and refreshed Vertex credentials on every call (the copy received them) and skipped wrapt patches installed onMessages.createbefore the instrumentor. Both are fixed by the hook above; the copy is gone.Messages.createcalls (which this instrumentor traces), are unaffected.messages.stream()and asyncmessages.count_tokens(which the instrumentor has never wrapped, so they keep the SDK's span without Future AGI context) and SDKs before 1.14.0 are unchanged. No version, changelog or release-manifest edits; release-please owns the bump.Behavior to note: users who pass the Future AGI provider to the client and call
AnthropicInstrumentor().instrument()get the instrumentor'sMessagesspan, notanthropic.messages.create, forcreate/count_tokens(also withcontent_mode="content"). For those calls the SDK doesn't add its trace-context headers, as before SDK 1.14.0. Without the instrumentor, the SDK's native spans are untouched.Design choice (review round 1): an earlier revision of this PR left these calls to the SDK's span, which dropped the Future AGI context attributes and input/output for global-provider users. The alternatives were to disclose that loss, or to add context attributes to non-traceAI spans at the
fi_instrumentationprovider level (a separate package release, and it still would not restore input/output orFI_HIDE_*). Keeping the instrumentor's span restores exactly what these users had before SDK 1.14.0, without the duplicate.Reproduction (real SDK, synthetic transport,
using_attributes(session_id, user_id, metadata, tags))register()(docs quickstart)MessagesMessagesanthropic.messages.create(no context) +MessagesMessageswith contextanthropic.messages.create(no context) +MessagesMessageswith contextMessages(native goes to the other provider)open_telemetry=False, FI globalMessagesMessagesSame results on anthropic 1.14.0 and 2.3.0.
Tests
tests/test_native_ownership.py: the scenario matrix for sync/async create, rawstream=Trueandcount_tokens; Future AGI context (session.id,user.id,metadata,tag.tags,gen_ai.span.kind,input.value) on the single span for the global and client-passed provider;FI_HIDE_INPUTS/FI_HIDE_OUTPUTSapplied; disabled by option and by environment; sampling out the instrumentor span does not bring back the SDK's; application clients andwith_options()copies keep their option and parent context; partial streams and errors recorded once;messages.stream()keeps the SDK's span; an SDK without the span hook keeps the instrumentor and warns once;uninstrument()restores the SDK's span start; one-time INFO notice; the instrumented call sends the same method, URL, headers (including auth, default and extra headers) and body as the untraced client, sync and async; AnthropicVertex sync and async load Google credentials once and keep them on the application's client; a wrapt patch installed onMessages.create/AsyncMessages.create/Messages.count_tokensbefore the instrumentor runs once per call, shared provider or not; a call whose SDK span goes to another provider, made inside an instrumented call, keeps that span. The Vertex, wrapt and nested-call tests fail on the previous revision (8 failed) and pass now.tests/test_native_otlp.py: register() provider -> OTLP/HTTP -> loopback harness Receiver at/tracer/v1/traceswithX-Api-Key/X-Secret-Key; native route: one span withgen_ai.operation.name=chat, model, cache-inclusive usage and content in content mode, no content in metadata-only mode; instrumented route: oneMessagesspan with session, user, span kind, input and output.python/tests/test_framework_anthropic.pypasses on 0.41.0, 0.57.1, 1.14.0 and 2.3.0.anthropic-python-tests.ymlruns 0.41.0 (Python 3.10,httpx<0.28), 1.14.0 (3.10) and 2.3.0 (3.12) with outbound network disabled.release_plan.py --checkpasses. All local runs used an OS sandbox denying outbound network, and synthetic credentials. No live Anthropic or Future AGI calls.The Python docs change (native route, and how the instrumentor and native tracing combine) only uses released packages and is in a separate docs PR (future-agi/docs#1081).
🤖 Generated with Claude Code