Skip to content

fix(openai): capture transcription usage and stream lifecycle - #411

Open
nik13 wants to merge 3 commits into
devfrom
fix/openai-python-audio-usage
Open

nik13 wants to merge 3 commits into
devfrom
fix/openai-python-audio-usage

Conversation

@nik13

@nik13 nik13 commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Keep traceAI-openai: OpenAI Python does not expose a native GenAI OTel exporter that replaces this request instrumentor. Fix missing audio transcription usage through the existing OpenAIInstrumentor entry point rather than adding or deleting a package.

  • Extract API-reported input/output/total token usage from typed transcription JSON responses and terminal SSE events, including SDK-floor dict extras. Preserve zero counts; never convert duration billing or absent/malformed usage into token counts.
  • Add gen_ai.operation.name=transcription before the request so failures retain their operation and requested model.
  • Finish SSE spans on explicit sync/async close() as well as exhaustion and context exit. Retain only the terminal transcription event, not uploaded audio or a growing list of deltas. Existing context-manager error/cancellation behavior is preserved.
  • Keep prompts out of duplicate invocation parameters so input masking also works with the older locked fi-core. Retain output masking and provider attribution.
  • Document a precise boundary: Responses HTTP/SSE is supported; responses.connect() WebSockets is not instrumented, and the separate Realtime package is not a substitute. Raw streaming-response wrappers, binary speech-download lifetimes and text/SRT/VTT transcription formats are outside this audio extraction contract.

Evidence and design

On base d6a9966dd4d5d5fb344f7f5ab11b7483c2c2a6bc, a real OpenAI client over a fake HTTP transport returned transcription usage 5/2/7 but emitted no normalized usage attributes on both SDK 1.69.0 and 3.42.0. The initial audio suite failed 42 cases on current and 34 on floor (eight unsupported diarized-format skips).

Typed endpoint coverage follows the design demonstrated by Braintrust's explicit audio transcription integration; no vendor code was copied.

OpenAI's direct WebSocket path bypasses OpenAI.request. Two fake-WebSocket regressions exercise the real sync/async SDK connections and assert that the HTTP instrumentor emits no spans, matching the documented boundary rather than claiming a nonexistent adapter.

Verification

All local Python tests ran under the existing macOS sandbox denying outbound network except localhost. Vendor calls use httpx.MockTransport; audio tests export real OTLP/HTTP protobuf to a localhost receiver. No vendor/model API, real credentials or Cloud collector was used.

Full python/frameworks/openai/tests results:

OpenAI SDK fi-core Result
3.42.0 repository 1.2.2 114 passed
1.69.0 (package floor) repository 1.2.2 104 passed, 10 skipped
1.93.1 (CI lock) released 1.1.0; OTel 1.34.1 102 passed, 12 skipped

Floor skips cover unavailable diarized responses and Responses WebSockets. The locked-core run additionally skips the two echoed-error privacy cases because that core predates hide_error_details; input/output masking and all usage tests still run. The current-core tests verify echoed errors are hidden on the wire. Existing 62 chat, Responses, embeddings, provider and context-manager tests remain green.

Coverage includes sync/async, non-stream/stream, zero/missing/duration/malformed usage, early/empty context exit, explicit/repeated close, consumer errors/cancellation, HTTP errors, SSE API errors, and privacy checks against the serialized OTLP payload.

Re-run with dependencies installed, in a network-denied sandbox permitting localhost:

python -m pytest python/frameworks/openai/tests --noconftest -q -o addopts='' -p no:cacheprovider

Release and docs

No version or changelog bump; release-please owns the patch release. No package deletion, merge, tag or publish is part of this PR. A corrected website page is held until this package fix is on the registry; no docs PR claiming unreleased behavior has been opened. The package README includes the support boundary now. Python audio examples were executed offline at floor/current; the held MDX page compiles. No live screenshot or docs-site visual verification is claimed.

E2E coverage

not applicable (no e2e/ harness) · flows hit: n/a · marker: n/a · verdict: not applicable

Self-review: no introduced blocking finding. New tests and git diff --check pass; no Ruff configuration, migrations, frontend or contract-generation changes apply.

🤖 Generated with Claude Code

Preserve typed audio token usage at the SDK floor and current versions, close SSE spans deterministically, and scope Responses WebSocket support explicitly.

Co-Authored-By: Claude Code <noreply@anthropic.com>
nik13 added a commit that referenced this pull request Nov 2, 2026
Require fi-instrumentation-otel 1.2.3 for OpenAI and vLLM, test built wheels, and preserve messages while hiding every legacy invocation-parameter copy. Finalize aclose on both routes without duplicating PR #411's shared close methods.

Co-Authored-By: Claude Code <noreply@anthropic.com>
nik13 and others added 2 commits November 2, 2026 13:59
Preserve the audio transcription and gateway guides, and pin shared early-close tracing on the existing fix branch with a real-SDK regression at the supported floor and latest version.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Reconcile DeepSeek accumulator routing and explicit stream close.

Co-Authored-By: Claude Code <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant