refactor(tracing)!: drop the Agentex span processor and own the Span model in lib

.stats.yml

@@ -1,4 +1,4 @@
 configured_endpoints: 75
-openapi_spec_url: https://storage.googleapis.com/stainless-sdk-openapi-specs/sgp/agentex-sdk-7acaeb315af90255109ae17afc71e32a8e5851bb8a956a2a284cb4d344dfab51.yml
-openapi_spec_hash: 3044e94b48d60311b6048e8df88e7552
+openapi_spec_url: https://storage.googleapis.com/stainless-sdk-openapi-specs/sgp/agentex-sdk-ee0c521f0612c31b874bd595b90cd9209545bab603983552a1a7a87f38ed931e.yml
+openapi_spec_hash: 917a1ffe9e353bed2740524dec786ed2
 config_hash: 593e89b291976a5e84e4c3c3f8324354

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### ⚠ BREAKING CHANGES
 
+* **tracing:** removed the Agentex-native span processor and its `AgentexTracingProcessorConfig` (the Agentex server is retiring its Postgres spans API), along with `Trace.get_span` / `Trace.list_spans` and their async twins. `SGPTracingProcessorConfig` is the only processor config and registering any other type raises `ValueError`. The in-memory `Span` is now `agentex.lib.types.tracing.Span` (also exported as `agentex.lib.core.tracing.Span`); the generated `agentex.types.span.Span` disappears with the next client generation. Its `to_dict()` / `to_json()` return the full JSON-mode dump rather than only the fields that were set.
 * **harness:** removed the deprecated bespoke LangGraph tracing handler `create_langgraph_tracing_handler` (and its `AgentexLangGraphTracingHandler` class) from the public `agentex.lib.adk` surface. Span tracing is now derived from the canonical `StreamTaskMessage*` stream by `UnifiedEmitter` — wrap your run in the harness `*Turn` and drive `UnifiedEmitter.yield_turn` / `auto_send_turn`. The `agentex init` templates were migrated accordingly.
 * **harness:** removed the deprecated bespoke Pydantic-AI tracing handler `create_pydantic_ai_tracing_handler` (and its `AgentexPydanticAITracingHandler` class) from the public `agentex.lib.adk` surface. Span tracing is now derived from the canonical `StreamTaskMessage*` stream by `UnifiedEmitter` — wrap your run in `PydanticAITurn` and drive `UnifiedEmitter.yield_turn` / `auto_send_turn`. The `agentex init` templates were migrated accordingly.
 * **harness:** each harness now exposes exactly `_<harness>_sync.py` + `_<harness>_turn.py` under `agentex.lib.adk._modules`. The OpenAI harness `OpenAITurn` and `convert_openai_to_agentex_events` moved to `agentex.lib.adk._modules._openai_turn` / `_openai_sync`; back-compat shims remain at `agentex.lib.adk.providers._modules.{openai_turn,sync_provider}` for one release. Public facade names (`stream_pydantic_ai_events`, `stream_langgraph_events`, `emit_langgraph_messages`, etc.) are unchanged.

examples/tutorials/10_async/10_temporal/020_state_machine/project/state_machines/deep_research.py

@@ -3,7 +3,7 @@
 
 from pydantic import BaseModel
 
-from agentex.types.span import Span
+from agentex.lib.types.tracing import Span
 from agentex.lib.sdk.state_machine import StateMachine
 
 

src/agentex/lib/adk/__init__.py

@@ -31,6 +31,9 @@
 
 # Data-source refs for lineage (SGP-6513); implementation lives in core.tracing
 from agentex.lib.core.tracing import lineage
+
+# Opt-in commit-SHA stamping (AGX1-969); implementation in core.tracing
+from agentex.lib.core.tracing import code_revision
 from agentex.lib.core.tracing.lineage import DataSourceRef, data_sources
 
 # Unified harness surface (AGX1-375)
@@ -73,6 +76,7 @@
     "TurnSpan",
     # Lineage data-source refs (SGP-6513)
     "lineage",
+    "code_revision",
     "DataSourceRef",
     "data_sources",
     # Checkpointing / LangGraph

src/agentex/lib/adk/_modules/tracing.py

@@ -11,7 +11,6 @@
 from temporalio.exceptions import ActivityError, TimeoutError as TemporalTimeoutError, is_cancelled_exception
 
 from agentex import AsyncAgentex  # noqa: F401
-from agentex.lib.adk.utils._modules.client import create_async_agentex_client
 from agentex.lib.core.services.adk.tracing import TracingService
 from agentex.lib.core.temporal.activities.activity_helpers import ActivityHelpers
 from agentex.lib.core.temporal.activities.adk.tracing_activities import (
@@ -22,7 +21,7 @@
 from agentex.lib.core.tracing.span_error import set_span_error
 from agentex.lib.core.tracing.tracer import AsyncTracer
 from agentex.lib.core.harness.types import TurnUsage
-from agentex.types.span import Span
+from agentex.lib.types.tracing import Span
 from agentex.lib.utils.logging import make_logger
 from agentex.lib.utils.model_utils import BaseModel
 from agentex.lib.utils.temporal import in_temporal_workflow
@@ -145,46 +144,17 @@ def __init__(self, tracing_service: TracingService | None = None):
 
         Args:
             tracing_service (Optional[TracingService]): Optional pre-configured tracing service.
-                If None, will be lazily created on first use so the httpx client is
-                bound to the correct running event loop.
+                If None, one is created on first use.
         """
         self._tracing_service_explicit = tracing_service
         self._tracing_service_lazy: TracingService | None = None
-        self._bound_loop_id: int | None = None
 
     @property
     def _tracing_service(self) -> TracingService:
         if self._tracing_service_explicit is not None:
             return self._tracing_service_explicit
-
-        import asyncio
-
-        # Determine the current event loop (if any).
-        try:
-            loop = asyncio.get_running_loop()
-            loop_id = id(loop)
-        except RuntimeError:
-            loop_id = None
-
-        # Re-create the underlying httpx client when the event loop changes
-        # (e.g. between HTTP requests in a sync ASGI server) to avoid
-        # "Event loop is closed" / "bound to a different event loop" errors.
-        if self._tracing_service_lazy is None or (loop_id is not None and loop_id != self._bound_loop_id):
-            import httpx
-
-            # Keepalive ON: connections are reused within a single event
-            # loop, eliminating the TLS-handshake-per-span penalty under
-            # load.  Cross-loop safety is preserved by rebuilding the
-            # client whenever loop_id changes (the conditional above).
-            agentex_client = create_async_agentex_client(
-                http_client=httpx.AsyncClient(
-                    limits=httpx.Limits(max_keepalive_connections=20),
-                ),
-            )
-            tracer = AsyncTracer(agentex_client)
-            self._tracing_service_lazy = TracingService(tracer=tracer)
-            self._bound_loop_id = loop_id
-
+        if self._tracing_service_lazy is None:
+            self._tracing_service_lazy = TracingService(tracer=AsyncTracer())
         return self._tracing_service_lazy
 
     @asynccontextmanager

src/agentex/lib/cli/debug/debug_handlers.py

@@ -16,6 +16,7 @@
     pass
 
 from agentex.lib.utils.logging import make_logger
+from agentex.lib.cli.utils.cli_utils import SUBPROCESS_STREAM_LIMIT
 
 from .debug_config import DebugConfig, resolve_debug_port
 
@@ -66,6 +67,7 @@ async def start_temporal_worker_debug(
         env=debug_env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 
@@ -119,6 +121,7 @@ async def start_acp_server_debug(
         env=debug_env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 

src/agentex/lib/cli/handlers/run_handlers.py

@@ -12,6 +12,7 @@
 from agentex.lib.cli.debug import DebugConfig, start_acp_server_debug, start_temporal_worker_debug
 from agentex.lib.utils.logging import make_logger
 from agentex.config.agent_manifest import AgentManifest
+from agentex.lib.cli.utils.cli_utils import SUBPROCESS_STREAM_LIMIT
 from agentex.lib.cli.utils.path_utils import (
     get_file_paths,
     calculate_uvicorn_target_for_local,
@@ -23,6 +24,11 @@
 logger = make_logger(__name__)
 console = Console()
 
+# How many consecutive unreadable lines to skip before giving up on the stream.
+# Skipping is only known-safe for the limit-overrun case; this bounds the damage
+# if some other error repeats without consuming anything.
+MAX_CONSECUTIVE_READ_ERRORS = 100
+
 
 class RunError(Exception):
     """An error occurred during agent run"""
@@ -215,6 +221,7 @@ async def start_acp_server(
         env=env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 
@@ -234,23 +241,68 @@ async def start_temporal_worker(
         env=env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 
 async def stream_process_output(process: asyncio.subprocess.Process, prefix: str):
-    """Stream process output with prefix"""
+    """Stream process output with prefix.
+
+    This loop is the only reader of the child's stdout pipe. If it ever stops
+    reading, the pipe fills and the child blocks forever inside ``write()``,
+    which presents as a silent freeze: 0% CPU, no further logs, no traceback.
+    So a single unreadable line must never end the loop.
+    """
     try:
         if process.stdout is None:
             return
+        consecutive_read_errors = 0
         while True:
-            line = await process.stdout.readline()
+            try:
+                line = await process.stdout.readline()
+            except ValueError as e:
+                # readline() raises ValueError when a line exceeds the stream limit.
+                # In *that* case it has already discarded the line and resumed the
+                # transport, so skipping it makes guaranteed progress. Any other
+                # ValueError carries no such guarantee, and retrying it forever would
+                # spin without draining. We cannot tell the two apart (readline
+                # flattens LimitOverrunError into a bare ValueError), so bound the
+                # retries and let the outer handler report the hang risk.
+                consecutive_read_errors += 1
+                if consecutive_read_errors > MAX_CONSECUTIVE_READ_ERRORS:
+                    raise
+                logger.warning(
+                    f"Skipping an unreadable line from {prefix}: {e!r} "
+                    f"(consecutive failure {consecutive_read_errors}/{MAX_CONSECUTIVE_READ_ERRORS}). "
+                    f"If this says the chunk exceeded the limit, raise limit= on this "
+                    f"process's create_subprocess_exec."
+                )
+                continue
+
+            consecutive_read_errors = 0
+
             if not line:
                 break
-            decoded_line = line.decode("utf-8").rstrip()
+
+            try:
+                decoded_line = line.decode("utf-8").rstrip()
+            except UnicodeDecodeError as e:
+                logger.warning(f"Dropped an undecodable log line from {prefix} ({e}).")
+                continue
+
             if decoded_line:  # Only print non-empty lines
                 console.print(f"[dim]{prefix}:[/dim] {decoded_line}")
     except Exception as e:
-        logger.debug(f"Output streaming ended for {prefix}: {e}")
+        # The escalation path, including for the re-raise above. Anything reaching
+        # here ends the loop, so the child is now at risk of blocking on a full pipe.
+        # Warning rather than debug: this used to be a debug() that make_logger could
+        # never emit, which is why three freezes produced no clue.
+        # CancelledError derives from BaseException, so the auto-reload path that
+        # cancels these tasks passes straight through and is unaffected.
+        logger.warning(
+            f"Output streaming for {prefix} stopped on {e!r}. "
+            f"Nothing is draining its stdout now, so {prefix} will hang once the pipe fills."
+        )
 
 
 async def run_agent(manifest_path: str, debug_config: "DebugConfig | None" = None):

src/agentex/lib/cli/utils/cli_utils.py

@@ -5,6 +5,18 @@
 
 console = Console()
 
+# asyncio's StreamReader defaults to 64 KiB, and a single log line above that makes
+# readline() raise. Agents legitimately emit large lines (serialized charts, payloads
+# echoed back by validation errors), so give the reader room before it has to drop one.
+#
+# Lives here rather than beside its users so that both the normal spawns in
+# cli/handlers/run_handlers.py and the debug spawns in cli/debug/debug_handlers.py can
+# import it: run_handlers imports cli.debug, so the constant cannot live in either one.
+# Keep the two in step. A subprocess left on the asyncio default overruns far more
+# easily, and enough consecutive overruns exhaust the reader's retry bound and stop it
+# draining, which is the deadlock the bound is there to avoid.
+SUBPROCESS_STREAM_LIMIT = 8 * 1024 * 1024
+
 
 def handle_questionary_cancellation(
     result: str | None, operation: str = "operation"

src/agentex/lib/core/services/adk/tracing.py

@@ -2,7 +2,7 @@
 
 from typing import Any
 
-from agentex.types.span import Span
+from agentex.lib.types.tracing import Span
 from agentex.lib.utils.logging import make_logger
 from agentex.lib.utils.temporal import heartbeat_if_in_workflow
 from agentex.lib.utils.model_utils import BaseModel

src/agentex/lib/core/temporal/activities/adk/tracing_activities.py

@@ -5,7 +5,7 @@
 
 from temporalio import activity
 
-from agentex.types.span import Span
+from agentex.lib.types.tracing import Span
 from agentex.lib.utils.logging import make_logger
 from agentex.lib.utils.model_utils import BaseModel
 from agentex.lib.core.services.adk.tracing import TracingService

src/agentex/lib/core/tracing/__init__.py

@@ -1,4 +1,4 @@
-from agentex.types.span import Span
+from agentex.lib.types.tracing import Span
 from agentex.lib.core.tracing.trace import Trace, AsyncTrace
 from agentex.lib.core.tracing.tracer import Tracer, AsyncTracer
 from agentex.lib.core.tracing.span_error import (

src/agentex/lib/core/tracing/code_revision.py

@@ -0,0 +1,105 @@
+"""Opt-in stamping of the agent's source commit onto its spans.
+
+Nothing is stamped until the agent calls :func:`enable`, mirroring the
+``lineage`` registry next door: a process-wide switch the agent sets once at
+import, rather than automatic behaviour every agent inherits. When enabled the
+resolved commit lands in span data under ``__commit_sha__`` and is searchable in
+the SGP Traces UI as ``__commit_sha__:<sha>``.
+
+This is deliberately separate from ``__agent_version__``, which is automatic and
+carries the deployed image tag verbatim ("image tag or git sha"). That tag is a
+real commit on some build paths but an ``<image-name>-<sha>`` composite (AWS
+ECR), ``latest``, or a hand-passed tag on others -- so a field named for a commit
+must not simply mirror it. Values that are not git object names are refused, and
+a field named ``__commit_sha__`` therefore only ever holds one.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+
+from agentex.lib.utils.logging import make_logger
+
+__all__ = ("COMMIT_SHA_KEY", "enable", "disable", "is_enabled", "commit_sha")
+
+logger = make_logger(__name__)
+
+COMMIT_SHA_KEY = "__commit_sha__"
+
+# A git object name: 40 hex for SHA-1, 64 for SHA-256, or an abbreviation down to
+# git's own 7-character minimum.
+_GIT_SHA_RE = re.compile(r"[0-9a-fA-F]{7,64}")
+
+_COMMIT_SHA_ENV = "AGENT_COMMIT_SHA"
+# Fallback only: automatic, and only usable when it happens to be SHA-shaped.
+_AGENT_VERSION_ENV = "AGENT_VERSION"
+
+# Resolved once at enable() rather than per span: the value is fixed for the
+# life of the process, and resolving eagerly means a bad value is reported at
+# startup instead of silently producing unstamped spans.
+_commit_sha: str | None = None
+
+
+def enable(commit_sha: str | None = None) -> None:
+    """Opt this process in to stamping ``__commit_sha__`` onto every span.
+
+    Value precedence: the explicit ``commit_sha`` argument, else
+    ``AGENT_COMMIT_SHA``, else ``AGENT_VERSION`` when the deployment happened to
+    set it to a bare commit SHA. A value that is not a git object name is
+    refused with a warning and leaves stamping off -- better an absent field
+    than one named for a commit that holds an image tag.
+    """
+    global _commit_sha
+
+    for value, source in (
+        (commit_sha, "the commit_sha argument"),
+        (os.environ.get(_COMMIT_SHA_ENV), _COMMIT_SHA_ENV),
+        (os.environ.get(_AGENT_VERSION_ENV), _AGENT_VERSION_ENV),
+    ):
+        candidate = (value or "").strip()
+        if not candidate:
+            continue
+        if _GIT_SHA_RE.fullmatch(candidate):
+            _commit_sha = candidate
+            logger.info("code revision stamping enabled from %s", source)
+            return
+        # An explicit argument or AGENT_COMMIT_SHA is a direct statement of
+        # intent, so a bad value there is worth surfacing. AGENT_VERSION is only
+        # a fallback and is expected to be a non-SHA tag much of the time, so
+        # falling through it quietly is correct, not a silent failure.
+        if source != _AGENT_VERSION_ENV:
+            logger.warning(
+                "%s=%r is not a git commit SHA; __commit_sha__ will not be stamped.",
+                source,
+                candidate,
+            )
+            _commit_sha = None
+            return
+
+    _commit_sha = None
+    logger.warning(
+        "code revision stamping was enabled but no commit SHA was found "
+        "(checked the commit_sha argument, %s, and %s); __commit_sha__ will not "
+        "be stamped. Set %s in the agent's environment -- e.g. bake it at build "
+        "time with a Dockerfile ARG/ENV.",
+        _COMMIT_SHA_ENV,
+        _AGENT_VERSION_ENV,
+        _COMMIT_SHA_ENV,
+    )
+
+
+def disable() -> None:
+    """Turn stamping back off (also used for test isolation)."""
+    global _commit_sha
+    _commit_sha = None
+
+
+def is_enabled() -> bool:
+    """Whether a commit SHA resolved and will be stamped."""
+    return _commit_sha is not None
+
+
+def commit_sha() -> str | None:
+    """The resolved commit SHA, or ``None`` when stamping is not enabled."""
+    return _commit_sha