factory r3: telemetry core (P0 retention, claim_mismatch, PR facts, versions, spans, cost ledger, heal, read routes, export)

docs/FACTORY.md

@@ -258,6 +258,7 @@ if they stay silent the board is still right.
 | `GET /v1/factory/demos/{demo_id}/files/{file_id}` | human (READ on the PL) | Demo files with Range/206 (a ranged read of the store) and ETag; only images and video inline, never HTML or SVG. |
 | `POST /v1/factory/agent/decisions`, `POST …/agent/prs`, `PUT …/agent/envs`, `POST …/agent/demos` (multipart), `GET …/agent/charter` | runner (the tools below) | Agent reports. |
 | `GET /v1/factory/preflight` | human | GitHub connected, agents seeded, sandbox host, factory enabled; `default_repo`, `default_base` and `max_projects` for the start form. |
+| `GET /v1/factory/factories/{fid}/spans?kind=&project=&pr_id=&task_id=&since=`, `GET /v1/factory/prs/{id}/timeline`, `GET …/{fid}/cost?group_by=day,project,role,agent,model`, `GET …/{fid}/export?tables=&format=ndjson\|csv` | human (READ on the root; hidden projects dropped) | Telemetry (see below). The export streams and never carries transcript content. |
 
 ### Tools
 
@@ -507,6 +508,7 @@ transcript folds.
 | `OMNIGENT_FACTORY_PREVIEW_TOOLING_REF` | empty (recommended: `main`) | caffeinelabs/app ref the preview tooling is overlaid from onto PR heads that predate it; stored in new charters. Not the project's base branch. Empty: env_keeper uses `main`. |
 | `OMNIGENT_FACTORY_QUIET_BEATS`, `OMNIGENT_FACTORY_MAIL_REPLAY` | on | `0` turns off skipping unchanged heartbeats / replaying queued mail after a restart. |
 | `OMNIGENT_FACTORY_INVARIANTS` | on | `0` leaves the `factory_invariants` admin default policy unseeded (see Invariants). |
+| `OMNIGENT_IMAGE_TAG`, `OMNIGENT_SERVER_SHA` | unset | Image tag and commit recorded in the platform version (`factory_versions`). |
 
 ### Rollback
 
@@ -524,6 +526,42 @@ transcript folds.
   (caffeinelabs/app#9640). PR heads that include it use their own copy; only
   older heads read it from the overlay ref (`main`).
 
+## Telemetry
+
+Durable facts every metric builds on (`omnigent/server/factory/telemetry/`):
+
+- **Retention.** The nightly prune drops only `message.*`, `notice.*` and
+  `preview.health` after 14 days and `heartbeat.*` after 2; every lifecycle
+  event is kept. `pr.claim` stores `claim_mismatch`; PRs record GitHub's
+  `opened_at` and `first_ready_at` (`ready_at` stays the latest).
+- **Versions.** After the bundle seed the server upserts its platform version
+  (`factory_versions`: image tag, server sha, each factory bundle's content
+  sha, the `OMNIGENT_FACTORY_*` knobs with secrets redacted), deduped by
+  content hash. Members carry `omnigent.factory.version` and
+  `omnigent.factory.bundle=<agent_id>/<sha12>` from factory start (the root
+  lead), spawn, restart (cockpit or peer) and relaunch.
+- **Spans** (`factory_spans`, never pruned, idempotent on `dedupe_key`). An
+  event-log recorder writes `project`, `stage`, `ready_wait`, `decision`,
+  `task`, `env`, `preview`, `demo`, `incident`, `intervention` and `proposal`
+  spans; every GitHub refresh writes `pr`, one `pr_round` per head and one
+  `gate` per head and gate with `attrs.first_pass`; the supervisor puts a
+  sub-agent's cost on its `task` span. Spans carry the version id and the
+  root's `def_version`. Only a sha seen as the PR's head (`headRefOid`, a
+  `gate.changed` event) opens a round; marker shas other than 7-40 hex are
+  ignored, and a verdict on a sha without a round adds nothing.
+- **Cost ledger** (`factory_cost_daily`). Every turn's display cost and
+  tokens, booked by factory, project (`""` for the leads), role, agent, model
+  and UTC day from the session's labels or its root's.
+- **Heal.** The nightly loop rebuilds and closes spans from the PR, decision,
+  demo and project tables, the sessions and the remaining events before it
+  prunes; a second run changes nothing, and the first one backfills. Each
+  section, PR and replayed event heals on its own (a failure skips only it),
+  and a span a live writer closed after heal read the log keeps that close.
+- **Export.** Rows the cockpit hides are dropped (sessions outside the
+  factory need READ, as in the activity feed); summaries that quote agent
+  text (mail, reasons, blockers, free-text labels) and free-text data keys
+  are left out; CSV cells that start with `= + - @` are prefixed with `'`.
+
 ## Safety
 
 - **`factory_role_guard`** ([`policies/builtins/factory.py`](../omnigent/policies/builtins/factory.py))
@@ -709,6 +747,7 @@ with a managed sandbox launcher configured.
 | Tier-0 invariants | `omnigent/policies/builtins/factory_invariants.py`, `omnigent/server/factory/origin.py`, `custom_bundle.py`, `omnigent/server/bundles.py` | `tests/policies/test_factory_invariants.py`, `tests/server/factory/test_human_origin.py`, `tests/server/test_bundle_validation_factory.py` |
 | Org chart and badges | `omnigent/server/factory/org_chart.py`, `web/src/lib/factory.ts` | `tests/server/routes/test_child_sessions_include_factory.py`, web tests |
 | Base lane and demo mirror | `omnigent/server/factory/base.py`, `demo_mirror.py`, `omnigent/factory_gates.py` (`base` gate) | `tests/server/factory/test_base_watcher.py`, `test_demo_mirror.py`, `tests/test_factory_gates.py` |
+| Telemetry | `omnigent/server/factory/telemetry/`, `api/telemetry_read.py`, `background/prune.py` | `tests/server/factory/test_telemetry_*.py` (replays `fixtures/live_run_2026_10_26.json`), `test_prune.py` |
 | Bundles | `examples/factory/` | `tests/examples/`, `tests/server/factory/test_factory_agents_listed.py` |
 | Demo harness | `examples/factory/factory-pl/agents/demo/skills/factory-demo/harness/` | `tests/examples/test_factory_demo_harness.py` (runs `factory_demo_harness.test.mjs` under `node --test`); `node run.mjs --selftest` records real demos of a local fixture |
 | End to end | — | `tests/integration/test_factory_tree.py` (live server and runner, mock LLM) |

omnigent/runner/factory_dispatch.py

@@ -51,13 +51,15 @@
     DEMO_MAX_TOTAL_BYTES,
     ENV_STATES,
     HEALTH,
+    LABEL_BUNDLE,
     LABEL_HEARTBEAT_S,
     LABEL_ID,
     LABEL_PARENT,
     LABEL_PROJECT,
     LABEL_ROLE,
     LABEL_STATE,
     LABEL_SUPERVISOR,
+    LABEL_VERSION,
     MESSAGE_KINDS,
     MESSAGE_MAX_CHARS,
     PR_CLAIMS,
@@ -590,6 +592,26 @@ async def _resolve_builtin_agent(
     )
 
 
+async def _version_labels(
+    server_client: httpx.AsyncClient, caller: str, agent_id: str
+) -> dict[str, str]:
+    """The platform version and bundle labels for a new member; ``{}`` when unavailable."""
+    found = await _get_object(
+        server_client,
+        "/v1/factory/agent/version",
+        tool_name=SYS_FACTORY_SPAWN,
+        params={"from_session_id": caller, "agent_id": agent_id},
+    )
+    labels = found.get("labels") if isinstance(found, dict) else None
+    if not isinstance(labels, dict):
+        return {}
+    return {
+        key: value
+        for key, value in labels.items()
+        if key in (LABEL_VERSION, LABEL_BUNDLE) and isinstance(value, str) and value
+    }
+
+
 def _default_supervisor(
     role: str,
     caller: str,
@@ -811,6 +833,7 @@ async def _spawn_member(
     }
     if spawn.project:
         labels[LABEL_PROJECT] = spawn.project
+    labels.update(await _version_labels(server_client, caller, agent_id))
     create_body: _JsonObject = {
         "agent_id": agent_id,
         "host_type": "managed",

omnigent/server/app.py

@@ -1481,6 +1481,10 @@ async def _lifespan(
         # warmup blocks deleted here.
 
         _ensure_default_agents(agent_store, artifact_store, agent_cache)
+        # The factory's platform version (image, bundles just seeded, knobs).
+        from omnigent.server.factory.telemetry.versions import record_platform_version
+
+        await record_platform_version(app_inst.state, note="server start")
 
         # Populate the policy registry (builtins + user-configured
         # modules) so GET /v1/policy-registry serves the catalog.

omnigent/server/factory/__init__.py

@@ -106,9 +106,9 @@ async def start_factory_services(app_state: Any) -> Callable[[], Awaitable[None]
     Start the factory's background services for one server lifespan.
 
     The mailer always (replaying mail a previous process left undelivered);
-    the supervisor (and the status-edge notifier it reads) and the
-    auto-discovered ``background`` services only when
-    ``OMNIGENT_FACTORY_ENABLED=1``.
+    the telemetry recorders (spans and the cost ledger), the supervisor (and
+    the status-edge notifier it reads) and the auto-discovered ``background``
+    services only when ``OMNIGENT_FACTORY_ENABLED=1``.
 
     :param app_state: ``app.state`` of an app passed to
         :func:`register_factory_routes`.
@@ -131,11 +131,14 @@ async def _noop() -> None:
         await mailer.replay_undelivered()
     stop_supervision: Callable[[], Awaitable[None]] | None = None
     stop_background: Callable[[], Awaitable[None]] | None = None
+    stop_telemetry: Callable[[], None] | None = None
     unseed_invariants = _seed_invariants()
     if factory_enabled():
+        from omnigent.server.factory import telemetry
         from omnigent.server.factory.background import start_all
         from omnigent.server.factory.supervisor import start_supervision
 
+        stop_telemetry = telemetry.start(app_state)
         stop_supervision = start_supervision(app_state)
         started = ["supervisor"]
         stop_background = await start_all(app_state, started=started)
@@ -148,6 +151,8 @@ async def _stop() -> None:
             await stop_background()
         if stop_supervision is not None:
             await stop_supervision()
+        if stop_telemetry is not None:
+            stop_telemetry()
         await mailer.close()
         if getattr(app_state, MAILER_STATE_ATTR, None) is mailer:
             setattr(app_state, MAILER_STATE_ATTR, None)

omnigent/server/factory/api/agent_state.py

@@ -2,9 +2,10 @@
 
 ``sys_factory_ask`` opens decisions, ``sys_factory_pr`` registers PRs and
 claims, ``sys_factory_env`` reports dev envs and factory previews, and
-``sys_factory_spawn`` reads a project's stored charter. Callers authenticate
-as ``from_session_id`` like ``/v1/factory/messages``; the project always comes
-from the caller's root PL, so a member only writes its own project's records.
+``sys_factory_spawn`` reads a project's stored charter and the version labels
+it stamps on the new member. Callers authenticate as ``from_session_id`` like
+``/v1/factory/messages``; the project always comes from the caller's root PL,
+so a member only writes its own project's records.
 """
 
 from __future__ import annotations
@@ -52,6 +53,7 @@
 from omnigent.server.factory.charter import project_base_branch, render_pl_charter
 from omnigent.server.factory.demos import public_base_url, review_url
 from omnigent.server.factory.routes import FactoryRouteError
+from omnigent.server.factory.telemetry import versions
 from omnigent.stores.factory_store import DecisionNotOpen, norm_id
 
 router = APIRouter(prefix="/v1/factory/agent", tags=["factory"])
@@ -403,6 +405,12 @@ async def register_pr(request: Request, body: PrBody) -> dict[str, Any]:
         row = updated or row
     if created:
         await announce_new_pr(app_state, caller, row)
+    nudge.nudge("prs", row.id)
+    refreshed = await _await_refresh(caller, row.id)
+    if refreshed is not None:
+        row = refreshed
+    # Against the fresh gates, and stored with the claim (claim calibration).
+    claim_mismatch = body.claim == "ready_for_human_merge" and not row.ready
     if body.claim is not None or body.sentinel_verdict is not None:
         what = body.claim or f"sentinel {body.sentinel_verdict}"
         await events.emit(
@@ -420,12 +428,11 @@ async def register_pr(request: Request, body: PrBody) -> dict[str, Any]:
                 "sentinel_verdict": body.sentinel_verdict,
                 "head_sha": head_sha,
                 "note": note,
+                "claim_mismatch": claim_mismatch,
+                "server_ready": row.ready,
+                "refreshed": refreshed is not None,
             },
         )
-    nudge.nudge("prs", row.id)
-    refreshed = await _await_refresh(caller, row.id)
-    if refreshed is not None:
-        row = refreshed
     gates = dict(row.gates or {})
     project_base = await asyncio.to_thread(_project_base, caller, row.project_slug)
     base = public_base_url(str(request.base_url))
@@ -447,7 +454,7 @@ async def register_pr(request: Request, body: PrBody) -> dict[str, Any]:
         "gates": gates,
         "reasons": list(gates.get("reasons") or []),
         "refreshed": refreshed is not None,
-        "claim_mismatch": body.claim == "ready_for_human_merge" and not row.ready,
+        "claim_mismatch": claim_mismatch,
     }
     if not gates:
         reply["note"] = "gates come from the server's GitHub poller and are not computed yet"
@@ -745,3 +752,18 @@ async def get_charter(
             404, "charter_not_found", f"project {project!r} has no stored charter"
         )
     return {"project": row.slug, "text": render_pl_charter(row)}
+
+
+# ── Versions ───────────────────────────────────────────────────
+
+
+@router.get("/version")
+async def get_version_labels(
+    request: Request,
+    from_session_id: str = Query(...),
+    agent_id: str | None = Query(None),
+) -> dict[str, Any]:
+    """The version and bundle labels a member spawned now carries (``sys_factory_spawn``)."""
+    caller = await authorize_agent(request, from_session_id)
+    labels = await asyncio.to_thread(versions.labels_for_agent, caller.deps.agent_store, agent_id)
+    return {"version_id": versions.current_version_id(), "labels": labels}