[LittleChimera]

Related issue

Closes #

Round-3 work package R3-05 (telemetry core). Spec: ROUND3-SPEC §14.1–§14.4 and the export part of §14.7. Builds on R3-03 (#137), whose tables and store methods it fills.

Summary

Stop losing data, and record durable, versioned facts that every metric and the loop engine can build on. Everything new lives in omnigent/server/factory/telemetry/; the hot files get small hooks.

P0: stop losing data (first commit)

• background/prune.py keeps every lifecycle event. Only message.*, notice.* and preview.health (after 14 days) and heartbeat.* (after 2 days) are pruned. The 2026-10-26 run's events would otherwise age out on 2026-11-09.
• register_pr judges a claim against the refreshed gates first, then stores claim_mismatch (and server_ready) in the pr.claim event.

PR facts. prs.apply_refresh stores GitHub's createdAt as opened_at and sets first_ready_at once; ready_at still means "latest".

Versions (telemetry/versions.py)

• After the bundle seed, the server upserts its platform version into factory_versions: image tag, server sha, each factory bundle's content sha, and the OMNIGENT_FACTORY_* knobs. Knob names that look like secrets are redacted, and credentials are stripped from URLs. Rows are deduped by content hash.
• Members carry omnigent.factory.version and omnigent.factory.bundle=<agent_id>/<sha12>:
  • at spawn: _spawn_member asks the new GET /v1/factory/agent/version;
  • at restart and relaunch: control.execute_control.

Spans (telemetry/spans.py, recorder.py, pr_rounds.py)

• write_span is the reusable writer for the R3-09 to R3-12 spans. It is idempotent on dedupe_key: a repeated write merges into the existing row. It handles set-once attrs, forced closes, reopen, fill-missing fields and named overwrites.
• A recorder on events.add_listener writes these spans:
  • project, plus stage episodes;
  • ready_wait, decision, task;
  • env and preview state episodes;
  • demo, incident (needs_human, issue.*), zero-length intervention (controls) and proposal.
• A hook in apply_refresh turns the marker, commit and check history the poller already fetches into PR spans:
  • one pr span;
  • one pr_round per head;
  • one gate per head per gate (checks, review, verify, demo) with attrs.first_pass, which is set once.
• The supervisor's child close puts the sub-agent's session usage on its task span. The span covers the whole session, so this is the sum of every run's delta.
• Spans carry version_id and the root's def_version.

Cost ledger (telemetry/cost.py)

• A hook in _record_daily_cost books every turn into factory_cost_daily. Each row has the display cost, tokens and turns, keyed by factory, project ('' for leads), role, agent, model and UTC day.
• The attribution comes from the session's labels, or its root's labels for sub-agents (spec role), through a 60 s cache.
• The two orchestration.py call sites now pass the turn's tokens and model.

Heal (telemetry/heal.py)

• Runs in the nightly prune loop, just before the prune.
• Rebuilds and closes spans from factory_projects, factory_prs, factory_decisions, factory_demos, the sessions (sub-agents whose task.done was lost) and the remaining events.
• Uses the live writers' dedupe keys, so a second run changes nothing. The first run backfills; it also fills first_ready_at.

Read routes (api/telemetry_read.py)

• All need READ on the root and apply the cockpit's hidden-project filter:
  • GET /v1/factory/factories/{fid}/spans?kind=&project=&pr_id=&task_id=&since=&until=&limit=
  • GET /v1/factory/prs/{id}/timeline
  • GET /v1/factory/factories/{fid}/cost?group_by=day|project|role|agent|model (comma-combinable)
  • GET /v1/factory/factories/{fid}/export?tables=spans,cost,decisions,prs,demos,events&format=ndjson|csv
• The export is streamed and never includes transcript content:
  • no mail text or excerpts;
  • no summaries of message, notice, chat or alert.failed events;
  • no agent notes, decision context or free-text answers.

Review fixes (hardening)

• Marker shas. Marker, evidence and demo shas are free text from GitHub. They are now accepted only as 7-40 hex and only add verdicts. A round opens only for a sha seen as the PR's head (the refresh's headRefOid, a gate.changed event, or an existing round), and short shas resolve to the round they abbreviate. A pasted sha256 in a marker can no longer overflow head_sha (40) or dedupe_key (96) on PostgreSQL, and intermediate-commit or typo shas no longer add rounds.
• Heal isolation. Each section, PR and replayed event heals on its own. A failure is logged, listed in HealReport.partial and skips only that part. Heal re-reads each PR row before healing it, and a span that a live writer closed after heal read the log keeps that close (no reopen until the next night).
• Export. decision.withdrawn, needs_human* and label.changed summaries are dropped, and so are free-text label values (only state, health, heartbeat and supervisor values are kept). CSV cells starting with = + - @ (or tab/CR) get a ' prefix. Sessions outside the factory now need READ in the export and the span/timeline routes, as in the cockpit's activity feed.
• Version labels. The root lead is stamped at factory start, and peer restarts through /v1/factory/messages restamp.
• Column widths. Span writes clip role, agent_name and actor to their columns, the ledger role is clipped to 16, and one formula computes a task span's start.

events.emit ──► recorder ──► factory_spans ◄── pr_rounds ◄── prs.apply_refresh (GitHub poll)
supervisor child close ──► task span cost          ▲
_record_daily_cost ──► cost.record ──► factory_cost_daily
startup (after seed) ──► factory_versions ──► member labels (spawn/restart/relaunch), span version_id
nightly: heal (tables + events ──► spans) then prune (chatter only)

Test Plan

• uv run --frozen pytest -q tests/server/factory tests/runner tests/examples tests/policies tests/test_factory_gates.py: all pass locally. Also ran the session-usage and cost suites (tests/server/routes/test_native_usage_attribution.py, test_usage_report.py, test_session_updates_ws.py, test_sessions_runner_relay.py, tests/runtime/policies/test_user_daily_cost_routing.py, tests/server/integration/test_sessions_endpoints.py, tests/server/test_app.py).
• New tests:
  • test_prune.py: on a seeded database (SQL and memory), a month-old lifecycle survives and the chatter is pruned.
  • test_agent_state_routes.py: pr.claim carries claim_mismatch.
  • test_telemetry_spans.py replays fixtures/live_run_2026_10_26.json, the live run as an event log, through the real poller and event log:
    • fix(web): keep the iOS sidebar rename field above the soft keyboard omnigent-ai/omnigent#9630: three heads, merged. Checks failed on head a (first_pass=false); review asked for changes on head b; head c passed every gate first time.
    • fix(codex-native): preserve rotated bridges on relaunch and native controls omnigent-ai/omnigent#9631: ready, lost READY to a new head, ready again: two ready_wait episodes.
    • It also checks the other span kinds, writer idempotency and the version stamps.
  • test_telemetry_heal.py:
    • healing twice yields identical rows, and healing live-recorded spans changes none of them;
    • the first heal backfills the same PR structure from events only;
    • a lost task.done is closed from the session;
    • the nightly loop heals before it prunes.
  • test_telemetry_cost.py: an IC turn goes to its project and role, a lead turn to project ''; also unpriced turns, the TTL cache, and the real native-usage path.
  • test_telemetry_versions.py: content-hash dedupe, knob redaction, member labels, the spawn route, restart stamping.
  • test_telemetry_routes.py: filters, the timeline, cost grouping. The export excludes transcript content and hides projects the viewer cannot read (NDJSON and CSV).
  • tests/runner/test_factory_spawn.py: spawn stamps the version and bundle labels.
  • Review fixes: a 64-hex, a typo and an intermediate-commit marker during the live replay add no round and no over-wide write (the store refuses widths the way PostgreSQL does); an oversized evidence sha does not stop heal; one failing PR skips only itself (report.partial); a close that landed after heal read the log is kept; export drops withdrawal reasons, blockers and free-text labels; CSV cells cannot run as formulas; outsider sessions need READ; the root lead and peer restarts get version labels; long role labels are clipped.
• pre-commit on the changed files: ruff and format pass; pyrefly reports no new errors (24 pre-existing elsewhere). Alembic still has one head (fz1a2b3c4d5e); this PR adds no migration.

Live acceptance (on omnigent.caffeinelabs.io after the image bump, temp-main factory):

• After 24 h, spans?kind=pr_round shows one row per head per PR, and its gate rows carry first_pass.
• cost?group_by=project,role,day sums to within 2% of the cockpit's subtree_cost, for turns after the deploy.
• New, restarted and relaunched members carry the version and bundle labels.
• Lifecycle events older than 14 days survive a prune.

Demo

N/A. Server-only: no UI in this package. The routes feed the round-3 Insights and Timeline pages.

Type of change

• Bug fix
• Feature
• UI / frontend change
• Refactor / chore
• Docs
• Test / CI
• Breaking change

Test coverage

• Unit tests added / updated
• Integration tests added / updated
• E2E tests added / updated
• Manual verification completed
• Existing tests cover this change
• Not applicable

Coverage notes

The live-run replay runs the real PR poller (fake GitHub), the real events.emit and the real apply_refresh on one fake clock; the heal and route tests reuse it. Live acceptance on the deployed temp-main factory follows the image bump.

Notes for reviewers:

• Version record timing. The platform version is recorded once per boot, right after the bundle seed in the lifespan, so a new image is never paired with the previous boot's bundles. This one call covers both "startup" and "bundle seed".
• Image tag and server sha. These come from OMNIGENT_IMAGE_TAG and OMNIGENT_SERVER_SHA, which the deployment does not set yet. Until it does, versions differ by bundles and knobs only.
• Cost turns. For native harnesses, turns counts usage reports that added cost or tokens, not prompt turns.

🤖 Generated with Claude Code