[LittleChimera]

Related issue

N/A (Test / CI)

Summary

The Pytest (server-rest) lane is red on staging and on every round-3 PR. The two original failures were test drift, not product regressions: factory label writes, factory session creation and GitHub credentials in sandboxes all still work. Once those were fixed, a flake and a memory leak surfaced in the same lane. The leak was killing the runner.

• test_sessions_factory_labels (added in factory r3: Tier-0 hardening (push-ref allowlist, merge lockout, human origin, bundle fence, label audit) #135): the parametrized cases built their runner and person bearers at collection time, and those bearers have a 600 s TTL. Under xdist the lane reaches this module about 16 minutes after collection, so every case got 401 instead of the 403 it asserts. The fix mints the headers inside the test. I also added a positive case: a runner bearer (org: runner) still creates a session through POST /v1/sessions carrying exactly the factory labels _spawn_member sends (id, role, parent, supervisor, state, heartbeat_s, project).
• test_start_host_injects_ssh_and_env: this one has been failing since Sync5 (Sync5: merge upstream/main into staging #83), well before round 3. Sync5: merge upstream/main into staging #83 removed the github_token / github_login / ssh_authorized_keys parameters from start_host. GitHub tokens moved to the server-side credential broker (omnigent host runs configure_host_git / configure_host_gh at startup). The test now asserts that the launch carries no GitHub credentials. It makes no claim about SSH keys, because the broker does not handle them (see below).
• test_a_looping_sender_logs_one_refusal_per_recipient_per_minute (added in factory cockpit: read API and truthful supervision (snapshot/events/decisions/preflight, mail replay, quiet beats, preview keepalive + TTL, server decisions, reconcile) #123): this flake failed the lane on this PR. The lead and ic refusals were logged on the same FakeClock millisecond, and find_events breaks ts ties by an event id minted inside asyncio.to_thread, so under xdist load they came back in either order. The test now advances the clock 1 s before the ic post, well under the 60 s refusal interval.
• Runner killed at 91-97% (memory leak in omnigent/server/factory/live.py): since factory r3: telemetry core (P0 retention, claim_mismatch, PR facts, versions, spans, cost ledger, heal, read routes, export) #139/factory r3: projects through proposals, server-spawned PLs and dependency gating (R3-07) #140/factory r3: people (roster, grant mirroring, viewer block, attribution, presence, decision routing and escalation, access requests) #141, every server-rest job on every branch has died 14-19 minutes in. The symptom was "runner has received a shutdown signal", "operation was canceled", or the 30 min timeout, with no failing test and no common test in flight. announce_factory_changed parks a debounce task per (loop, factory) in a module-level dict and removed it only after the 0.5 s sleep. When a loop closes inside that window, which every async test that emits a factory event does, the task is cancelled and the entry stays. That entry pins the task, its loop, the app state, the stores and their engines, including the compiled migration statements. The cost was about 5 MB per factory test: locally tests/server/factory alone peaked at 5.8 GB across 4 workers and the whole lane at about 8 GB. A done callback now drops the entry however the timer ends. The lane peaks at about 4.3 GB locally (with more tests, since factory r3: work graph (factory_tasks as server truth, binder, graph computations, graph API, human task add/edit, sys_factory_task, PL playbook) (R3-09) #143/factory r3: ops — live 'now', fleet API, interventions on any agent, issues and detectors (R3-11) #144/factory r3: the Room (threads, routing, coalesced delivery, reply fallback, cards, sys_factory_chat, planner) (R3-08) #145 are included), and a 19-test factory slice runs in 40 s instead of 300 s. In production the loop never closes, so the only effect there is that a cancelled timer no longer lingers.

Factory flows I checked against the code:

• Spawn (sys_factory_spawn) sends POST /v1/sessions with factory labels. The factory r3: Tier-0 hardening (push-ref allowlist, merge lockout, human origin, bundle fence, label audit) #135 guard (_reject_runner_factory_labels) only covers the generic PATCH /v1/sessions/{id}, so spawn is unaffected. The new test covers this case.
• sys_factory_set and mutual supervision go through PATCH /v1/factory/sessions/{id}. That route is not guarded, and its tests pass.
• Supervisor/reconcile label writes happen on the server through the stores. No runner-side code writes omnigent.factory.* labels through the generic PATCH.

Out of scope, found in review (both predate this PR):

• SSH keys for VS Code Remote / SSHPiper. The credential broker did not replace SSH keys. Before Sync5: merge upstream/main into staging #83 the Kubernetes launcher wrote ~/.ssh/authorized_keys. Nothing in omnigent writes it on staging now, but the deployed SSHPiper config still reads the sandbox's client_authorized_keys. VS Code Remote through SSHPiper has probably been broken since Sync5. ssh_authorized_keys_setup_commands and resolve_sandbox_identity should stay until someone decides whether SSHPiper is still supported.
• Runner-origin session create with server-only factory labels. A runner bearer can POST /v1/sessions with server-only labels (e.g. omnigent.factory.engine_mode, def_version, role=root) and get 201. The R3-04 checks only guard the PATCH routes. sys_factory_spawn never sends these labels, so a follow-up can refuse them on runner-origin creates without breaking spawn.

Test Plan

• Reproduced the label-test failure locally with a scratch pytest plugin that moves the clock back 700 s during collection: 6 failed / 1 passed on staging, all pass with this change.
• Leak: a scratch plugin counted live objects after a 19-test factory slice. Before the fix it found 11 event loops, 11 engines and 1177 alembic Script objects retained, all rooted in live._pending. After the fix it found 0 / 0 / 0. test_a_loop_closed_inside_the_debounce_leaves_no_pending_timer fails without the fix and passes with it.
• Whole server-rest lane locally, -n 4 as in CI, rebased on staging 760d0ab: 6077 passed, peak worker RSS 4.3 GB (8 GB before). The only failures are local-env gaps (databricks / asgiref not installed).
• tests/server/factory, the labels, github-injection and tests/runner/test_factory_spawn.py modules on the final rebase: 1447 passed.
• Credential broker and sandboxes: tests/server/routes/test_host_credentials.py tests/test_git_credential_github.py tests/onboarding/sandboxes: 669 passed.
• pre-commit clean on the changed files (pyrefly reports only pre-existing errors in other files).

Demo

N/A

Type of change

• Bug fix
• Feature
• UI / frontend change
• Refactor / chore
• Docs
• Test / CI
• Breaking change

Test coverage

• Unit tests added / updated
• Integration tests added / updated
• E2E tests added / updated
• Manual verification completed
• Existing tests cover this change
• Not applicable

Coverage notes

github_sandbox_env and git_identity_env are unused since #83. ssh_authorized_keys_setup_commands and resolve_sandbox_identity are also unwired, but they are tied to the open SSHPiper question above, so this PR leaves them in place.

🤖 Generated with Claude Code