Skip to content

feat(signal): run one production 15-check certification workflow - #963

Open
Kevin Cho (chokevin) wants to merge 42 commits into
Azure:mainfrom
chokevin:chokevin-upstream-mpijob-adapter
Open

Kevin Cho (chokevin) wants to merge 42 commits into
Azure:mainfrom
chokevin:chokevin-upstream-mpijob-adapter

Conversation

@chokevin

@chokevin Kevin Cho (chokevin) commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

One immutable CertificationRun reserves its participant set once and executes exactly 15 ordered checks through one serial Argo Workflow. Argo completion is not certification success: aggregation requires identity-bound task outputs and durable typed backend evidence.

  • Standalone Pod and MPIJob execution through one fixed adapter WorkflowTemplate; Kueue is intentionally deferred.
  • Immutable Profile, participant, script, and check snapshots with positional R01-R15 identity.
  • Signal-owned Node reservation and cordon. Success releases automatically; Failed, Inconclusive, and Canceled Runs remain cordoned until monotonic spec.release: true.
  • Detailed immutable evidence is persisted before controller teardown. Uncertain timed-out or partially created backends are captured and quiesced before later checks can start.
  • The public contract excludes generic Workflow wiring, retry/TTL knobs, arbitrary Kubernetes pass-through, and queue integration.
  • Production-only surface: no tracked Signal test files or Signal-specific CI job.

Evidence

Architecture overview: narrated 20-second fframes animation covering deployment prerequisites, the operator-owned Profile/scripts, Node request label, immutable admission, serial Argo tasks, and Pod/MPIJob execution.

signal-deployment-trigger-20s.mp4

This is a code-backed architecture animation, not a live execution recording. Deployment commands contain placeholders and were not executed. Signal does not install Argo, MPI Operator, operator scripts, runner images, or Profiles.

The earlier archival integration explainer showed 15 task nodes, 10 Pass and 5 Inconclusive, before the production-only reduction and lifecycle hardening. Neither video establishes successful certification of the current head. No new live Argo/MPI/GPU run or soak was performed.

Verification

Current head cd641dc76b58a1839d93f83de6aa69b7175e8b24:

  • Complete PR diff: 31 files, 10,307 insertions and 6 deletions against main.
  • 136 focused offline behavioral assertions pass under the race detector using an ignored standalone harness with fake Kubernetes clients and a local log HTTP server. Coverage includes creation-intent fencing, crash recovery, cancellation/deletion evidence, positional identity hydration, aggregate suite deadlines, exact Node UID ownership, replacement-safe release, orphan reservation recovery, retained cordon repair, Pod/MPI scheduling on cordoned participants, mounted-script digest verification, long-name MPI uniqueness, MPI worker cleanup, per-check wall-clock timeout and quiescence, evidence-gated cleanup, and interrupted chunk-generation recovery.
  • Signal API, controller, adapter, workload, and command packages compile; they intentionally report [no test files].
  • Scoped go vet, go generate ./api/signal/v1alpha1, make signal-manifests, and git diff --check pass.
  • The repository-pinned local linter cannot decode the installed Go 1.27 export data, so exact-head CI is the lint source of truth.
  • Exact-head CI passed all 33 executed checks. The update job was skipped as expected.

Remaining limits and review guide

No committed Signal regression suite and no new real-cluster verification. Foreground garbage collection/RBAC, Argo output persistence, MPI Operator cleanup ordering, and the fixed adapter image's required shell/hash/MPI tooling still need deployment-level validation.

Review api/signal/v1alpha1/ for the immutable CR contracts; internal/signal/controller.go, evidence.go, and projection.go for admission, recovery, audit, cleanup, and release; internal/signaladapter/ for backend execution and evidence capture; and deploy/signal/ for RBAC, templates, and operator documentation.

AI assistance: GitHub Copilot assisted with implementation, investigation, validation, video preparation, and PR drafting. Independent review used Claude Opus 5.5 and GPT-6 Astra. The architecture video's narration was synthesized locally. No human verification or approval is implied.

@chokevin Kevin Cho (chokevin) changed the title feat(signal): add participant backend adapter feat(signal): add grouped certification and backend execution Oct 20, 2026
@chokevin
Kevin Cho (chokevin) marked this pull request as ready for review October 21, 2026 01:33
@chokevin
Kevin Cho (chokevin) requested a review from a team October 21, 2026 01:33
@chokevin

Copy link
Copy Markdown
Contributor Author

Typed workload policy, Pod queue labeling, MPI worker isolation, and focused contract tests (15 seconds):

signal-workload-policy-evidence.mp4

@chokevin Kevin Cho (chokevin) changed the title feat(signal): add grouped certification and backend execution feat(signal): run grouped certification suites Oct 22, 2026
@chokevin Kevin Cho (chokevin) changed the title feat(signal): run grouped certification suites feat(signal): run one production 15-check certification workflow Oct 23, 2026
@chokevin

Copy link
Copy Markdown
Contributor Author

Copilot-generated 20-second fframes explainer. The source excerpts remain present in the production cleanup fixes. The recorded integration is explicitly archival, before the final production-only trim: 15 task nodes, 10 Pass and 5 Inconclusive. This is not a new live run or evidence that the updated head passed certification. The updated implementation is being validated separately; the live cluster and soak were not modified.

signal-pr963-20s.mp4

@chokevin

Copy link
Copy Markdown
Contributor Author

Updated 20-second fframes architecture explainer with local narration: why Signal is needed, how the controller/CRDs/RBAC/adapter template are deployed alongside Argo, operator-owned scripts and Profile, the exact Node request label, immutable admission, serial tasks, and script execution in Pods/MPIJobs. This is an architecture animation, not a live execution recording. Placeholder deployment commands were not executed. It supersedes the archival clip as the architecture overview, not as runtime validation.

signal-deployment-trigger-20s.mp4

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant