Skip to content

fix(kubernetes): wait for pod termination on stop - #2858

Closed
drew wants to merge 1 commit into
mainfrom
codex/kubernetes-stop-completion/drew
Closed

fix(kubernetes): wait for pod termination on stop#2858
drew wants to merge 1 commit into
mainfrom
codex/kubernetes-stop-completion/drew

Conversation

@drew

@drew drew commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make Kubernetes stop complete only after the backing pod disappears. This closes the lifecycle race where Suspended=True could be published before the old supervisor finished and acknowledged its canonical process exit, allowing an immediate restart to receive the stale exit report.

Replaces #2857 with the fix at the compute-driver lifecycle boundary.

Related Issue

No issue required: this is an obvious localized Kubernetes lifecycle race with a focused regression test.

Changes

  • Wait for the backing pod to return 404 before completing stop for both v1beta1 and v1alpha1 Sandbox resources.
  • Preserve terminal Suspended=False / PodNotOwned failures while waiting.
  • Add a mock Kubernetes API regression test proving Suspended=True is insufficient while the pod still exists.
  • Document the pod-disappearance completion contract in the Kubernetes driver README.

Testing

  • mise run pre-commit passes
  • cargo test -p openshell-driver-kubernetes (209 passed)
  • mise run test with an isolated OPENSHELL_SYSTEM_GATEWAY_DIR
  • Unit tests added/updated
  • Kubernetes E2E locally (this host does not have k3d, kubectl, or Helm; requested through the test:e2e label)
  • mise run ci fully passes locally: all non-Go lanes passed; Go SDK gateway-list tests read the host default from hard-coded /etc/openshell/gateways and failed their empty-directory assertions

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (not applicable; the existing architecture invariant is unchanged, and the crate README is updated)

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
drew requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners August 21, 2026 03:09
@drew drew added the test:e2e Requires end-to-end coverage label Aug 21, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 53f0d4f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@drew drew closed this Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant