[Dhirenderchoudhary]

When stopping a Nosana or Akash job fails (sidecar down, network error), the node gets marked terminated with metadata.deprovision_failed = true so the delete doesn't get stuck. But nothing ever reads that marker. The job keeps running and billing, and the only trace is an ERROR log saying "ops must verify/stop it manually".

This makes the DePIN liveness worker (already runs every 45s) clean these up:

• Each tick it lists nodes with deprovision_failed, skipping non-DePIN providers.
• It asks the provider for the job status first. If the job has already finished, nothing is billing, so it just clears the marker. That way an Akash deployment that's already closed (its close call returns non-200) doesn't get retried forever.
• Otherwise it retries _deprovision_direct_node with the pool's credential, and clears the marker once that works. If it still fails, it leaves the marker and tries again next tick.

Also, the worker's own _fail_and_teardown ignored the (ok, err) that _deprovision_direct_node returns, so a failed stop there didn't even get the marker. It now stamps it.

New repo methods: list_deprovision_failed_nodes() and clear_deprovision_failed(node_id=...).

Testing:

• 5 new tests in test_depin_liveness.py: retry succeeds -> marker cleared, retry fails -> marker kept, job already gone -> cleared without calling stop, AWS node -> skipped, teardown failure -> marker stamped. They fail on main and pass here. The rest of the file still passes.
• Ran the two new queries against Postgres in docker: the marked node is listed with its pool credential, and clearing removes only the two marker keys (other metadata stays).
• Full suite with the CI ignore list: same failures as main.

Not covered: if a pool delete hard-deletes the node row before a retry works, the marker goes with it.

[Priyanshu-u07]

@Dhirenderchoudhary Good fix and checking status before stopping is the right design. Two things before merge:

• Akash is retried forever. AkashAdapter has no get_node_status, so it always gets the base class’s “unknown”. An already-closed deployment is closed again every tick, the close returns non-200, and the marker never clears (with an ERROR log each time). The “job already gone” test uses a Nosana fake, so this path isn’t covered. An Akash get_node_status needs a sidecar route first: the depin sidecar only has create, close and logs for Akash, with no /deployments/status/{id}. wait_for_ready already polls that missing route, so Akash readiness can’t work either. I’ll open an issue for both.
• A failed Nosana stop clears the marker. NosanaAdapter.deprovision_node doesn’t check the stop response, so a 500 counts as success, and the retry clears the marker while the job may still be running. Raising on ≥ 400 in the adapter would break the normal stop flow (stopping an already-ended job returns 500), so handle it in the retry instead: keep the marker until a later tick sees a terminal status, and trust the stop only when the provider can’t report status (k8s has no get_node_status).

Please add tests for both using the real adapters (fake only the HTTP layer).

[Priyanshu-u07]

Pushed the retry fix to this branch: when the provider still reports the job running, the marker is kept after the stop and cleared once a later tick sees it stopped; with unknown status (k8s, Akash) a successful stop clears it as before. Tests added for both cases. The Akash status route is tracked in #435