Repository navigation
A Redis restart loses all Task state and orphans the Substrate actors #442
Description
What happens: AX keeps Task state in Redis, and the deployed Redis has no durable storage. RDB snapshotting is on by default, but it writes to an ephemeral /data with no volume or PVC, and AOF is off (appendonly no). A pod restart replaces the container, so /data is empty and every Task record is gone. The Substrate actors are not: they persist in Substrate's own store and keep running. So each one becomes an orphan, a live sandbox that holds a worker slot while AX no longer knows about it. ax delete task cannot remove it, because AX has no record; it needs kubectl ate delete actor <name> -a <atespace> --any-state.
A Redis restart is not rare. A node reschedule, an eviction, an upgrade, or an out-of-memory kill all do it. The worker pods in this setup have no memory limit on Tasks, so an overfilled worker already OOM-kills, and the same pressure can reach Redis.
Evidence (kind install, AX ac23328, Substrate 1d7ca8c):
ax apply -f task.yaml # one Task
ax resume task orphan-repro
# BEFORE: ax describe task -> Phase Running; kubectl ate get actors -> 1 actor
kubectl -n ax-system delete pod -l app.kubernetes.io/name=ax-redis
kubectl -n ax-system rollout status deploy/ax-redis
# AFTER: ax describe task orphan-repro -> "not found" (Task state lost)
# kubectl ate get actors -a default -> orphan-repro still ACTOR_STATE_RUNNINGResult: the Task is gone from AX, and the Substrate actor is still ACTOR_STATE_RUNNING. The two stores have diverged. The worker slot stays used until the actor is deleted directly through the Substrate CLI.
Suggested fix: give the AX Redis persistence (a PVC with AOF, or RDB snapshots), or reconcile against Substrate on startup so AX rebuilds Task state from the live actors, or both. A reconcile would also recover the orphaned actors that a past restart already leaked.
Validation (a crash matrix from a healthy state, 3 Tasks each, AX ac23328):
| Killed pod | Result |
|---|---|
ax-server |
all Tasks RECOVERED: still known, resume works, marker intact. The server is stateless and reads from Redis. |
ax-redis (3 separate trials) |
every Task orphaned, 9 of 9: ax describe task returns not found, the Substrate actor stays ACTOR_STATE_RUNNING. |
The ax-server result is the control: a stateless component recovers cleanly, so the loss is specific to the Redis store, which has no persistence. The Redis result reproduced on every trial.
Suspend, resume, and ssh races across 8 Tasks for 90 s kept each workspace intact. The loss only appears when Redis itself restarts.
Environment: google/ax ac23328 and Agent Substrate 1d7ca8c, on a local kind cluster (Docker Desktop on Apple Silicon), gVisor sandbox class.
Activity
belentani7 commentedon Oct 5, 2026
This is a critical data loss bug. Losing all Task state on Redis restart means in-flight tasks are orphaned and actors are left in inconsistent states.
You need persistent task state with write-ahead logging or snapshotting. On restart, Redis should replay the log to restore task state and re-lease orphaned actors.
Also implement a heartbeat mechanism so actors that died during the restart are detected and their tasks reassigned.
eren23 commentedon Oct 7, 2026
One more effect of this that I hit while testing the task lock on ac23328. The lock key lives in the same Redis, so a restart also drops any lock that is held at that moment.
I froze atelet for 14 s so that
ax delete task Twas still running, restarted Redis, and then ranax applyfor a new task with the same name. The apply got the lock and saved its record while the old delete was still going. When the old delete finished, it removed the new task's record. This happened in 3 of 3 trials.In my runs the new create failed first on the DELETING actor, so nothing was left behind. I think a create that gets further would leave a live actor with no task record, the same orphan as above.
Durable Redis storage would mostly fix this too, since the lock and the record would both survive a restart.
Update. I put a fix on my fork, eren23@21e218b. It gives ax-redis a PVC on
/data, turns on AOF withappendfsync always, and switches the Deployment toRecreatebecause the volume is ReadWriteOnce. In my lab a Redis pod restart now keeps the task records and the locks. My chaos run with Redis, worker and ax-server crashes went from 25 violations to 0, and the restart during a delete above no longer lets a new apply run next to it.kubectl applyupgrades an existing install and restarts Redis once.rkb32 commentedon Oct 19, 2026
Hi @eren23 @mayawang I’d like to work on the startup cleanup for when Redis restarts.
The problem: when the Redis pod restarts, it loses every Task record, but the Substrate actors keep running. ax delete can’t remove those actors, because it returns not found when AX has no Task record.
What I’d add: on AX startup, list the Substrate actors and log any that have no matching Task record. It only logs, and nothing gets deleted.
The other part: keeping Redis data across restarts. eren23’s fork comment (eren23@21e218b) changes the Redis deployment YAML to add a persistent volume and AOF. I checked that the diff matches his description, but I haven’t tested the fix. The cleanup helps even if that change lands.
If this scope isn’t what you want, Lmk on what you'd like to change. Thanks
mayawang commentedon Oct 19, 2026
Thanks @rkb32, a log-only check at startup is a safe first step and fits the plan to move state to Postgres and reconcile against Substrate. PRs are limited to collaborators for now (#448), so push it to a branch on your fork and link it here.
rkb32 commentedon Oct 19, 2026
Hi @mayawang. I’m building the log-only startup check now and will link the branch here once it’s up. I’m leaving Redis persistence out of this change, since that’s eren23’s fork work.
mastersingh24 commentedon Oct 20, 2026
We hit this for real on a GKE cluster this week, and it was worse than losing the tasks.
We drained a node pool to replace it, which restarted ax-redis. AX lost every Task, as described here, but it also lost every Workspace. Our agents read their instructions from a file in their workspace, and they exit at startup if it isn't there. So for the next hour every new task crashed on boot, with Substrate reporting only that the wakeup probe never returned 200. It took a while to connect the crash to the Redis restart.
Two things that might help with the move to Postgres and the reconcile against Substrate that @mayawang mentioned:
EnsureActorgetsAlreadyExistsfrom Substrate and uses the existing actor. We re-created eleven agent tasks that way. The two we checked came back from their memory snapshots, still running processes that had been up for nearly four days. That makes a startup reconcile that rebuilds Tasks from actors look quite doable.--appendonly yes --appendfsync everysec --dir /data), with aRecreatestrategy so the disk only attaches to one pod. After that, restarting the pod keeps all tasks and workspaces. The change todeploy/redis.yamlis here: mastersingh24@464baebIt doesn't solve the orphan problem if Redis is lost some other way, but it stops a routine restart from wiping AX in the meantime.
rkb32 commentedon Oct 21, 2026
Hi @mayawang. The log-only startup check is on branch
orphan-checkin my fork: https://github.com/rkb32/ax/tree/orphan-checkWhat it does: at startup it lists Substrate actors, reads all tasks in one call, and logs any actor with no matching task. It doesn't delete anything, and it doesn't change how the server behaves once it's running.
Known behavior: any Substrate actor with no AX task gets logged, including actors created outside AX. If Substrate hangs at startup, the check gives up after 30 seconds and the server starts without it.
Verification:
go test ./internal/controller/ ./internal/substrate/. I also ran it against a local kind cluster at the version AX pins: an actor with no task is logged, and one with a matching task is not. A 50,000-task Redis store took about half a second to check. Two replicas running at once both logged correctly. I haven't run it on a shared or GKE cluster.The Postgres reconcile is not in this change.
rkb32 commentedon Oct 22, 2026
Hi @itssimrank, I saw you took this issue. I'd already started on a log-only startup check, which is on branch
orphan-checkin my fork: https://github.com/rkb32/ax/tree/orphan-check. It lists Substrate actors at startup and logs any with no matching task. It deletes nothing. @mayawang asked for this approach on 2026-10-19. Could you tell me what you're planning for this issue, so we don't duplicate work?mayawang commentedon Oct 22, 2026
Thanks @rkb32, you were first and I asked for exactly this, so let's go with your branch. One thing first: the Substrate v0.5.0 bump on main changed ListActorsRequest (PageSize and PageToken are gone), so the branch no longer builds. Could you rebase onto main? I'll open the PR from it right after. @itssimrank sorry for the overlap, could you review it instead and suggest the 30s bound from #507? Then we can close #507.
itssimrank commentedon Oct 22, 2026
Works for me, @mayawang — happy to review instead; I'll close #507 once the PR from @rkb32's branch is up.
@rkb32, sorry for the overlap. For the rebase: v0.5 moved the paging fields under ListActorsRequest.Options — the adaptation is in #507 (16d9b7b) if you want to lift it. New files also need the Apache license header.
rkb32 commentedon Oct 22, 2026
@mayawang Rebased onto main (Substrate v0.5.0) and pushed to
orphan-check-v050on https://github.com/rkb32/ax. The build and tests pass. I ran the check against a live Substrate v0.5.0 cluster: an actor with no task is logged, and one with a matching task is not, in bothdefaultand a named atespace. Known edge cases, all low severity and log-only: a repeated page token from Substrate, and a corrupt task record gets dropped without an error. Ready for review.rkb32 commentedon Oct 22, 2026
@itssimrank Np and thanks for catching the license headers. I’ve added them in a new commit on orphan-check-v050 (https://github.com/rkb32/ax/tree/orphan-check-v050), and the build and tests still pass. I also rebased onto main (v0.5.0), so the paging change is in the branch.
itssimrank commentedon Oct 22, 2026
@rkb32 could you open a PR for the changes and add me as reviewer?
rkb32 commentedon Oct 22, 2026
@itssimrank My account can’t open PRs on google/ax (collaborator-only, #448). @mayawang said they’d open the PR from https://github.com/rkb32/ax.
itssimrank commentedon Oct 23, 2026
Ahh okay, likely why the issue got passed on me. I've opened #510 using your fork.
mayawang commentedon Oct 26, 2026
Thanks @rkb32 for building this and for testing it against a live cluster. We're going to close #510: #515 moves state to Postgres, so a Redis restart no longer loses tasks, and listing every actor at startup gets expensive in large deployments. Sorry the direction changed after you did the work. Your offer on #515 to add a restart test against Postgres would be really valuable, if you're still up for it.
rkb32 commentedon Oct 26, 2026
@mayawang Thanks for the rebase review and for explaining why #510 is closing. I'm glad the branch helped get #515 to the restart fix. I'd love to take the restart test against Postgres on #515. Should it also check that locks held before the restart get released afterward, or only that tasks and specs survive? I'll start on it once you answer and post results here.