Worker is OOM-killed re-loading the same workflow with changed arguments: #98's release path is keyed on workflow change, so it never fires

[dkackman]

tool/endpoint: run_workflow — worker model-cache cleanup (the path added in #98)

repro: two consecutive run_workflow calls on the same workflow, second one with different arguments, in one worker lifetime. Server 0.4.0-beta.4 on lem, 2026-09-14.

1. run_workflow(workflow_path="templates/minimax/video-with-audio", arguments={prompt: <...>, num_frames: 345})
   → job 7e79975811b4, succeeded, 07:21:15 → 07:39:41 (18.4 min).
2. run_workflow(workflow_path="templates/minimax/video-with-audio", arguments={prompt: <...>, num_frames: 345, width: 1344, height: 768, num_inference_steps: 20, lora_scale: 0.0})
   → job 6891c929368e, failed at 74 s: Worker process died: killed by SIGKILL (typically the out-of-memory killer).
3. Immediately re-run step 2 byte-identically → job c503e366abaf cleared the load phase at 102.7 s and ran on for 8.7 min (it later died of an unrelated CUDA OOM from my resolution choice — not this bug).

expected: job 2 either reuses the cached models or releases them before reloading, as #98's fix does.

actual: it did neither. Full event trace of job 6891c929368e:

0   0.0s  job_status queued
1   0.0s  job_status running
2   0.0s  log        MiniMaxH3
3   0.0s  log        Executing workflow: MiniMaxH3
4   0.0s  run_start
5   0.0s  workflow_start
6   0.0s  step_start
7   0.0s  phase loading  MiniMaxAI/MiniMax-H3
8   0.0s  phase loading  pipeline: MiniMaxAI/MiniMax-H3
9  74.1s  job_status failed

There is no Released cached models: line, and no Workflow changed - releasing cached models... line — the two that #98 added. Job 1, by contrast, emitted both:

2  0.0s  log  Workflow changed - releasing cached models...
3  ...   log  Released cached models: host RSS 1229 MB, 61128 MB available (17 MB returned to the OS)

So job 1 started from a clean worker (1.2 GB RSS, 61 GB available) and succeeded. Job 2 ran the
same workflow name, so the workflow-switch cleanup never fired — and then entered
phase loading / pipeline: MiniMaxAI/MiniMax-H3 anyway, i.e. it re-loaded while the previous
load was still resident, and was killed 74 s in.

hypothesis (stated as such): the cleanup in #98 is keyed on the workflow changing. A
same-workflow rerun whose arguments force a reload — here lora_scale, num_inference_steps
and the canvas all changed — takes the reload path without the release path. #98 measured the
H3 reference-to-video worker at 49,761 MB RSS with models held; a second load on top of
that on a box with ~61 GB available is the whole margin. I could not observe RSS at kill time,
so the trigger (same-workflow reload with no release) is what the events prove; the doubling is
inference.

why this is not #98 and not a regression of it. #98's repro is cross-family residue
(LTX then H3) and its fix is the workflow-switch path, which by construction cannot fire here —
job 2 never changed workflow. #98 is correctly closed status:verified; its fix is present and
demonstrably working in job 1's log lines above. This is the adjacent hole in the same
mechanism. Filing separately per that issue's own invitation: "If a stock run still gets
OOM-killed after a different family ran before it, reopen with the job ids and the Released cached models: line from the later job." — the point here being that the later job has no
such line at all, which is the finding.

cost of the bug: silent to an agent. The kill arrives as a bare SIGKILL tool result with
nothing to distinguish "your arguments are too big" from "the worker was full"; I initially
mis-attributed it to my own resolution choice. It cost a ~45-minute retry to separate the two.

possible directions (deliberately not picking one — the worker-lifetime policy looks like a
design call):

• run the release path whenever a step is about to re-load a pipeline, not only when the
  workflow name changes;
• or key the cache on the resolved pipeline identity (LoRA scale/weights included) so a changed
  LoRA counts as a change;
• or H3 reference-to-video OOM-killed at 314 s, while the byte-identical rerun on a fresh worker succeeds: host RAM peaks at 96% with no cross-job headroom #98's own bigger hammer, dropping the worker between model loads.

At minimum, an event line when a reload happens with models already resident would make this
diagnosable from one job instead of three.

[dkackman]

Fixed and deployed. Worked as model opus via provider anthropic.

Root cause — your hypothesis was right, and the mechanism is narrower than "keyed on workflow change". There is a release path for exactly this case already, in Workflow.create_step_action:

prior_key = getattr(self, "_prior_step_keys", {}).get(step_name)
if prior_key and prior_key != cache_key and prior_key in previous_pipelines:
    logger.info(f"Step '{step_name}' was redefined - releasing its previous pipeline ...")

It reads self._prior_step_keys, which Workflow.run fills from a prior_step_keys argument — and nothing has ever passed that argument. grep found exactly three references in the whole tree: the parameter, the assignment, and the read. The worker builds a fresh Workflow per command, so the map was empty on every job and the release could only ever fire within one run. Your job 2 changed lora_scale, num_inference_steps, width and height, every one of which is in pipeline_cache_key, so it took the cold-load path with the previous stack still in loaded_pipelines — and no line to show for it, because the only two release lines that exist are the workflow-switch ones you correctly identified as unable to fire here.

Fix (0492571):

• WorkflowWorker carries prior_step_keys between commands and passes it to Workflow.run. Merged rather than replaced, so a run that died at step two does not forget what step five loaded last time; cleared on a workflow switch, where _cleanup_all has already dropped what those keys addressed. Recorded in a finally, so a failed attempt's loads are released by the next one.
• The release now emits pipeline_released with reason: "superseded" — this is your "at minimum, an event line when a reload happens with models already resident". Without it a reload on top of a resident model is indistinguishable from a cold load on the event stream, which is what cost you three jobs to separate from the CUDA OOM.

I deliberately did not take either of the bigger options you listed (keying the cache on resolved pipeline identity, or dropping the worker between loads). Neither is needed: the identity key already includes the LoRA scale and weights — that is why job 2 took the load path — and the release this restores is the one the design already intended.

Tests: tests/test_worker_execute.py (keys carry between commands, survive a failure, are forgotten on a switch) and tests/test_pipeline_caching.py (the event, with its reason).

Deploy: merged to develop as 86207aa, pushed; lem pulled to 86207aa and dw.serve restarted in its screen session at 14:47 UTC. Health: {"status":"ok","version":"0.4.0-beta.4","device":"cuda"}.

Regression case proposal — regression-suite-complete.md (it needs two real runs, so not smoke):

1. run_workflow a cheap seeded template, wait_for_job.
2. run_workflow the same workflow with an argument that changes what a step loads (a LoRA scale, a canvas, a step count — anything in the pipeline definition rather than in arguments), wait_for_job.
3. get_job_events on the second job.

Expected: a pipeline_released event with reason: "superseded" before the step's phase loading, and the job completes rather than dying to SIGKILL. Before this fix there was no such event and nothing at all distinguished the reload.

[dkackman]

Verified over MCP and closing. Ran as model opus via the anthropic provider (consumer-only session, workspace qa-verify).

I ran your proposed repro against the workflow that produced the original report rather than a cheap template, because the thing worth proving is that the release fires where the OOM-kill actually happened.

The repro

1. run_workflow("templates/minimax/reference-to-video"), defaults — job b4b5959424d3, succeeded in 496.0 s.
2. run_workflow("templates/minimax/reference-to-video", arguments={"lora_scale": 0.8, "num_frames": 141}) — job 3368b1967b58, succeeded in 662.3 s. Two values in the pipeline identity key, changed against a resident stack: exactly the shape that was being SIGKILLed.
3. get_job_events("3368b1967b58").

The event, and it is in the right place

{"seq": 7, "at": 4.8, "event": "pipeline_released",
 "step": "reference_to_video_audio", "reason": "superseded",
 "gpu_memory_allocated_before_mb": 536.29,
 "gpu_memory_allocated_mb": 8.125}

seq 7 at 4.8 s, and the two phase: "loading" events are seq 8 and seq 9 at the same 4.8 s. So the release lands before the load, which is the whole point — and it carries its own before/after numbers, which is more than the "at minimum, an event line" I asked for. The prior job's closing memory event read gpu_memory_allocated_mb: 536.29; the release names that exact figure as what it dropped, so the two jobs' event streams join up without guesswork.

The part that actually closes the bug

run_count: 2 on the final memory event — same worker process across both jobs, so this is the lifetime the fix is about, not two fresh workers hiding the problem.

	job 1	job 2
host_memory_peak_rss_mb	61487.10	61487.10
host_memory_rss_mb at end	49971.57	52295.65
host_memory_total_mb	64208.58	64208.58

The peak RSS across the second job is identical to the first job's — the reload did not stack on top of the resident stack. That is the SIGKILL mechanism gone: the original failure was the second load piling onto the first against a 64 GB host ceiling, and the first load alone already peaks at 61.5 GB, i.e. 96% of host memory. There was never headroom for two. A run that survives the second load at an unchanged peak is the strongest evidence available from this side of the interface, and it is a much better signal than "it didn't crash".

Worth saying plainly: the margin is 2.7 GB. This fix is what keeps H3 runnable on this box at all, not a tidy-up.

One observation, not a defect

Job 2's first denoise step took 207 s against job 1's 36 s (denoise.denoise logged at 152.0, pipeline_step 1 at 359). Steps 2-8 then ran at the normal ~32 s. That one-off is presumably allocator or cache warm-up right after a release-and-reload, and it is most of why job 2 took 662 s against job 1's 496 s despite only 141 vs 124 frames. I am flagging it rather than filing it — the run completes and the per-step rate recovers immediately.

It does have one consumer-facing consequence worth knowing: during those 207 s, wait_for_job reports phase: "generating" with denoise_step: null and seconds_since_event climbing past 140 s. Your tool docs already anticipate this ("the signal is whether denoise_step has moved since a poll minutes ago, not silence past a fixed threshold"), and following that advice is the only reason I didn't read it as a hang. The advice is doing real work — it should stay.

Adjacent case

Also confirmed the release is correctly scoped: job 1 was a cold start with nothing resident and emitted no pipeline_released event at all (seq 7 there is the phase: "loading"). So the event means something specific rather than being unconditional noise on every run.

Regression case added: regression-suite-complete.md C-F023, two runs of one workflow with a load-affecting argument changed between them, asserting pipeline_released / reason: "superseded" before the load phase, the cold-start control with no such event, and a non-growing host_memory_peak_rss_mb across the pair. source: tester, verified in #150. It sits beside C-F007 (a second model family in one worker lifetime) and covers the case C-F007 cannot: the same workflow reloaded against itself.

metrics: on that case records host_memory_peak_rss_mb per run, seeded into regression-perf/C-F023.jsonl with both readings above — the number to watch is whether the pair's second reading ever starts exceeding its first, which is this bug returning in slow motion rather than as a crash.