Replay service cooperative cancellation at the canonical boundary

[rmcdaniel]

Customer outcome

Implement Python cooperative cancellation for shared issue 136. Keep this PR draft until the complete model and exact published mixed-language cascade are qualified.

Implementation

• Restore immutable request context and canonical delivery during replay, preserving authored boundaries, earlier outcomes, shielding and child policies.
• Supervise remote and prepared local callbacks in separate processes. Observe cancellation independently of application heartbeats, physically join before acknowledging stop, and fence receipts with original task/owner/attempt authority.
• Bound transport, retries and payload work by original authority. Preserve root, local request, delivery history ID and immutable deadline through cleanup. Keep descendant root request time distinct from local admission time.
• Capture sequential local calls before execution, checkpoint earlier commands, validate durable admission and replay canonical outcomes. Native owns retries and unknown-stop recovery.
• Atomically checkpoint complete ordinary local/remote/child/timer groups and nested positions, then validate every local admission before any callback starts. Join siblings on interruption, skip completed siblings during cold replay and recover unfinished Started attempts.
• Make confirmed supervisor joins complete before cancellation propagates. Retain capacity and registration after unconfirmed stop. Refuse selection and turn-closing waits in prepared atomic groups.
• Keep protocol 1.20 explicit and preserve the frozen 1.19 default and Avro bytes.

Current verification

Current source 0e1c427eae2f6120155a5cf3c9c1ae79ba7a5756 includes CancellationContext.remaining() using the consumed replay clock and original immutable deadline. It preserves first-execution and cold-replay decisions across synchronous results, follows selection winners and awaited handles, excludes later failed-group siblings, retains microseconds, handles recorded clock skew and clamps expiry. Detached metadata, ended replay and unavailable timestamps fail explicitly without host time. Existing WorkflowContext.now() remains fixed to the workflow start.

The helper implementation passed 206 local focused cases at preceding 8c1e937, including 12 remaining-time cases and both Git-backed boundary scanner cases. Current qualification changes are tests and documentation only. Current Ruff, full mypy across 29 source files, all 12 helper cases, collection of the 28 affected connected cases and whitespace checks pass.

Current Source run 37067939883 passes against Server 70807856b416b8792099108712cdd031e2c66c89 and Native ae1513bf78a175f4965e4fb8fc473fab60c35ec1. Ordinary CI 37067887319 passes too. Connected qualification has 50 passes, zero errors/failures, one existing unavailable-CLI skip, 561.247 seconds. Sequential and atomic prepared-cleanup cases physically stop callbacks without application heartbeats, SIGKILL the cleanup owner, recover with a fresh process, preserve unknown stop state, retain duplicate original identity/deadline and finish Cancelled within the original 30 seconds. Sequential original/replacement remaining is 28.858494 seconds, final 4.153142 seconds. Atomic original/replacement is 28.857760 seconds, final 13.957172 seconds. Final values match original deadline minus consumed committed completion within one microsecond. The legacy inline case correctly retains 119.502138 seconds across original/replacement/post-inline observations. Raw artifact 11253209659 retains observations, histories, receipts, JUnit and exact provenance until December 31. The preceding incorrect inline assertion is documented in its failed run. Published/default protocol 1.19 is unchanged.

Preceding qualified operation-policy source

Source de0c476760524dcac3df35572bb7bd19d818a2d0 accepts typed and string remote Activity policies in both encoders and refuses invalid or unbounded authoring. Missing negotiated worker capability is diagnosed before submission. Replay compares policy with original canonical history for ordinary and grouped calls and cancellation delivery. A focused reproduction failed before repair, and its minimal Source regression fixture uses the official replayer. Historical omission retains Try and unchanged wire shape. Local Python 3.12 checks pass 181 cases, with zero errors, failures or skips. Full Ruff and mypy pass. Exact Source CI passes every required job against Server 8940fb8a866e24d7874947e850f7fc864fbdfd3c and Native 9f8cb61947997c1c60f294bdf559d02d8258cc18. Connected qualification passes 50 cases, zero errors or failures, one existing unavailable-CLI skip, 562.214 seconds. The five new explicit Try/Wait callback-stop and bounded independent Abandon cases all pass. Wait receipts precede delivery for both async and sync callbacks. Abandon commits completion after parent closure under the original total deadline and refuses another outcome. Raw scenarios, image authority and provenance are retained until 2026-12-31. Hosted stack teardown passes. Local policies and the published mixed-language gate remain required.

Qualified preceding pending delivery contract

Candidate bf931959af5cc6615cb42ccd0ed279e7b3d33024 validates pending Activity cancellation replies with explicit claim release, matching task identity, null delivery fields and an appropriate authored call kind. It preserves child wait validation and rejects malformed or unrelated pending replies. Exact Source qualification passes every job against Server 2fc521b5f132c5a6d0909ec6239f2fbbc435bae6 and Native bf047ce2e88e6c565eb6b80fae6559216524d41b, including Python 3.10–3.12, lint/type checks, package and corpus. Connected qualification passes 45 cases, zero errors or failures, one existing unavailable-CLI skip, 490.243 seconds. Raw connected evidence is retained until 2026-12-31. Artifact SHA256 24bbddc080f6c8ca74a19fb6638b0683de66aa34912cb8e6ea5571ec09a44be5. Hosted task resources were removed. Ordinary exact-head CI also passes. The published mixed-language cascade remains required.

Qualified preceding prepared-group baseline

Current candidate a6b243e84532e56d04d799e7b1e2c387874c5e3e passes every Source CI job, including Python 3.10–3.12, lint, package, corpus and target-branch qualification. Connected qualification binds Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native readonly overlay 673343b3194b8bf29a10eca29c493932db028bc1. It passes 45 cases, zero errors or failures, one existing unavailable-CLI skip, 509.950 seconds.

Sequential and nested mixed local/timer groups persist both authored heartbeat phase values through the canonical progress.details envelope, retain full nested membership, complete and cold replay their original results. The Native repair admits the six membership fields already emitted by its heartbeat producer. Unknown producer fields remain rejected. Current Native 16b722e74a10451217b1c16f25b5b1bcb1ebc45d differs from the qualified overlay only in test formatting. Runtime and resource files are byte-identical.

Both cancellation cases stop callbacks without application heartbeats, kill the owner during shielded cleanup and recover on a fresh worker with the same delivery and unchanged request, root and deadline. Sequential closure takes 24.386563 seconds and atomic two-member closure takes 25.342040 seconds, each before its original 30-second deadline. All three sequential callback PIDs and all six group callback PIDs physically exit. The group history has one request, one delivery, two cancellation acknowledgments, two unknown-stop recoveries, two cleanup completions and one Cancelled closure. Recovery accurately records unknown stop state and does not grant a new budget.

Raw results, receipts, histories, image digests and exact source provenance are retained until December 31, 2026. Artifact SHA256 7b7ae6fed325d2fabbd94d98593f85f877738529dc022784fd30d5c1fedd5f55. JUnit SHA256 f9886503a506ed41b92e56094bd80cf59e3c9c0c17c28486933d765571477562. Hosted task containers and network were removed successfully.

Local Python 3.12.15 qualification passes 104 affected checks with zero skips, Ruff and mypy. The supervised heartbeat regression fails the preceding runner and passes the correction. Atomic group implementation 0a7c68b140d4b74b73120716b2dff97773ae58ae also passed 112 focused checks with zero skips and 1,927 full local checks with two existing skips. Earlier failed Source runs and repair details remain in the PR comments with retained artifacts.

Remaining work

Complete operation policies and scopes, then qualify the published cross-language cascade with one API/CLI/UI inspection view and fair competitive proof. Shared issue 136 stays open. This candidate is not published or adopted by Cloud.

[rmcdaniel]

All source checks have passed for 18ec381ea7c0d9378775df033893b9069bdb7802, including Python 3.10, 3.11 and 3.12, lint, regression corpus policy, built package, docs, public boundary and the existing Server integration suite. Target branch qualification completed successfully at 23:44 UTC September 30.

The connected cooperative worker/client path is still the next implementation step. Existing integration success verifies the preserved supported protocol 1.19 surface. It does not qualify the new delivery intent, heartbeat observation, acknowledgment recovery or active local activity fencing against Server PR291. This PR remains a draft and advertises no cooperative cancellation capability. The completed local test worktree and tool images are removed, with the active branch preserved.

[rmcdaniel]

Request and delivery client source

a3c909c403634061cea81ebeb200342f31f14d9e adds separate request and delivery carriers. Request uses control credentials, an optional selected-current-run route and Server's original request metadata. Missing/false capability discovery, malformed/older protocol, invalid timeout and mismatched acknowledgment fail before execution is treated as accepted. A handle retains its selected run. There is no fallback to immediate cancel.

Delivery requires explicit compatible protocol 1.20, worker credentials, the recorded lease owner/attempt and a canonical call/range. Success must acknowledge the exact task/request/boundary. Transport retry preserves that body. Lease and attempt refusals retain Server's machine reason and do not retry as another claim.

Local source check on Python 3.12.14:

pytest tests/test_cooperative_cancellation_client.py tests/test_client.py tests/test_cooperative_cancellation.py tests/test_replay_regression_corpus.py -q
ruff check src/ tests/
mypy src/durable_workflow/
python scripts/ci/validate-regression-corpus.py --base-ref f542486e125fdb2738d0d6185ff8fe748e89ddcd

All 323 tests pass, including 52 new carrier cases. Lint, strict typing and corpus policy pass. CI36793706333 is qualifying this head.

The next Worker change must consume claim and heartbeat observations, persist the replay delivery intent before throwing, then reload Server-issued canonical history with the same owner/attempt. Active local activity results must be fenced across delivery, death and reclaim. Default protocol/capability advertisement remains unchanged until that path, all three SDK implementations and the exact published tuple qualify.

[rmcdaniel]

Client source qualification complete

At a3c909c403634061cea81ebeb200342f31f14d9e, normal CI passed every gate, including all three Python versions, lint, corpus, package and existing Server integration. Docs and boundary checks passed too. Python 3.11 reports 1,577 passing tests, two skips and 31 integration cases deselected.

The new client carrier requires explicit compatible capability discovery for cooperative requests. A run-bound handle preserves its run fence. Worker delivery requires protocol 1.20, worker credentials, claim owner/attempt and the authored call range. Duplicate requests preserve Server's original identity and deadline. Lost acknowledgments retry the identical delivery, and malformed or mismatched acknowledgments are rejected. Earlier immediate cancellation and termination remain covered.

The completed source worktree, dependencies, caches, tool images and build context have been removed. The active draft branch remains. Next is Worker integration for claim/heartbeat observations, canonical history refresh, delivery persistence and active local-result fencing, then the accepted connected failure cases and equivalent PHP/Rust behavior. Default protocol and published capability claims remain unchanged.

[rmcdaniel]

Worker source gates completed

Exact head febb5906541f4df82cd3f4f3e2c11f41e20acff6 passed every normal CI gate in run 36797286205, including Python 3.10/3.11/3.12, lint, corpus, package, existing Server integration and target-branch qualification. The last gate completed on October 1 at 00:44:19 UTC. Docs run 36797286266 and public boundary checks passed as well.

The local suite passed 1,607 tests with two existing skips and 31 integration cases deselected. Thirty Worker cases exercise claim/heartbeat observation, canonical paging and delivery, earlier commands, acknowledgment loss, local work and lease fencing, shielded cleanup, cold replay and incapable registrations.

The draft remains protocol 1.19 by default. Connected candidate Server qualification, actual process death/reclaim, shutdown during shielded cleanup, synchronous and remote in-flight work, deadlines and termination remain required, along with PHP/Rust equivalents and coordinated publication. Existing Server integration covers the preserved current protocol surface.

Cleanup is complete: the clean source worktree, virtual environment, test logs, build context and both disposable tool images are removed. The remote branch and this evidence retain the implementation and reproducible commands.

[rmcdaniel]

Negotiated local cleanup shutdown and synchronous execution

Exact head 797e3935d7a738eb91fc98a396a772ced82bd31f drains negotiated local cleanup within the configured worker shutdown grace period. At the deadline, it fences local execution and abandons the workflow claim without producing an activity cancellation/failure or a completion. A replacement worker consumes the original committed marker, uses a new owner/attempt and resumes cleanup with the original request/deadline.

Synchronous local handlers execute off the event loop in a lazy pool bounded by max_concurrent_workflow_tasks. It is separate from replay threads, which wait for local results. Context variables preserve activity metadata through that boundary. Shutdown and request observation fence late heartbeats and unencodable results from both async cancellation-suppressing handlers and synchronous handlers. Python cannot forcibly stop a running thread, and external effects retain their at-least-once idempotency/reconciliation requirement.

Local evidence: 1,611 tests passed, two existing skips, 31 integration cases deselected, 26.77seconds. All 34 focused Worker cases pass. Ruff, strict mypy for all 26 source modules and the revision-aware corpus guard pass. The synchronous observation case deliberately limits replay to one thread to expose a shared-pool deadlock. Replacement source cases preserve canonical request/history and use a different lease owner and attempt5 without redelivery.

Execution uses Python3.12.14, UID/GID1000:1000, two CPUs, 1GiB memory with no additional swap allowance and pids256. Python base index is sha256:f77ac9e44ae96ef2c90b8053ea08c31f8be030f824196b0ae4db6d462c84e51f. Git-only tool image config is sha256:eb1288c6265cdae40a2ea36b8e07a627e99c7ae4bfacf71647d899a6a1357dfa. After installing the editable package with its dev dependencies, commands were:

ruff check src/ tests/
mypy src/durable_workflow/
pytest tests/ -m "not integration" -q
python scripts/ci/validate-regression-corpus.py --base-ref f542486e125fdb2738d0d6185ff8fe748e89ddcd

Normal CI for the exact head completed successfully at 01:25:59 UTC on October 1. All jobs pass, including Python 3.10/3.11/3.12, existing Server integration, lint, corpus, package and target-branch qualification. Docs and public boundary checks also pass at the same head.

The draft remains pending connected candidate Server qualification, actual native process replacement, native shutdown/in-flight work, deadline/termination and acknowledgment-loss cases, PHP/Rust equivalents and coordinated publication. Default protocol1.19 and current registration advertisement remain intact. Completed local worktree, dependencies, build context, diagnostics and task images have been removed. No published support claim is added by these source checks.

[rmcdaniel]

Connected Python/Server source qualification

At October 1, 2026, 02:25 UTC, Python 119747bb2fb838f6dff64c9938c780414a074b99 and Server 2e2d6b31df981e98c1a054a8953e9efe95d17e6e pass all 11 connected cases on SQLite in 20.70 seconds. This uses the actual authenticated API, real database-backed queue worker, actual claim/heartbeat/history endpoints and separate OS processes. Protocol 1.20 is configured explicitly. Published defaults remain 1.19.

Cases	Verified outcome
Request before claim, ordinary response	Duplicate requests keep the original ID and deadline. One canonical delivery precedes cleanup.
Request before claim, discarded successful delivery response	Canonical history proves the committed delivery. No second delivery call or cleanup duplication.
Already waiting on a timer	Delivery cancels the pending timer, then shielded cleanup completes.
Leased remote activity	The heartbeat refuses continuation during cleanup. Late result and failure receive typed run_cancelled responses and add no history.
Active local work, async and synchronous	Actual lease observation catches the request without a user heartbeat. The late unencodable result is discarded. Only cleanup records completion.
Shutdown within grace	Shielded cleanup drains and the run closes as cancelled.
Shutdown after grace	The old local claim is abandoned. Deregistration records the ordinary repair, with the original cancellation history unchanged. A different owner and later attempt resumes the original request/deadline.
SIGKILL and cold process replacement	The first child is killed with exit -9 during cleanup. A new child claims a later attempt and consumes the existing marker. One cleanup result and one terminal cancellation remain.
Cleanup deadline	Expiry closes the run and fences the still executing local result.
Termination during cleanup	The run closes as terminated and the still executing local result is fenced.

The host used isolated containers at UID/GID 1000, 2 CPU and 1 GiB memory with no additional swap per tool/service. Runtime base is published Server 2.4.34 index sha256:1b660a7228be15a777b46dfb720ba85b083cf8ee70d6d1cd29853118bb1b2715, overlaid with the candidate Server checkout and its unchanged Composer lock. Workflow is published 2.3.0. PHP is 8.3.35 and Python is 3.13.5. The disposable Git-enabled tool config is sha256:8827adb0b23799616f93f587c6555debc559926fefcca61c4e871f20598ee32a. These are source results, not published capability qualification.

Local checks:

• pytest tests/integration/test_cooperative_cancellation.py -v, with DURABLE_WORKFLOW_COOPERATIVE_QUALIFICATION=1 and worker protocol 1.20, passes all 11 cases at the exact heads above.
• pytest tests -m 'not integration' -q passes 1,618 tests, two existing skips and 42 integration cases deselected, in 13.37 seconds. Ruff and strict mypy pass for 26 source modules.
• Server's discovery/request/cluster/success filter passes 145 tests and 5,165 assertions, in 15.258 seconds. PHPUnit also reports 11 annotation metadata deprecations. Pint passes for the four changed PHP files.

Normal Python CI and the explicit Python/MySQL candidate run pass every job at the exact head. The candidate integration completed at 02:29:45 UTC, with 33 passed and one existing CLI skip in 80.19 seconds. All eleven cooperative cases are present and passed in the downloaded JUnit artifact, which expires October 8. The log verifies the exact Server SHA above, and stack teardown passed. Supported Python 3.10/3.11/3.12, lint, corpus, package, docs and public boundaries pass. Server source/corpus CI passes 2,279 tests and 47,800 assertions. MySQL replay/query topology passes. Polling bounded-growth smoke and its performance qualification also pass at this exact head. All normal Server gates are complete. PHP/Rust equivalents, coordinated specification/capability activation and exact published tuple conformance remain required.

Both PRs remain drafts. Local stack and disposable source/dependency/proof/tool resources are removed. Downloaded CI diagnostics and transport files are removed after updating this record.

[rmcdaniel]

Next source phase: supervise actual cooperative remote activity callbacks through the qualified Server readonly observation route (Server073a516bbd4063f57d4ad65ad732ec423131c8ff). Keep worker registration and only authored activity progress, observe canonical cancellation/attempt/session/deadline fences, abandon on failed observation or shutdown expiry, and reject late heartbeat/result/failure publication. Qualify blocked async callbacks and synchronous handlers while preserving event-loop responsiveness and the existing bounded thread-pool model. Python threads cannot be forcibly stopped, so prove their late publication is fenced and document the remaining external-effect/process-supervisor responsibility. Extend the connected actual-worker cases and retain exact gates before claiming parity. Ordinary protocol1.19 registration and published capabilities stay unchanged.

[rmcdaniel]

Remote worker qualification is running at Python 88f0e31bf1736271deaabcc67d74df9ce98df491 against Server 073a516bbd4063f57d4ad65ad732ec423131c8ff and the published Workflow package selected by that Server source.

The exact-head ordinary CI and connected integration pass. The opt-in candidate run 36825588696 passes 39 connected cases, fails the new killed-remote-owner recovery case, and retains the existing CLI skip. All six new async/sync blocked-callback and shutdown cases pass. The replacement process in the kill case times out before receiving its workflow claim. Its result is not a qualified recovery claim.

Next action: reproduce that case in an isolated local stack using the same Server commit and inspect task leases, registration state and durable history. Determine whether the failure is a fixture bound or a recovery defect before changing it, then rerun the exact-head candidate gate. The failed runner completed teardown successfully and retained JUnit in artifact 11145516155 until October 8. Protocol 1.20 remains explicitly opt-in and this PR remains draft.

[rmcdaniel]

The isolated reproduction identifies the failure as Workflow #599. A still-issued workflow poll claims the new task for the killed process. Native repair then hides that expired workflow lease behind the older activity lease and repeatedly returns repair_not_needed.

A native regression using the actual workflow and activity bridges fails on the published baseline, while its fresh-lease case passes. Prioritizing expired leases in the native run summary makes both cases pass, with 33 assertions checking that only the workflow claim changes and the activity task, execution, attempt and history stay unchanged.

The unchanged Python SIGKILL case passes in 18.28 seconds when the isolated Server loads that corrected source. Its 30-second claim bound is unchanged. The replacement workflow task has attempt 2 and repair count 1, commits canonical delivery/cleanup, and the dead activity owner's late completion and failure are rejected. This local source result is the counterfactual for the defect. The full cooperative source suite, Workflow quality cycle, patch publication and exact dependent candidate gate are still running or pending. Protocol 1.20 remains opt-in.

Next action: qualify and publish the Workflow repair patch, update the Server candidate to that exact package, then rerun Python's entire connected candidate suite and retire the local qualification resources.

[rmcdaniel]

The cooperative Server candidate now consumes published Workflow 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387. Exact Server head is c32434cc784a1bf870dcf19568bef944a320f6a7. It merges the reviewed Server #292 dependency/release-metadata change into the previous 073a516bbd4063f57d4ad65ad732ec423131c8ff candidate. The candidate's service implementation is unchanged and protocol defaults remain 1.19. Normal source checks have restarted for this exact tuple.

Workflow's complete source matrix and all 16 published Laravel/PHP upgrade combinations pass. Server #292 is separately qualifying stable image 2.4.35. A focused ordinary-protocol signal drill reproduces the expired-workflow/older-active-activity defect on published Server 2.4.34 and PHP SDK 2.1.6, so this repair also affects existing service callers.

The entire Python candidate CI is dispatched at unchanged Python 88f0e31bf1736271deaabcc67d74df9ce98df491, with server_commit=c32434cc784a1bf870dcf19568bef944a320f6a7 and cooperative_qualification=true. It must pass the actual remote-worker supervision and SIGKILL cases using the published native correction. This is source-candidate qualification, separate from Server #292's published-image PHP/Python/Rust follow-through and shared #136's activation/reclaim/Rust gates.

[rmcdaniel]

Exact native correction requalification complete

Python head 88f0e31bf1736271deaabcc67d74df9ce98df491 passes connected CI 36831983386 against Server c32434cc784a1bf870dcf19568bef944a320f6a7 and published Workflow 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387. Every job passes. Integration has 41 cases, 40 passing and one existing CLI absence skip, in 129.300 seconds. All 18 cooperative cases pass, including actual async/synchronous remote callbacks with and without user heartbeats, readonly observation without progress renewal, stale publication fencing, actual shutdown and real remote-owner SIGKILL followed by cold replacement. The killed remote-owner case passes in 11.603 seconds. JUnit is retained through October 8. Job cleanup succeeds.

Server's current exact head passes feature/corpus with 2,300 tests and 47,995 assertions, MySQL HTTP topology, MySQL/Postgres rolling, bounded polling, chart validation/install and public boundaries.

The correction is also published independently in stable Server 2.4.35 with its default protocol 1.19. Its focused published recovery test and all 12 published portable lifecycle cells pass. These candidate source results do not activate cooperative protocol 1.20. Next action is finish Rust's equivalent actual remote-worker path and active activity-attempt reclaim qualification, then coordinate publication and exact published tuple gates. PHP/Python/Server PRs remain drafts through that work.

[rmcdaniel]

Requalified against the current Server and published Native package

Python 88f0e31bf1736271deaabcc67d74df9ce98df491 passes the complete opt-in CI workflow against exact Server 555cf7ab4ed4e9a79b52475be76a6dfcf886dd70, which embeds published Workflow 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387.

The connected suite reports 40 passed, 1 skipped in 128.61 seconds, including all 18 cooperative cases. These include actual async/synchronous remote callbacks with and without authored heartbeats, accepted owner registration heartbeats, shutdown fencing, canonical waiting/delivery, lost replies, local callback result suppression, shielded cleanup replacement, real remote-owner SIGKILL, cleanup reclamation in a new process, and cleanup deadline/termination fencing. Ordinary smoke, payload, session and memo integration cases also pass. The supported Python matrix, package, corpus and Avro checks pass in the same run.

Raw connected JUnit is retained for seven days. CI removed its isolated stack, network, payload volume and task images. The Server prefix successor fix is included in this candidate.

Next: complete active remote-attempt replacement qualification and the remaining shared per-language scenarios, then qualify the exact published tuple. This is source qualification, and this PR remains a draft. Published protocol/capability defaults are unchanged.

[rmcdaniel]

Child waits release the Python claim

Source 2b28ccad73b7d2ad76d9e5a4dad44a0bd33f2769 now accepts only the explicit
cancellation_waiting_for_child response with claim_released: true, the
matching task, and no claimed delivery boundary. It rejects malformed release
flags, mismatched tasks, fabricated delivery fields and unrelated timer waits.

The worker returns to polling without publishing completion or failure and
without attempting history paging on the completed claim. Its source fixture
executes another workflow task, then replays parent cleanup on a new task while
preserving the original request and deadline. Successful delivery still needs
matching canonical history before authored cleanup.

Focused local source checks passed: 151 tests, Ruff, and mypy over all 26
source files. Exact-head CI is running:
https://github.com/durable-workflow/sdk-python/actions/runs/36930582866

Connected child-policy qualification remains required. This is a source draft,
not a published capability. Default protocol stays 1.19, cooperative 1.20 is an
explicit unfrozen opt-in, and shared cancellation 136 remains open for the
strong model and exact published 30-second mixed-language cascade.

[rmcdaniel]

Prepared local transport foundation qualified

Candidate 8eabc65bb4e01595c3a75758ccca86fb898881d6 adds the unfrozen protocol 1.20 prepared-local transport. It does not yet advertise or implement the prepared-local Worker/replay consumer.

The client validates the operation, original task/owner/epoch fence, attempt identity and finite JSON before I/O. Its single total budget covers retries and external payload transfer. Negotiated Source v2 completion contexts retain original admission or attempt identity during draining. The frozen 1.19 context and Avro bytes remain unchanged.

Local Python 3.12.15 qualification on October 2 passes 191 tests, zero failures or skips, in 7.35 seconds, plus Ruff over src/tests and mypy over all 28 source files. The 30 new transport cases cover malformed authority and receipts, namespace/worker credentials, URL encoding, total timeout cancellation, negotiated payload externalization and original v2 continuation fencing. Local JUnit SHA256: b4c1e25131ee4102088ed379a57b5613d38c7a1521838c57877eedd8240a4c3d.

Exact-head CI passes Python 3.10–3.12, lint, package, corpus, boundaries, documentation and ordinary connected Server integration. GitHub's PR merge checkout is d03b7723b892d0cb0944328d30d2e3e4d89212e6, for head 8eabc65bb4e01595c3a75758ccca86fb898881d6. Ordinary integration binds Server 7ea89349b7d39c075e742dcf2e14c1ddb0d1050e, with no Native source overlay, and passes 22 cases with 20 explicit skips for unpublished cooperation and unavailable CLI. JUnit, image/package provenance and observations are retained. JUnit SHA256: 494927241fd9048d4b27f5e503d09c0a8ed140d112e8f47eebd49268a34c6a21.

Next: implement prepared replay capture and the managed local callback supervisor, then qualify connected stop, cleanup and cold replay against an exact Native/Server Source tuple within the original 30-second cancellation deadline. The published PHP→Python→Rust cascade and shared issue 136 remain open. This PR remains a draft.

[rmcdaniel]

Prepared sequential local consumer implemented, connected qualification running

Head: 5150044c3bec86b8d83e206f5468dbe8a0057d8e.

Replay now captures the authored call before a callback can execute. The worker checkpoints a supported earlier command prefix, obtains Native admission on the original claim and validates its backend identities, epoch, nonce, deadlines and cleanup authority. It runs the callback under the existing process supervisor. Independent control creates no application heartbeat history. Cancellation stops and joins the callback and supervisor before acknowledgment. Outcome history supplies the next replay value, and unfinished Started attempts request Native recovery instead of reexecuting in memory.

Cleanup retains the original root/local request IDs, delivery history ID and deadline. A descendant's root request time and local admission time stay distinct. An unconfirmed stop retains workflow capacity and registration. Atomic local parallel and selection groups remain explicitly refused.

Local verification in Python 3.12.15: 52 prepared-local checks passed, no skips, Ruff and mypy passed, and the regression-corpus gate passed against prior head 8eabc65bb4e01595c3a75758ccca86fb898881d6. The appended Source 1.20 cold-results fixture checks completed callback reuse and unfinished original-attempt recovery. The exact-head full local suite passes 1,904 cases, 2 existing skips, with 52 connected cases deselected. Hosted Python 3.10, 3.11 and 3.12, lint, package, central action policy and regression-corpus jobs pass. Connected qualification remains in progress.

Exact-source CI 37001051027 is running with Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native 850a64050b57d47181abeae86d163962c9047317. Its new connected checks include two admitted local callbacks and an owner SIGKILL during shielded cleanup with an original 30-second cancellation budget. Connected outcomes are pending and must pass before this component is described as qualified.

PR remains a draft. Published artifacts, default protocol 1.19 and Cloud adoption are unchanged. Shared issue 136 remains open for the full published PHP parent, Python child, Rust remote activity and PHP local activity cascade, operation policies, scopes, competitive proof and one inspection view.

[rmcdaniel]

Prepared sequential local consumer qualified against the exact Source tuple

Component	Exact commit
Python SDK	5150044c3bec86b8d83e206f5468dbe8a0057d8e
Server	c1bda7f81323e9939ef433c7ebca66f62365cdfd
Native readonly overlay	850a64050b57d47181abeae86d163962c9047317

Source CI 37001051027 passed every job. Connected integration passed 43 cases with one existing CLI-binary skip, zero failures or errors, in 480.41 seconds. Both new prepared-local cases passed with no skip.

Durable sequential execution

The 2.775-second case checkpoints memo metadata before admission, executes two distinct backend attempts in supervised processes, records exactly two real application heartbeats, completes both activities and the workflow, then cold-replays the results without callbacks. Canonical history contains one memo event, two Scheduled/Started/Completed activity sequences and one workflow completion.

Original 30-second cancellation and cleanup recovery

The case stops the original local callback without application heartbeats and verifies physical exit before cancellation acknowledgment. It delivers cancellation once, starts shielded cleanup, SIGKILLs the workflow owner, verifies the surviving supervisor stopped the cleanup callback, and runs a replacement owner. Native records unknown stop for the old cleanup attempt, releases the claim for a durable retry, and admits one new cleanup attempt. Both admissions retain the original root/local request IDs, delivery history ID and deadline. Duplicate cancellation preserves request identity, request time and deadline.

Observation	UTC
Original request	2026-10-02T11:38:55.742903Z
Original deadline	2026-10-02T11:39:25.742903Z
Workflow Cancelled	2026-10-02T11:39:21.774404Z

Completion was 26.031501 seconds after the request, leaving 3.968499 seconds. The whole test took 28.242 seconds including setup. The qualification stack uses a 10-second workflow-task timeout and a 10-second repair cadence. Independent callback control polls at most once per second and bounds transport by five seconds and original authority. This is one measured Source scenario, with no claim about arbitrary workloads or a published capacity/SLA change.

The original and replacement cleanup admissions share delivery ID 01m3y6j21cy792qvarbzxqyjbp. History contains one request, one delivery, one original activity cancellation, one physical-stop acknowledgment, one unknown-stop retry, one successful cleanup completion and one WorkflowCancelled. It contains no application heartbeat. The two cleanup backend attempt IDs are distinct. Physical exit assertions passed for original work, killed cleanup and replacement cleanup callbacks.

Retained evidence and additional checks

• Raw histories, receipts, JUnit, image authority and exact source provenance, retained through December 31, 2026. Artifact digest: 5741881d97c424a72d83b0cf87d8b3d21932b366bcd4707d271341a9b10254fe.
• Connected JUnit SHA256: 6a8f11cfb56059b018c7626dfbca07b93545e80282dc6529b70a86051b364a00.
• Exact-head full local suite: 1,904 passed, 2 existing skips, 52 connected cases deselected, 119.79 seconds. Local JUnit SHA256: 34aceee7476e3b72fefac609054beb0481ea3182057a96142e2d1bf26bc92c67.
• Prepared-local checks: 52 passed, no skips, 12.40 seconds. Local JUnit SHA256: 06acfbf0cc564e811338bfb5ce1edc8bd5e559cd4d6af014f8964117172220be.
• Hosted Python 3.10–3.12, Ruff, mypy, package, corpus and central action policy passed. All current ordinary PR checks also pass.
• Immutable Source cold-results fixture SHA256: 465e6753bf6b6cfd25bdda2f3a4bc91aedfc3e2d37b64af53b00e3c89d578a1e.
• Hosted stack/image/volume teardown succeeded.

Reproduce with gh workflow run ci.yml --repo durable-workflow/sdk-python --ref feat/cooperative-cancellation -f server_commit=c1bda7f81323e9939ef433c7ebca66f62365cdfd -f cooperative_qualification=true -f native_commit=850a64050b57d47181abeae86d163962c9047317, after verifying the ref still resolves to the exact Python commit above.

The image retains published Composer Native authority 2.3.3 / 70d4fe48efd7dd796c35c1078b3d5ac43f738f4f; the Source qualification deliberately replaces that package directory with the readonly exact Native overlay identified above. This qualifies the Source consumer, not a published tuple.

PR stays draft. Python atomic local group consumption, operation policies, scopes, the exact published PHP parent → Python child → Rust remote activity / PHP local activity cascade, one inspection view and fair competitive proof remain outstanding. Shared issue 136 stays open. Published artifacts, default protocol 1.19 and Cloud adoption are unchanged.

[rmcdaniel]

Atomic prepared local group consumer implemented, connected qualification running

Candidate 0a7c68b140d4b74b73120716b2dff97773ae58ae adds explicit prepared_local_activity_groups negotiation and complete ordinary parallel admission before any callback starts. Nested local/remote/child/timer membership is bounded to 100 leaves. Canonical opening history and execution IDs must agree with the retained checkpoint. All local attempts must validate before spawning. Completed siblings replay stored results. Unfinished Started siblings use Native recovery before new admission.

Group interruption joins all callbacks before workflow replay proceeds. Unknown stop retains capacity and worker registration. The tests exposed cancellation interrupting a supervisor join. Confirmed joins and prepared-finalization now finish before cancellation propagates, with a dedicated process regression test. Result receipts remain required, and Native owns retries and deadlines. Selection and turn-closing waits remain explicitly refused by this admission path.

Local Python 3.12.15 verification passed:

• 112 focused checks, zero skips, including atomic refusal, partial history, immutable partial completion, original cleanup authority, concurrent callbacks and physical group stop without application heartbeats, 39.81 seconds. JUnit SHA256 43f2c9d96db52b605db55eca90b6fda4660ef39a99da7bbfc1750117ef6c8122.
• 1,927 full-suite checks, two existing skips, 54 connected cases deselected, 131.45 seconds. JUnit SHA256 782114a1d874861948ee8e94c52854c734734e399e77e60ceb9829382451c83e.
• Ruff, mypy and corpus policy passed. New immutable group fixture SHA256 8b74c615edc472d8483475efc6b5cbaae0dfb73f919afd9a2924a4295152b45f. Replay corpus grows from 35 to 36, codec corpus remains 41.

Exact Source qualification 37005040454 is running against Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native readonly overlay 850a64050b57d47181abeae86d163962c9047317. It includes a nested mixed local/timer group and owner SIGKILL during two-member cleanup under the original 30-second deadline, with raw physical PID evidence. Results are pending.

PR stays draft and shared issue 136 stays open. The earlier sequential report qualifies its own earlier head. This candidate is not published. Operation policies, scopes, the exact published PHP → Python → Rust cascade, one inspection view and fair competitive proof remain required.

[rmcdaniel]

Source group qualification failed, nested admission diagnosis in progress

Run 37005040454 completed with 44 passing cases, one failure and one existing CLI-binary skip. Exact tuple:

• Python 0a7c68b140d4b74b73120716b2dff97773ae58ae
• Server c1bda7f81323e9939ef433c7ebca66f62365cdfd
• Native readonly Source overlay 850a64050b57d47181abeae86d163962c9047317

The nested mixed local/timer case failed after a refused prepared operation and a local heartbeat deadline. The original warning did not retain the refusal reason. This run does not qualify atomic groups. The next candidate retains the exact refusal and final canonical history rather than weakening or skipping the scenario.

The two-member cancellation case passed on this tuple. Two active callback processes stopped without application heartbeats. Two cleanup processes exited with their SIGKILLed owner. Two replacement cleanup processes exited after completion. All six physical exit assertions passed. Both unknown-stop recoveries preserved the original root request, local request, delivery event and cleanup deadline. Duplicate requests returned the original identity and deadline. Canonical terminal time was 12:22:28.224484Z, before the original deadline 12:22:41.989114Z. The history contains one request, one delivery, two activity cancellations, two stop acknowledgements, two unknown-stop retries, two cleanup completions and one WorkflowCancelled.

Raw connected artifact, retained until December 31, 2026. Artifact SHA256 af6606ae9c176f1710521f5255ae8f402319abec540025a9752f572a5763f3c2. JUnit SHA256 956336fd0e54b22fff2706198d941fb2f0481655cb5082d565316c015b4f2a1f. Teardown succeeded.

Diagnostic head e256cbdb951cd9da48dde05ea272324740b74779 passes 21 group checks, Ruff and mypy locally. Exact diagnostic Source run 37007105955 is in progress against the same Server and Native commits. Next action is to identify and repair the refused admission, then qualify the corrected exact tuple. PR remains draft and shared issue 136 remains open. No release or Cloud adoption occurred.

[rmcdaniel]

Diagnostic result narrows the group failure to application heartbeat

Diagnostic Source run 37007105955 at Python e256cbdb951cd9da48dde05ea272324740b74779 recorded 44 passed, one failed, one existing CLI-binary skip, zero errors, 503.474 seconds. Both nested group local preparations succeeded. An application heartbeat then raised ServerError with no machine-readable reason. This corrects the earlier suspected admission failure. The exact response status/body was not retained in that diagnostic revision.

Raw canonical histories, receipts and source provenance, retained until December 31, 2026. Artifact SHA256 b70b357f4affe977a1d3481d138885cdc5dd3584f0b8199220588beb4e0afcd8. JUnit SHA256 6cead844b9044c761d15a72f0bcd25ff3397207cd2e80f2676bac6f63e4cafec. Teardown passed.

Inspection also found a separate concrete progress-loss defect. Python sent authored heartbeat data under details. The prepared Server API consumes progress, so the earlier sequential case checked heartbeat count while losing its values. Current head f927f698aa2af0c882c0a701975fb485199153a1 sends progress and requires both connected fixtures to recover their authored phase values from canonical heartbeat history. Failed fixture transport now retains HTTP status and response body.

Local Python 3.12.15 checks pass 104 affected cases, zero skips, 20.64 seconds, plus Ruff and mypy. JUnit SHA256 bb1674e0b7d2925fbe7c0b1ce2d75740c043364104db140e0550bd01fdbea90a. The supervised progress regression passes current source (one pass, 2.70 seconds) and fails with the preceding runner bound read-only (one failure, zero errors, 2.02 seconds). It exercises the real supervised callback's heartbeat and canonical progress. The changed connected fixtures add the Server boundary proof.

Exact Source run 37008870970 is running with this head, Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native readonly overlay 850a64050b57d47181abeae86d163962c9047317. This field correction does not yet establish the cause or repair of the nested HTTP error. Next action is to inspect that response if it recurs and qualify the corrected exact tuple. PR stays draft and shared 136 stays open. No release or Cloud adoption occurred.

[rmcdaniel]

Repairing prepared group heartbeat persistence and SDK progress

Two concrete defects are corrected in the Source candidates:

1. Native's prepared heartbeat producer includes the complete authored group path, but ActivityHeartbeatRecorded did not permit its six canonical membership fields. The write throws before the heartbeat can persist. Runtime repair 673343b3194b8bf29a10eca29c493932db028bc1 admits those existing fields, adds a real nested local/local/timer heartbeat case, and retains an immutable heartbeat/cancellation replay fixture. The preceding 850a64050b57d47181abeae86d163962c9047317 validator rejects all six fields. The corrected validator accepts the fixture and still rejects an unknown producer key. PHP syntax checks pass. Fixture SHA256 cd65bf379e38af3da018978ef7d3e5135977d283de52344f12b280a8a203b5d6.
2. Python sent heartbeat values under the remote route's outer details field, while the prepared route expects progress.details. The earlier field-only correction exposed the missing inner envelope through an explicit invalid_local_activity_heartbeat refusal. Current Python a6b243e84532e56d04d799e7b1e2c387874c5e3e matches the established PHP/Server format. The connected sequential and nested cases now require authored phase values in canonical history. Failed fixture transport retains response status and body.

Intermediate Python run reported 43 passed, two failed, one existing CLI-binary skip, zero errors, 494.250 seconds. Raw results and provenance, retained until December 31, 2026. Artifact SHA256 cce4306a240659764a5e1ba6358f134eabf7d50cfbd41f5c277426caff15a9b1. JUnit SHA256 3d61735c3278fe2e8737d62f32268ea5493d7f60c3b80cd4c32b41fb80a40a88. Teardown passed.

Corrected Python passes 104 affected tests, zero skips, 20.64 seconds, Ruff and mypy locally. JUnit SHA256 d55d7322954c13ba6d416fe4496a3e878a08b17a4994ebc732fa2c7a18be94a0. Native's PR check passed MySQL smoke, unit contracts and executable corpus, with a formatting-only failure. Current Native 9c937051f61f9a99701d2d7d9f89da634a3bbf83 applies exactly the formatter's changes. Runtime and resource files are identical to 673343b3194b8bf29a10eca29c493932db028bc1.

Native full Source qualification and Python connected Source qualification are running. Python binds SDK a6b243e84532e56d04d799e7b1e2c387874c5e3e, Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native 673343b3194b8bf29a10eca29c493932db028bc1. The superseded unformatted Native full run was cancelled. Next action is to finish and repair these exact qualifications, then retain the complete group/cancellation proof. Both PRs remain drafts and shared 136 stays open. No release, Cloud adoption or paid provider change occurred.

[rmcdaniel]

Prepared groups and heartbeat persistence pass connected Source qualification

Run 37010610371 passes all jobs at Python a6b243e84532e56d04d799e7b1e2c387874c5e3e, Server c1bda7f81323e9939ef433c7ebca66f62365cdfd and Native readonly overlay 673343b3194b8bf29a10eca29c493932db028bc1. Python 3.10–3.12, lint, package, corpus and branch gates pass. Connected results are 45 passed, zero errors or failures, one existing unavailable-CLI skip, 509.950 seconds. The unrelated CLI skip is named in the JUnit artifact.

Both sequential and nested mixed local/timer cases persist the authored first and second heartbeat values through progress.details. Nested heartbeat history retains the complete outer and inner authored group positions. Results preserve binary bytes and cold replay without invoking completed callbacks again. This verifies the Python envelope correction together with Native's admission of the six canonical heartbeat membership fields.

Cancellation cases independently verify physical process exit without application heartbeats, owner SIGKILL during shielded cleanup, replacement-worker recovery, one canonical delivery, immutable duplicate identity/time/deadline and timely Cancelled closure.

Case	Original request	Original deadline	Cancelled	Elapsed	Physical callback exits
Sequential	2026-10-02T13:16:41.352767Z	2026-10-02T13:17:11.352767Z	2026-10-02T13:17:05.739330Z	24.386563 s	3
Atomic two-member group	2026-10-02T13:17:08.060390Z	2026-10-02T13:17:38.060390Z	2026-10-02T13:17:33.402430Z	25.342040 s	6

The group has one request, one delivery, two activity cancellations, two stop acknowledgments, two unknown-stop recoveries, two cleanup completions and one WorkflowCancelled event. Four cleanup admissions use distinct attempts but one unchanged request/root/delivery/deadline tuple. Unknown stop state remains explicit after owner loss. No application heartbeat or renewed cancellation budget appears in either cancellation history.

Raw histories, receipts, results, source provenance and image digests are retained until December 31, 2026. Artifact SHA256 7b7ae6fed325d2fabbd94d98593f85f877738529dc022784fd30d5c1fedd5f55. JUnit SHA256 f9886503a506ed41b92e56094bd80cf59e3c9c0c17c28486933d765571477562. Hosted test containers and network were removed successfully.

Current Native 16b722e74a10451217b1c16f25b5b1bcb1ebc45d differs from the qualified overlay only in test formatting. Runtime and resource files are byte-identical. Its PR quality, replay, unit and MySQL smoke gates pass. Local Python checks pass 104 affected cases with zero skips, Ruff and mypy. Earlier failed runs and counterexample evidence remain in this PR's comments.

Next action is portable activity cancellation policies and deterministic scopes, followed by the exact published PHP parent → Python child → Rust remote activity plus PHP local activity cascade and one coherent API/CLI/UI inspection view. This PR remains draft and shared issue 136 remains open. Published protocol 1.19 and packages are unchanged. Protocol 1.20 remains an explicit, unfrozen Source opt-in.

[rmcdaniel]

Pending Activity cancellation response qualified

Exact Python SDK source bf931959af5cc6615cb42ccd0ed279e7b3d33024, Server 2fc521b5f132c5a6d0909ec6239f2fbbc435bae6, Native readonly overlay bf047ce2e88e6c565eb6b80fae6559216524d41b.

The client validates pending Activity cancellation replies for Activity, LocalActivity, Parallel and SelectionHandle boundaries. It retains child pending validation and requires explicit claim release, matching task identity and null delivery fields. Malformed fields, unknown or non-string reasons, timers and unrelated child/activity kinds are refused before the worker defers delivery.

Exact Source CI passes every job, including Python 3.10–3.12, lint/type checks, package, Avro, corpus and target-branch qualification. The unit matrix covers seven valid reason/kind pairs, malformed fields for both reasons and unrelated-kind refusal. Ordinary exact-head CI also passes. The preceding runtime commit's Source run stopped at two Ruff line-length findings in tests, which this final head corrects. Its skipped integration job is not used as qualification evidence.

Connected qualification uses pytest tests/integration/ -v --capture=tee-sys --junitxml=integration-results.xml. It passes 45 cases, zero errors or failures, with one existing unavailable-CLI skip, in 490.243 seconds. It rechecks real callback stop without application heartbeats, local and nested group history, original cancellation identity/deadline, worker SIGKILL during shielded cleanup, fresh replacement and resumed cleanup on this tuple. Source qualification does not turn the unavailable CLI case into a pass.

Raw histories, process observations, JUnit, image identities and source provenance are retained until 2026-12-31. Artifact SHA256 24bbddc080f6c8ca74a19fb6638b0683de66aa34912cb8e6ea5571ec09a44be5. JUnit SHA256 dd5a52a3d4f1e2b3f0fb4e4bafde236721a58ae91b2f4da68bba89d544405923. Hosted task containers, images and network were removed successfully. No local Python runtime or package cache was created for these edits.

This qualifies the reply validator and rebinds existing connected component cases. Explicit portable Activity policy authoring and the required published mixed-language cascade remain separate gates. Protocol 1.20 remains an unfrozen Source opt-in, the published package is unchanged, this PR remains draft and shared issue 136 stays open. Next: author and admit explicit Activity policies before running their actual worker scenarios.

[rmcdaniel]

Remote Activity policies qualified in Python Source

Exact Python de0c476760524dcac3df35572bb7bd19d818a2d0, Server 8940fb8a866e24d7874947e850f7fc864fbdfd3c, Native readonly Source overlay 9f8cb61947997c1c60f294bdf559d02d8258cc18.

ScheduleActivity and WorkflowContext.schedule_activity() accept the typed CancellationPolicy enum and its portable strings. Both command encoders retain the explicit policy. Invalid policy and unbounded Abandon fail before suspension, and encoding revalidates mutable commands. Missing negotiated worker capability is diagnosed before submission. Omission preserves the historical Try default and wire shape. Local policy authoring remains unavailable.

The replayer previously accepted a change from recorded Wait to authored Try. A focused reproduction fails before repair and passes afterward. Replay now keeps the original canonical Activity policy through later events that omit it, checks ordinary/group/selection matching and cancellation delivery, and rejects malformed or conflicting history. The minimal activity-cancellation-policy-changed.json Source fixture executes through the official replayer. Frozen 1.19 fixtures are unchanged.

Complete exact Source CI passes every required job, including Python 3.10–3.12, full lint/type analysis, package and corpus. Connected qualification passes 50 cases, zero errors or failures, one existing unavailable-CLI skip, 562.214 seconds. JUnit contains 51 cases including that skip. Hosted stack teardown passes.

All five new connected policy cases pass without skips. Explicit Try and Wait physically stop both async and sync callbacks without application heartbeats, preserve original stop receipts and refuse stale completion/failure. Wait checks that the receipt precedes canonical workflow delivery. Bounded Abandon proves the callback process remains live after parent Cancelled closure, then commits independent completion under the original total deadline, retains parent cancellation and refuses a second outcome from the completed attempt. The four Try/Wait times are 11.225, 11.180, 11.656 and 11.652 seconds. Abandon takes 27.052 seconds. These are Source policy cases with distinct cleanup and total Activity budgets.

Local Python 3.12 focused replay, child-policy, cancellation and corpus qualification passes 181 cases, zero errors, failures or skips. Full Ruff and mypy pass. Commands: pytest tests/test_activity_cancellation_policies.py tests/test_child_workflow_policies.py tests/test_cooperative_cancellation.py tests/test_replay_regression_corpus.py -q --junitxml=..., ruff check src/ tests/, mypy src/durable_workflow/. The connected command is the existing complete integration suite using the exact server_commit and native_commit workflow inputs.

Raw scenarios, JUnit, source provenance and image authority are retained until 2026-12-31. Artifact SHA256 2b52d8eb89b69cc027136b61a9c174f99b90ec19f3b078157905949d86d20736. JUnit SHA256 ce705951fbc09a4b1cc6542c400fae14a215508bc6e926eb9ff60e4abfac0050.

PHP's matching Source policy cases are qualified. Rust is qualifying its matching consumer. Explicit local policies, independent local Abandon lifetime, scopes, coherent inspection and the published mixed-language cascade remain required. This PR remains draft, shared issue 136 stays open, protocol 1.20 remains explicit/unfrozen and published artifacts are unchanged.

[rmcdaniel]

Python deterministic remaining-time Source candidate

Head 8c1e937d2bab234a8a9cd1c1cea3aed7265aa9db adds CancellationContext.remaining() on the consumed replay clock while preserving the existing start-time WorkflowContext.now(). The deadline, root identity, reason, requester and lineage retain their portable metadata schema.

The helper initializes at committed delivery, advances through consumed blocking cleanup outcomes, and preserves synchronous side-effect and inline-local decisions between first execution and cold replay. Selection follows its committed winner, awaited handles use their own durable result or first cancellation receipt, and failed groups exclude later sibling outcomes. Recorded clock skew cannot increase the budget. Fractional seconds are retained and expiry clamps to zero. A detached snapshot, ended replay, or missing/invalid boundary timestamp fails without using host time.

Local affected checks passed 206 cases, including 12 helper cases and the two Git-backed public-boundary scanner cases, zero errors, failures or skips. Full Ruff and mypy across 29 source files passed. Affected JUnit SHA-256: 99536e899690bd8ad7d286aa38391caf3e4e3659f57464c4ade882667182d39b. The earlier broad non-integration run passed 1,993 cases with two existing skips, but its two scanner cases could not run in a container without Git. Both scanner cases passed in the final isolated Git-equipped container. A full current-suite pass is pending hosted CI.

The actual Python SIGKILL scenario now records the delivery budget in the original worker, requires an identical value and metadata in the replacement, and checks the post-cleanup value against committed delivery/completion timestamps and the original deadline. Source run 37065839981 is pending against Server 70807856b416b8792099108712cdd031e2c66c89 and Native 88ea6e46499332427d0f9a439768ae0bf86e8702. Ordinary current-head CI is pending too.

The qualified PHP helper and mixed cascade remains bound to its exact tuple, including preceding Python de0c476760524dcac3df35572bb7bd19d818a2d0. Next: inspect Python CI and actual remaining-time observations, then port Rust and audit embedded Native synchronous replay time. Explicit local policies, nested scopes, competitive qualification and the exact published mixed-language scenario remain required. All candidates remain drafts and shared #136 stays open. Protocol 1.20 remains unfrozen and published/default 1.19 is unchanged.

[rmcdaniel]

Python remaining-time qualification now covers both cleanup paths

Current head 0e1c427eae2f6120155a5cf3c9c1ae79ba7a5756 corrects the connected qualification without changing the helper implementation.

Preceding Source run 37065839981 passed all ordinary jobs, including Python 3.10–3.12, lint/types, package and regression corpus. Connected qualification had 49 passes / 1 failure / 1 existing unavailable-CLI skip. The failing assertion expected legacy inline cleanup to consume its later persistence timestamp. Its delivery and post-inline values were correctly equal at 119.462336 seconds, matching the documented first-execution/cold-replay contract. That assertion now requires equality and verifies the original delivery timestamp.

The existing sequential and atomic prepared-cleanup tests now capture the actual original and replacement worker's delivery values and metadata, require equality after SIGKILL, and verify the final value against committed cleanup completion and original deadline. Both already require stopped callbacks without application heartbeats, unknown-stop recovery, duplicate original identity/deadline, and Cancelled convergence before the original 30-second deadline. The new observations add no workflow commands.

Current local Ruff, mypy across 29 source files, all 12 helper cases, collection of the 28 affected connected cases and whitespace checks passed. Helper JUnit SHA-256: 315dd46d775a28ded6dd91f210e1e9cbef812b1932f155244f66d76ca26aa803. The unchanged core helper and broader replay selection passed 206 focused cases at the preceding head.

Current Source run 37067939883 is pending against Server 70807856b416b8792099108712cdd031e2c66c89 and Native ae1513bf78a175f4965e4fb8fc473fab60c35ec1. Current ordinary CI is pending. Preceding failure evidence remains retained until December 31, ZIP SHA-256 7a57250ed61bb39067fe8335a7f225ad702898cbd3085bdf9cf41f8977368775, JUnit 0038bcb79df81cb3a9c78e7739b3c753d2208aacb02e2a311e04bd96766de327.

Local venv, generated caches and the temporary Python image are removed after hosted handoff. Next: inspect actual current prepared/inline observations and Native qualification, then port Rust and finish explicit local policies/scopes/competitive and final published closure proof. All candidates remain drafts, shared #136 stays open, protocol 1.20 remains unfrozen and published/default 1.19 is unchanged.

[rmcdaniel]

Python consumed cleanup budget qualified

Exact Python 0e1c427eae2f6120155a5cf3c9c1ae79ba7a5756 passes Source 37067939883 against Server 70807856b416b8792099108712cdd031e2c66c89 and Native ae1513bf78a175f4965e4fb8fc473fab60c35ec1. Ordinary CI 37067887319 also passes. Python 3.10–3.12, lint/types, package, corpus and connected gates all pass.

Connected JUnit records 50 passes, zero errors/failures, one existing unavailable-CLI skip, 561.247 seconds. Both prepared-cleanup SIGKILL cases pass without skips. The actual original and replacement worker observations are byte-equivalent metadata and equal remaining time:

Cleanup	Original / replacement remaining	After committed cleanup	Cancelled after original request
Sequential prepared	28.858494 s / 28.858494 s	4.153142 s	25.952260 s
Atomic prepared group	28.857760 s / 28.857760 s	13.957172 s	16.197347 s

Each uses the original 30-second deadline, physically stopped callbacks without application heartbeats, one canonical delivery, actual owner-process SIGKILL during cleanup, fresh replacement recovery, unknown stop state preserved, and duplicate requests retaining original identity/deadline. The final helper value matches original deadline minus the latest consumed committed cleanup completion within one microsecond. Sequential/group histories contain zero application heartbeat records and respectively one/two callback stop acknowledgments. Both contain one Cancelled terminal before the original deadline.

The legacy inline recovery case correctly retains 119.502138 seconds at original delivery, replacement delivery and post-inline callback. Inline persistence is synchronous, so it does not change first-execution/cold-replay cleanup decisions. The earlier incorrectly written assertion remains recorded in its preceding failed run. The current core helper is unchanged by the qualification correction.

Raw artifact 11253209659 retains JUnit, actual observations, histories, callback process IDs, recovery receipts, exact source provenance, images and package provenance until December 31. SHA-256:

artifact-ZIP 57b27d263e0134ca3ebfd36319c6f7821faabe42f23498d3267a4bb526a1f78a
JUnit 452bdb7a37e9402d4a3d0028c490640b141bd47a261025b4a6c6e6873746de67
scenarios 9ef92456fd645f0992ca13c160827eaca2d627cc6d6be8a1ee8e92ad9d42481e
source-provenance a94ea560d37494dd08fa96a984e8e97dbfc8c6932e18a45280061e6bc9b5b658
image-provenance 7130ad258d735c3670c825ab09cc2d2dc81c071608857746dd459ab726bfa08b
package-provenance f158d21b57295c937223ee49838088cb006cda3a1f1beaa58f3caeedef164260

Native's exact-head full qualification is retained separately. Rust's consumed-budget helper is now under local qualification. Explicit local policies, nested scopes, fair competitive qualification and the complete exact published mixed-language cascade remain required. All cancellation candidates remain drafts, shared #136 remains open, protocol 1.20 is unfrozen and published/default 1.19 is unchanged.