fix: renew workflow heartbeats under the ownership fence

[rmcdaniel]

Customer outcome

Addresses #293. A heartbeat renews only the actual workflow task owner and attempt. Concurrent Native lease replacement cannot make an old worker receive a positive acknowledgment for the new lease.

Lock the namespace-scoped task, validate ownership and worker freshness, and renew inside one transaction. Keep current response fields, refusal reasons, pressure retries, protocol 1.19 and capabilities.

The regression uses two actual HTTP kernels with separate SQLite connections. An IPC barrier pauses before published Native renewal after the original ownership check. Both clocks cross the one-second lease expiry, allowing a replacement worker to attempt an actual Native claim. It does not edit the task lease, owner or attempt. Positive acknowledgment must match the durable claim. If replacement wins, the old heartbeat must instead refuse without renewing the replacement lease.

Qualification

• Unchanged current main d0692b171a0cabe6eefdc8bfc0f82d249e16128d, Native Workflow 2.3.1: regression fails. Original heartbeat returns renewed=true under attempt 1 while the replacement actually owns attempt 2.
• Fixed source: the regression passes. Either renewal retains its original durable claim, or replacement wins and the original heartbeat is refused. PHP 8.3 uses deferred SQLite transactions, so the test covers its valid refusal after a storage retry as well as the writer-lock outcome in newer PHP.
• Local PHP 8.5.11 with locked dependencies: Pint and 79 focused concurrency, ownership/error contract, success contract and poll backpressure cases pass with 4,286 assertions. The minimal local Composer runtime lacks the Redis extension, so the whole suite is not claimed as qualified. Normal repository CI supplies the supported full runtime.
• Server 2.4.36 and Helm 0.1.132 release metadata. Native Workflow remains 2.3.1. No cooperative cancellation activation or SDK package changes.

Required before delivery

• All normal exact-head repository CI and review at 8840d6ddb73df0a78a83e748e38cd3688897d90e. Full feature CI passes 2,253 cases and 47,227 assertions, and all protocol/corpus, polling, replay/query, MySQL/Postgres upgrade, chart installation and boundary gates pass.
• Merged at 898c51375ddcb2c25588eb63e41fed26a3fd61b2. Protected image/chart release run 36868020232 passes. Server 2.4.36 image index is sha256:74cdf7feae85052151de905d877a2e1ba5fcbfabf9ef50f3a8aab7c285685204, with both architectures verified. Helm 0.1.132 anonymous installation passes.
• Exact published-image ownership regression passes 1 case and 21 assertions. Published Server 2.4.35 fails the same regression. Published PHP/Python/Rust lifecycle run 36868798600 passes all 12 required cells with no findings or unproven cells. The report and raw comparison evidence are attached to the Server 2.4.36 release and recorded in Fence workflow heartbeats against concurrent lease replacement #293.
• Carried the fix into cooperative cancellation draft feat: add cooperative cancellation requests and claim-bound delivery #291 at 24b54c9f8b3f84cad93ec58d5d7cd2f95160458d. Local focused qualification passes 25 cases and 565 assertions. Remaining draft CI and coordinator qualification stay owned by feat: add cooperative cancellation requests and claim-bound delivery #291 and the cancellation issue.

Rollback is the preceding immutable Server image and chart. No database migration or protocol change is introduced. Cloud deployment remains its own private qualification decision.

[rmcdaniel]

Heartbeat ownership interleaving reproduced and fixed at source

The ordinary protocol 1.19 regression fails against unchanged main d0692b171a0cabe6eefdc8bfc0f82d249e16128d with locked published Native Workflow 2.3.1. Its two HTTP kernels perform actual Native status, claim and renewal. The fixture only provides an IPC barrier and clocks crossing the one-second lease expiry. It does not edit lease rows or fabricate Native results.

Observed baseline:

{"heartbeat":{"workflow_task_attempt":1,"lease_owner":"original","renewed":true},"replacement_claim":{"workflow_task_attempt":2,"lease_owner":"replacement"}}

The same Native task ID was present in both responses. The positive heartbeat therefore identified a claim that no longer owned the task.

The fix in #294 head 226025ffe3e0c1af98a77c01932a50f41799d8d8 holds the namespace-scoped task lock across ownership validation and Native renewal. Local Pint and 79 focused protocol, ownership/error, success and poll-pressure cases pass with 4,286 assertions. The controlled race now retains the original durable owner/attempt and the replacement poll returns no task. Command: php vendor/bin/phpunit --filter 'WorkflowTaskHeartbeatRaceTest|WorkerProtocolOwnershipErrorContractTest|WorkerProtocolSuccessContractTest|SqliteWorkerPollLockPressureTest|WorkerPollBackpressureTest'.

Cooperative cancellation draft #291 carries the same locked renewal, plus the pending observation read inside that transaction, at 6be6cd39 (full exact revision available in its CI). Its 24 focused cancellation protocol and concurrency cases pass locally with 545 assertions, preserving the original request/deadline and one canonical request without inventing delivery.

Normal CI is running for both heads. This source result does not claim a published image or Cloud deployment. Next finish exact-head CI/review, publish Server 2.4.36/Helm 0.1.132 independently of cooperative cancellation and verify the affected published worker cells. Rust coordinator qualification then uses the requalified #291 head. Default protocol 1.19 and ordinary capabilities remain unchanged.

[rmcdaniel]

Published Server 2.4.35 reproduces the ownership defect

The same committed regression ran inside the unchanged published Server image durableworkflow/server@sha256:49560f7f861271931125348a9d01638cd722e13ee976b5b732ef122548f15dab with its own PHP 8.3.35, Laravel and Native runtime code. PHPUnit 11.5.55 and Mockery 1.6.12 were mounted as separate test tools, together with the repository's tests. No application or vendor file was replaced. Reflection receipts verify that the controller and Native bridge load from the artifact, and both artifact provenance records name Native 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387.

The two actual HTTP kernels reproduce:

{"heartbeat_status":200,"heartbeat":{"workflow_task_attempt":1,"lease_owner":"original","renewed":true,"reason":null},"replacement_claim":{"workflow_task_attempt":2,"lease_owner":"replacement"}}

Both responses identify the same actual Native task. The unchanged published image therefore fails the ownership assertion, with 16 assertions reached. SQLite clocks and an IPC barrier control the interleaving; the harness does not edit any lease, owner or attempt row.

The original full CI result exposed an overly strict test assertion, not a valid positive acknowledgment. PHP 8.3 uses deferred SQLite transactions even when transaction_mode is configured, allowing replacement to win. The fixed transaction retries and returns a correct owner-mismatch refusal. The regression now requires either a positive acknowledgment matching the durable original claim or that exact refusal together with proof of the replacement claim. The same change preserves the cancellation request and cleanup deadline in #291. This still fails the old published image.

Next complete revised-head CI and review, publish Server 2.4.36, repeat this exact-image regression and verify the published PHP/Python/Rust lifecycle tuple before closing #293.

[rmcdaniel]

The revised regression passes in the actual PHP 8.3.35 runtime with only the fixed controller supplied from source head 8840d6d: one case, 21 assertions. This exercises the valid stale-owner refusal under deferred SQLite transactions. PHP 8.5 local source also passes the writer-lock renewal outcome. The unchanged published image still fails with a positive old-owner acknowledgment after an actual attempt-2 replacement. Normal exact-head CI is running. No new published-image pass is claimed before publication.

[rmcdaniel]

Reviewed exact source 8840d6d against main d0692b1. The runtime change is the namespace-scoped lock plus ownership validation and Native renewal in one retryable transaction, preserving existing refusal and pressure responses. The two test files exercise actual claims and both valid race outcomes. The remaining changes synchronize the single source release record to Server 2.4.36 and Helm 0.1.132, with no dependency version changes. Full feature CI 36866148513 passes 2,253 cases and 47,227 assertions, along with protocol/corpus checks. MySQL/Postgres rolling-upgrade checks, replay/query HTTP topology, chart lint/install and public boundary checks pass. Final polling qualification is still running. After it passes, merge this reviewed head, publish the immutable release and verify the exact image and published worker tuple before closing #293.

[rmcdaniel]

Delivered in Server 2.4.36 through merged #294 at 898c51375ddcb2c25588eb63e41fed26a3fd61b2.

• The unchanged published Server 2.4.35 image fails the controlled ownership race. The unchanged published 2.4.36 image passes 1 case and 21 assertions, using its own controller and published Native 2.3.1 code.
• Protected image/chart publication passes both supported image architectures, bare-image readiness, source-free Compose bootstrap, protocol catalog convergence and anonymous clean Helm installation.
• Published PHP/Python/Rust lifecycle qualification passes all 12 required cells, with no unproven cells or findings. Exact tuple: Server 2.4.36, Native 2.3.1, PHP SDK 2.1.6, Python SDK 2.3.7, Rust SDK 2.1.2, CLI 2.1.3 and Waterline 2.0.9.
• Server OCI index: sha256:74cdf7feae85052151de905d877a2e1ba5fcbfabf9ef50f3a8aab7c285685204. Helm 0.1.132 package: sha256:67b8c865c836039ec2639bd068a1b001b7062e9bde5361d45adbd6469cb808e2.
• Full reviewed-source feature CI passes 2,253 cases and 47,227 assertions. The normal ownership/refusal, pressure, protocol/corpus, polling, replay/query, MySQL/Postgres upgrade and chart gates passed before merge.

The qualification report and raw before/after evidence with reproduction tooling are release assets. Both were downloaded again and their SHA-256 hashes matched the uploaded originals. The evidence archive hash is 3d4e27fe5debdf595cf5bab2495ca4d97cdb830156c7a72355001fcb768663b9.

Default protocol remains 1.19. No SDK release or database migration is required. Helm's immutable default is updated. Inspection found no affected public consumer pinned to the preceding Server 2.4.35 image that requires a separate update. Cloud deployment qualification remains separately owned.

The merged remote branch was deleted and verified absent. The fix is also carried into cancellation draft #291 at 24b54c9f8b3f84cad93ec58d5d7cd2f95160458d, where further cancellation coordinator and source qualification remain visible. This issue's published artifact acceptance is complete.