Skip to content

fix: enforce cleanup deadlines without a workflow worker - #604

Merged
rmcdaniel merged 2 commits into
mainfrom
fix/cooperative-cleanup-deadline
Oct 1, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
fix/cooperative-cleanup-deadline

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Customer outcome

An accepted cooperative cancellation reaches its terminal deadline even when the workflow worker has stopped or no compatible SDK worker is available. The next runtime repair pass closes the run and fences its outstanding tasks, activities and timers.

Cause and change

The watchdog excluded runs with ready or leased workflow tasks. A cleanup callback could correctly stop at its immutable deadline while its run stayed open, because no compatible worker remained to process a control task. Connected PHP and Rust Worker tests reproduced this against published Workflow 2.3.2.

Select expired accepted cleanup requests even when a workflow task remains open. Finalize cancellation transactionally through the existing executor without executing a workflow definition or dispatching another SDK task. Preserve the original request command in terminal history, lock the task before the run, retry transient database deadlocks, and report cancellation_deadlines_enforced in the repair-pass result. Ordinary execution and run timeout dispatch remains unchanged.

Verification

Focused repair command coverage passes: 15 tests and 117 assertions, including no open task, a ready foreign task, a leased foreign task whose lease extends beyond the cleanup deadline, queue scope and repeated-pass idempotency. MySQL feature checks pass: 10 tests and 136 assertions, including expiry at the exact recorded deadline, revocation of open resources and rejection of late completion. The deadline scan preserves fractional seconds.

The final head a8ba9e79782d882c157741eea8fcafab251274f6 passes every PR check, including coding style, static analysis, replay corpus and targeted unit / MySQL contracts. Local composer coverage runs all 2,146 unit cases with 16,401 assertions, five environment skips and 15 PHPUnit mock notices, with no errors or failures. Unit line coverage is 66.21%, above the required 60.00% floor.

Broader qualification at 6c333348293276903449e31dfdfa7bc6c1708864 passes all four MySQL shards, all four PostgreSQL shards, MariaDB, Laravel 9–13 upgrades, replay and combined coverage. Its remaining quality failure was the clock expression's formatting. The final head changes only that formatting and passes the quality gate. The exact merged main SHA must pass its complete matrix before tagging the patch.

Prepare Workflow 2.3.3 after the required gates pass. Verify its published package and update the affected Server candidate before repeating exact connected PHP, Python and Rust qualification. Ordinary Worker protocol remains 1.19. This patch is a foundation for the stronger cancellation model in #603.

Related: durable-workflow/.github#136. Keep the shared issue open until its stronger model, customer surfaces and published comparison evidence are complete.

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 1, 2026 19:54
@rmcdaniel
rmcdaniel merged commit 70d4fe4 into main Oct 1, 2026
16 checks passed
@rmcdaniel
rmcdaniel deleted the fix/cooperative-cleanup-deadline branch October 1, 2026 19:55
@rmcdaniel

Copy link
Copy Markdown
Member Author

Workflow 2.3.3 published deadline verification

Verified 2026-10-01 with a fresh Composer install of the published package.

  • Package: durable-workflow/workflow:2.3.3
  • Source reference: 70d4fe48efd7dd796c35c1078b3d5ac43f738f4f
  • Loaded watchdog: /work/vendor/durable-workflow/workflow/src/V2/TaskWatchdog.php
  • PHP: 8.3.35
  • Testbench: 11.3.0
  • PHPUnit: 12.5.37
  • Database: isolated SQLite file in container shared memory
  • Result: 15 tests, 117 assertions, zero errors, failures or skips

The installed package's repair-command suite includes
testCleanupDeadlineClosesForeignRunsWithNoWorkerWithinRequestedQueue.
It verifies closure without a compatible SDK worker, queue scoping, preserved
request identity and deadline, and idempotent repeated repair.

The accompanying JUnit report and Composer lockfile identify the executed
tests and installed dependencies. No source checkout or path repository was
used for this installation.

The exact merged source passed the complete database/Laravel matrix:
https://github.com/durable-workflow/workflow/actions/runs/36917693247

Fresh registry and supported Laravel upgrade checks passed:
https://github.com/durable-workflow/workflow/actions/runs/36919191410

Commands

In a PHP 8.3 tooling container with Composer and SQLite available, use an empty
directory and the included composer.json and phpunit.xml:

composer install --no-interaction --prefer-dist --no-progress
touch /dev/shm/published-native.sqlite
APP_ENV=testing APP_URL=http://localhost TMPDIR=/dev/shm \
DB_CONNECTION=sqlite DB_DATABASE=/dev/shm/published-native.sqlite \
REDIS_HOST= QUEUE_CONNECTION=sync CACHE_DRIVER=array CACHE_STORE=array \
php vendor/bin/phpunit --colors=never --log-junit published-deadline-results.xml

Consumer follow-through

Server 2.4.37 still pins Workflow 2.3.2. Server PR 291 is being updated to the
published 2.3.3 package before PHP, Python and Rust connected source
qualification is repeated. The stable Server image still needs its own
reviewed update and published-image verification. No Cloud deployment is
claimed.

This deadline repair is one prerequisite for shared issue 136. The required
published mixed-language cascade, successful cleanup after worker SIGKILL,
original 30 second budget, propagation and unified inspection remain open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants