Skip to content

Make start_tasks() apply task visibility before handlers run #656

Description

@jumski

Summary

pgflow.start_tasks() computes effective per-task visibility delays and calls pgflow.set_vt_batch() in an unreferenced SELECT CTE. PostgreSQL may skip an unreferenced SELECT CTE, so a claimed task can keep only the shorter visibility set by the initial PGMQ read.

The message may become visible while its step_tasks row remains started. Current workers repeatedly read that message and return fewer tasks than messages. Future per-step workers must not mistake this healthy duplicate visibility for unsupported work and stop.

Current path

pkgs/core/schemas/0120_function_start_tasks.sql contains:

timeouts as (
  select
    task.message_id,
    task.flow_slug,
    coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay
  from tasks task
  ...
),
set_vt_batch as (
  select pgflow.set_vt_batch(...)
  from timeouts
)
select ...
from tasks ...

The final query does not reference set_vt_batch. This does not guarantee that pgflow.set_vt_batch() runs.

Required behavior

For every task that changes from queued to started:

  1. compute the same effective timeout used by stalled recovery;
  2. extend that message's visibility before returning the task to a handler;
  3. increment the attempt exactly once;
  4. return no task whose visibility extension failed;
  5. preserve atomic behavior for the claimed batch.

A message that becomes visible while its exact task remains started is a benign duplicate. Reading it must consume no new attempt and must not stop the worker.

Implementation boundary

Make the visibility side effect structurally required. Use PL/pgSQL PERFORM, a referenced CTE whose result the final statement consumes, or another statement form whose execution does not depend on demand for an unreferenced SELECT CTE.

Keep timeout semantics aligned across:

initial PGMQ read visibility
start_tasks() effective visibility extension
#621 stalled-task threshold
requeue_stalled_tasks() visibility reset

Do not fold queue-per-step routing into this fix. #650 will later add canonical queue_name and group visibility updates by queue.

Tests

  • A claimed task's PGMQ visibility uses coalesce(step.opt_timeout, flow.opt_timeout) + 2 before start_tasks() returns.
  • A shorter step timeout overrides the flow timeout.
  • A longer step timeout is not shortened to the flow timeout.
  • A null step timeout falls back to the flow timeout.
  • A visibility-update failure rolls back the task transition and attempt increment.
  • A repeatedly visible message for a started task consumes no additional attempt.
  • A mixed batch cannot return a task whose visibility update did not execute.

Relationship to other work

0.15.1 release integration

Release coordination is tracked by #666. The final #656 PR must replace the three unreleased Core migrations currently on main plus its temporary development migration with one Atlas-generated task_lifecycle_hardening migration against the 0.15.0 baseline.

Atlas does not infer migration-only data repairs. Before deleting the old migrations, extract and preserve this carry-forward set:

  1. the final task-status constraint, including skipped and cancelled;
  2. every final changed function definition and attribute;
  3. the Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #638 repair from active tasks under skipped steps to skipped;
  4. the Failed runs leave unfinished sibling tasks queued or started instead of cancelled #645 repair from remaining active tasks under failed runs to cancelled;
  5. repair order: skipped-task repair before failed-run cancellation.

Audit function signatures, return types, SECURITY DEFINER, SET search_path, grants, and statement order. Regenerate atlas.sum after any migration-only manual edits.

Only disposable databases consumed the current unreleased migrations. The upgrade fixture must start from 0.15.0, apply the consolidated migration, and check both historical repairs plus final runtime behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpkgs/corepriority:p1Next batch: current correctness, user blocker, or operational safety

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions