Persist physical queue identity for flow tasks

[jumski]

Summary

Separate physical queue identity from concrete flow identity throughout the SQL Core while preserving today's one-flow/one-queue behavior.

This issue provides the task-identity foundation for private per-step queues in #651. A future #652 may reuse that identity, but it owns any metadata required for explicit shared queues. This issue adds no public routing API and no queue registry.

Dependencies

• Parent epic: Epic: staged per-step queues, aliases, and shared queue routing #653.
• Land after Remove ControlPlane and migration-based flow compilation #647 makes startup compilation the only deployment path.
• Completed prerequisite: Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #638 was fixed by Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #649 on 2026-08-28; keep its task-before-queue lock order.
• Land Failed runs leave unfinished sibling tasks queued or started instead of cancelled #645 first because it changes the same failure, archival, and stalled-recovery paths.
• Land requeue_stalled_tasks() uses flow timeout instead of effective step timeout #621's effective-timeout fix first because it changes queue-aware recovery.
• Land Make start_tasks() apply task visibility before handlers run #656's start_tasks() visibility-execution fix first because this issue changes the same claim path.

Keep those fixes in separate releases. This issue consumes their corrected behavior rather than rebasing old queue assumptions over them.

Current problem

The SQL Core currently uses flow_slug as queue identity in every path:

• flow creation provisions a same-named PGMQ queue;
• ready tasks are sent to that queue;
• task claiming treats one argument as both flow and queue;
• completion, failure, retry, skip, recovery, pruning, and deletion infer the queue from the flow.

PGMQ message IDs are queue-scoped. The durable identity is:

(queue_name, message_id)

A message ID without its queue is not globally meaningful.

Data model

Add resolved queue identity to definitions and immutable queue identity to runtime tasks:

pgflow.steps
  queue_name text not null

pgflow.step_tasks
  queue_name text not null
  message_id bigint null

Queue names stored by pgflow are lowercase canonical physical identities and must satisfy the existing 47-character compatibility limit.

Keep the task primary key:

(run_id, step_slug, task_index)

Add a partial unique index suitable for queue-message lookup:

create unique index ...
on pgflow.step_tasks (queue_name, message_id)
where message_id is not null;

message_id remains nullable. Existing cleanup tests deliberately cover tasks without a queue message; this issue must not redefine that state.

Do not add queue_name to runs or step_states. Do not add a pgflow.queues registry solely for generated private queues.

Deferred manual completion

#661 explores queue-less tasks completed by trusted external code. This issue intentionally models only queue-backed execution and keeps its queue identity non-null.

That choice is scoped to #650. It is not a rejection or final storage decision for #661. If the manual-task design advances, it must explicitly reconcile this model through nullable identity, an execution mode, or another migration. #661 is not a prerequisite for this issue.

Generated private queue ownership

The current stages create only deterministic private queues. Their persisted flow definition is sufficient ownership evidence:

plain flow
  generated default queue = lower(flow_slug)

step-queued flow in #651
  generated queues = the complete persisted step route

A generated queue may be provisioned or reused only when one of these conditions holds:

• the physical queue is absent and the compiler creates it while persisting the definition;
• an existing concrete definition has the exact matching persisted default or step route and the PGMQ objects are valid.

For a missing concrete definition, an existing physical queue with the generated name is a collision. Reject it instead of silently adopting it. Also reject a generated name already derived or referenced by another concrete flow.

Do not duplicate PGMQ metadata spelling in pgflow. Resolve the exact pgmq.meta.queue_name at metadata-sensitive operations such as drop_queue(). Require exactly one case-insensitive metadata match and reject aliases such as distinct Orders and orders rows because they address the same physical tables.

This derived model is intentionally limited to generated private queues. If #652 is adopted, it must add only the metadata needed for explicit shared queues and adoption without changing task identity again.

Provisioning boundary

The complete compiler preflight is the primary provisioning boundary:

compile complete definition
  -> resolve every required canonical queue name
  -> validate all names, collisions, PGMQ metadata, and physical objects before mutation
  -> persist the flow identity
  -> provision missing generated queues atomically
  -> create the steps with the complete route

create_flow()
  creates only the flow definition

add_step()
  defaults omitted queue_name to lower(flow_slug)
  uses the same generated-queue validation and provisioning path
  stores queue_name

start_ready_steps()
  performs no queue DDL

A plain flow with zero steps still provisions its generated default queue for backward compatibility. #651 rejects withStepQueues() for an empty flow.

Task creation

start_ready_steps() must:

1. Read each ready step's steps.queue_name.
2. Group messages by queue where batching needs it.
3. Send each batch to its resolved queue.
4. Store the same queue name on every inserted task.

The task snapshot never changes after insertion.

Queue-aware operations

Every PGMQ operation must use the task queue snapshot rather than the run's flow slug. This includes:

• claiming and visibility changes;
• completion and late-callback archival;
• retries and exhausted-task archival;
• condition failures and skip cascades;
• stalled-task recovery and permanent-stall handling;
• maintenance pruning;
• flow deletion.

Operations over several tasks must group by queue_name.

Separate the worker subscription from handler identity in claiming. The final claim boundary must carry at least:

queue_name
flow_slug
message_ids
worker_id

Preserve a compatibility wrapper for the existing plain-worker SQL signature. It resolves the flow's canonical default queue and delegates to the queue-aware boundary.

Represent PGMQ bigint message IDs as decimal strings at the JavaScript boundary. Cast them to bigint[] only in SQL calls; do not rely on unsafe JavaScript number precision.

#651 may add an exact step selector without changing queue identity again.

Deletion

delete_flow_and_data() must:

1. archive or remove active messages using each task's queue snapshot;
2. collect the concrete flow's generated queue names from its persisted default or step route;
3. delete runtime rows and step definitions while retaining the flow identity row;
4. resolve exactly one matching pgmq.meta.queue_name for each generated queue;
5. drop only those generated queues;
6. delete the concrete flow identity row last.

A missing, ambiguous, malformed, or differently referenced queue must fail safely instead of dropping an uncertain physical resource.

If #652 later introduces explicit shared queues, it must extend deletion so those queues are never dropped with one flow.

Upgrade preflight

Take the migration lock and explicit table locks before inspection so concurrent pgflow or PGMQ activity cannot invalidate the checked state.

Before mutation:

• detect case-insensitive physical queue aliases in pgmq.meta;
• confirm each metadata row has its expected queue, archive, and sequence objects;
• detect duplicate non-null (queue_name, message_id) identities;
• reject active PGMQ messages without matching pgflow task identities;
• confirm every legacy flow has exactly one unambiguous generated default queue;
• reject generated names already associated with another concrete flow;
• return bounded counts and sample keys with repair hints;
• leave the database unchanged on failure.

Backfill queue_name = lower(flow_slug) for existing steps and tasks, including tasks whose message_id is null. Resolve legacy exact spelling from pgmq.meta when needed rather than persisting a duplicate copy.

Add a mixed-case PGMQ 1.5.1 migration fixture that creates, migrates, archives, drops, and recreates a camelCase queue. Check metadata, queue tables, archive tables, and sequences after every operation.

Compatibility

This issue makes no routing change:

step queue = lower(concrete flow_slug)
task queue = lower(concrete flow_slug)
worker queue = the same canonical physical queue

A mixed-case worker argument still reaches the same PGMQ tables. Persisted step and task identity always use lowercase canonical names.

Support this rolling-upgrade matrix:

new database + old plain worker
  -> supported

new database + new plain worker
  -> supported

old database + new queue-aware worker
  -> fail before registration or polling

Plain flow workers and starts remain source-compatible. This stage must remain operational if #651 is delayed; it is not a stable release by itself.

Acceptance criteria

• steps.queue_name and step_tasks.queue_name store lowercase canonical queue identity without a new queue registry.
• step_tasks.message_id remains nullable.
• Non-null queue messages use unique (queue_name, message_id) identity.
• Existing steps and tasks backfill to lower(flow_slug).
• Generated queue validation rejects silent adoption, cross-flow collisions, malformed objects, and case aliases.
• An exact persisted definition may verify and reuse its generated queues idempotently.
• Metadata-sensitive operations resolve the exact spelling from pgmq.meta and reject zero or multiple matches.
• The complete compiler preflight provisions the default queue for an empty plain flow.
• create_flow() and start_ready_steps() perform no queue DDL.
• add_step() resolves and provisions the default queue through the shared generated-queue path.
• start_ready_steps() snapshots each resolved queue.
• Every visibility, archive, retry, skip, recovery, pruning, and deletion path uses task queue snapshots.
• Multi-task operations group by queue.
• JavaScript represents PGMQ message IDs without precision loss.
• Flow deletion drops only queues derived from that concrete flow's persisted private route.
• Upgrade preflight locks before inspection and leaves the database unchanged on any failure.
• Previous-version fixtures cover normal rows, null IDs, duplicate identities, active unmatched messages, missing PGMQ objects, and mixed-case metadata.
• Old and new plain workers pass the rolling-upgrade matrix.
• Existing one-flow/one-queue behavior passes unchanged end to end.

Out of scope

• A general queue registry.
• Persisted shared-queue ownership or adoption metadata.
• Per-step queue naming or workers.
• Public custom queue routing.
• Shared queue consumption.
• Multi-flow worker registries.
• Mutable routes.
• Automatic deletion of shared queues.
• Manual tasks or external completion, tracked separately in Design queue-less manual completion tasks (queue: false) #661.
• The requeue_stalled_tasks() uses flow timeout instead of effective step timeout #621 effective-timeout fix.