Design queue-less manual completion tasks (`queue: false`)

[jumski]

Summary

Explore an explicit way to represent work that waits for an external system or person instead of a pgflow worker handler.

The current preferred direction is a handlerless task without a physical queue:

flow.step({
  slug: 'wait_for_approval',
  queue: false,
  output: typeHint<Approval>(),
});

When the step becomes ready, pgflow creates a durable task but sends no PGMQ message. Trusted server-side code later completes or fails that task, after which normal workflow progression continues.

Status: This issue records a deferred design direction. The public API, persisted representation, exact semantics, and release sequence are not final. This is not currently a committed stage of #653.

Related to #653 and #650. Motivated in part by the durable external-signal discussion in #660.

Problem

pgflow currently assumes that ready work is placed on a PGMQ queue and completed by a worker handler. That does not directly model:

• human approval or review;
• a webhook or provider callback;
• externally performed work;
• a durable pause between workers;
• application-owned decisions that must release downstream steps.

Applications can build these gates beside pgflow, but then they must own task correlation, durable state, duplicate delivery, run-state checks, and downstream progression.

The core requirement is smaller than a general signal system: represent one explicit workflow task whose completion belongs to trusted external code rather than a worker.

Candidate authoring model

A possible shared routing primitive is:

queue?: string | false

Candidate meaning:

• omitted routing uses the normal queue default;
• a string selects queue-backed worker execution;
• false selects queue-less manual completion.

The exact spelling remains open. Other SDKs do not need to copy the TypeScript syntax as long as they compile to the same core task semantics.

Manual definitions would be handlerless. A possible TypeScript shape is:

flow.step({
  slug: 'wait_for_approval',
  queue: false,
  output: typeHint<Approval>(),
});

flow.array({
  slug: 'wait_for_batch_approval',
  queue: false,
  output: typeHint<Approval[]>(),
});

flow.map({
  slug: 'wait_for_each_approval',
  array: 'requests',
  queue: false,
  itemOutput: typeHint<Approval>(),
});

typeHint<T>() is one candidate for preserving TypeScript output inference without a fake handler. It would be compile-time-only authoring data, not a runtime validator or persisted value. The final API may use another mechanism.

Flow-level queue: false is not proposed initially. An explicit manual step keeps the wait visible in the DAG and avoids turning every step into external work accidentally.

Candidate task model

The important invariant is behavioral rather than a settled column layout:

manual task
  real pgflow task identity
  no physical queue
  no PGMQ message
  no worker handler
  durable until completed, failed, or terminalized

One possible representation is:

status          = started
queue_name      = NULL
message_id      = NULL
queued_at       = NULL
started_at      = now()
attempts_count  = 1
last_worker_id  = NULL

This representation conflicts with #650's current non-null queue proposal and is therefore not a decision. Before implementation, the design must choose whether manual execution uses nullable queue identity, an explicit execution mode, or another representation that preserves queue-backed task invariants.

The logical task address remains:

run_id + step_slug + task_index

Candidate behavior by step kind:

• a manual .step() creates one task at index 0;
• a manual .array() creates one task whose output is the complete array;
• a manual .map() creates one task per input item;
• map output aggregates in task_index order;
• an empty map keeps existing taskless completion behavior.

Candidate lifecycle

1. Dependencies and conditions make the manual step ready.
2. pgflow creates its task rows without calling PGMQ.
3. Trusted server-side code completes or fails an addressed task.
4. pgflow updates task and step state under the normal run locks.
5. Existing progression starts dependencies, evaluates conditions, handles taskless cascades, and completes or fails the run.

Manual APIs should use the immutable task snapshot rather than the current step definition when deciding whether a historical task is manual.

Worker-owned completion and failure APIs must reject manual tasks. Stalled-worker recovery must ignore them because there is no worker or queue message to recover.

Candidate completion and failure APIs

Possible SQL entry points:

pgflow.complete_manual_task(
  run_id uuid,
  step_slug text,
  task_index integer,
  output jsonb
)

pgflow.fail_manual_task(
  run_id uuid,
  step_slug text,
  task_index integer,
  error_message text
)

Client SDKs may expose matching methods. Any language may call the same server-side boundary through an application service or database client.

A transition should lock and validate the task, step state, and run. A new transition should require an active run, active step, active manual task, and matching task identity.

Candidate idempotency rules:

Existing result	Repeated operation
completed with equal JSON output	return the stored task without progressing twice
completed with different output	conflict
failed with the same error	return the stored task without progressing twice
failed with a different error	conflict
completed followed by failure, or the reverse	conflict

Manual failure would not retry because no worker owns the task. It should behave like final exhaustion and follow the step's configured whenExhausted behavior. Queue-backed downstream work keeps its normal retry semantics.

The exact error contract is open. Candidate stable application codes include:

manual_task_not_found
manual_task_not_manual
manual_task_not_active
manual_task_conflict

Shared progression, separate policy

Manual and worker APIs should not duplicate DAG progression logic.

A likely implementation would extract private progression internals for task and step terminal transitions, map aggregation, dependency release, condition and taskless cascades, and run completion. Queue-backed wrappers would retain archive, visibility, and retry behavior. Manual wrappers would call the same progression core without PGMQ side effects.

The public entry points should remain separate. Reusing worker failure behavior could accidentally put a queue-less task back into queued state without a queue or message.

Security boundary

Manual completion is server-only by default.

Candidate requirements:

• revoke execution from PUBLIC;
• grant no access to anon or authenticated by default;
• use caller privileges rather than a broad privileged function where possible;
• require explicit grants to trusted server roles;
• let applications expose their own authenticated approval or callback endpoint.

Applications remain responsible for caller authorization and domain payload validation. A TypeScript type hint alone does not validate JSON received from an external caller.

Why prefer an explicit manual step

The wait remains visible in the workflow:

create_order → wait_for_approval → charge

Compared with an inline await_signal() handler, the candidate model has different tradeoffs:

Concern	Inline handler wait	Explicit manual task
Wait location	hidden inside handler code	visible in the DAG
Worker use	handler starts and later parks	no handler starts
Resume	handler runs again from the top	downstream steps start
Side effects	pre-wait work must be idempotent	no pre-wait handler work
Runtime support	each SDK needs special control flow	all SDKs call the same task API
State changes	started → waiting → queued → started	started → completed or failed
Early payload	may be buffered by a signal store	needs an optional later layer
Timeout	may use a built-in sweeper	application-owned initially

The explicit model avoids implying that a suspended function invocation survives. Durable inline waits cannot preserve a JavaScript Promise, Python stack, process memory, or Edge Function invocation across crashes and time limits; they must exit and replay or use another task boundary.

Multi-language direction

This design aligns with a possible future where pgflow core coordinates multi-language flows and each step may run on a worker suited to that task.

For example, a flow authored in TypeScript could have one step processed by a Python GPU worker, another by a TypeScript worker, and an approval step completed by application server code. The database task protocol, rather than a TypeScript-specific exception or handler replay mechanism, would own progression.

This issue does not commit pgflow to that runtime architecture. It records a constraint worth preserving: manual completion should not depend on one language's control-flow semantics.

Relationship to durable signals

A manual task covers the common approval, webhook, and external-work gate once the task exists. It does not by itself provide the complete signal behavior discussed in #660:

• buffering a payload before the target task exists;
• several named signals for one task;
• replacing or deduplicating undelivered payloads;
• a core-managed wait deadline;
• an inline handler API.

Those capabilities can be evaluated separately. A later signal layer could buffer an early payload and complete the manual task when it becomes ready. A timeout service could fail an overdue manual task. Neither feature requires replaying a handler or keeping a worker invocation alive.

Unknown-length event streams and general event correlation remain outside this task model.

Relationship to the queue epic

This issue is related to #653 but is not a prerequisite for its current stages.

• Persist physical queue identity for flow tasks #650 establishes durable identity for queue-backed tasks. It currently proposes non-null queue identity and must not be read as settling manual-task storage.
• Add private per-step queues with typed step workers #651 and Add explicit shared queues with multi-flow dispatch #652 concern worker placement for queue-backed work. Manual tasks have no worker placement.
• Add explicit shared queues with multi-flow dispatch #652 explicitly keeps manual steps and external completion out of its current scope.
• A later design must define whether named-to-manual or manual-to-named changes require a new concrete flow version.

No current queue-routing stage should silently expose manual behavior. No current stage needs to implement it before the design and a real use case are validated.

Design spike

Before public implementation, one focused prototype should prove:

• a manual task waits durably without a PGMQ message or worker slot;
• completion and failure progress single, array, and map steps correctly;
• duplicate, conflicting, and terminal-run races cannot advance work twice or revive a run;
• task indexes and run identities isolate concurrent approvals;
• one language-neutral server boundary can complete work created by a differently authored flow.

The spike should record an adopt, revise, or stop recommendation and compare the result with the inline signal design from #660.

Open decisions

• Final authoring API, supported step kinds, and output typing or validation.
• Persisted representation and migration path alongside Persist physical queue identity for flow tasks #650's queue registry.
• Discovery, correlation, timeout, and optional early-signal buffering.
• Error, idempotency, failure, and whenExhausted contracts.
• Structural versioning, observability, deployment, and security documentation.

Out of scope

• Holding a handler, Promise, Edge Function, or database connection open while waiting.
• A generic event bus, unknown-length event stream, or general correlation engine.
• Automatically generated public callback URLs or provider-specific webhook helpers.
• Shipping manual tasks as part of the currently committed Epic: staged per-step queues, aliases, and shared queue routing #653 stages.
• Finalizing multi-language worker routing in this issue.