Skip to content

Document the worker-function enable fence for production deployments #654

Description

@jumski

Scope

Fix the current one-function-per-flow production update guide.

Document one copyable safety sequence that disables the existing pgflow.worker_functions row, deprecates and drains its workers, deploys the replacement while disabled, restores the previous enabled state, and checks live heartbeats.

Private per-step, alias-version, and shared-queue rollout documentation belongs to #651, #648, and #652 respectively. Those future APIs do not block this P1 documentation fix.

Problem

The production Update Deployed Flows guide currently says to:

  1. deprecate old workers;
  2. deploy the new function;
  3. wait for cron to start it.

It does not disable the corresponding pgflow.worker_functions row before deprecation. While that row remains enabled, ensure_workers() may start another instance of the old deployed function between deprecation and deployment.

Required sequence

For one existing worker function:

  1. Read and record its current enabled value.
  2. Set enabled = false.
  3. Deprecate its live worker rows.
  4. Wait until every affected worker stops polling and finishes in-flight callbacks.
  5. Deploy the replacement while the function row remains disabled.
  6. Restore the recorded enabled value.
  7. Confirm the expected worker starts and reports a current heartbeat when re-enabled.

Do not infer the final state after deployment. Restore the exact state recorded before the fence.

Copyable checks

The guide must provide copyable SQL for:

  • recording the current enabled state;
  • disabling the function;
  • deprecating its live workers;
  • checking that polling and callbacks have stopped;
  • restoring the recorded state;
  • checking the replacement worker and heartbeat.

State what operators should inspect when the drain or restart does not complete. Do not add deployment automation or another activation protocol.

Documentation placement

Update the current production guide and link to it from the existing worker-management documentation.

Future features own their additional rollout instructions:

#651  complete affected per-step worker set and new concrete-version drain
#648  alias activation and rollback after new queue coverage
#652  shared-worker registry replacement and version drain

Acceptance criteria

Out of scope

  • Private per-step deployment manifests.
  • New concrete-version rollout and drain.
  • Alias activation or rollback.
  • Shared-worker registry deployment.
  • Cross-worker activation transactions.
  • Automatic deployment orchestration.
  • Automatic old-version removal.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationpkgs/websitepriority:p1Next batch: current correctness, user blocker, or operational safety

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions