Document required scheduler migration handoff protocol - #44
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e19d69dc6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| If rollout is disabled before activation, V1 confirms abort of its exact inactive stage before | ||
| resuming. If activation has committed, recovery must finish the handoff or perform an explicit |
There was a problem hiding this comment.
If the rollout flag is disabled after step 3 seals History but before step 5 activates CHASM, this instruction aborts the only staged destination and resumes V1 even though the protocol explicitly makes source admission permanently sealed. The schedule then either rejects all subsequent mutations or requires reopening the fence, contradicting the failover invariant and reintroducing the stale-request race. Abort-and-resume must be limited to pre-seal failures; once sealing commits, recovery must finish activation regardless of the rollout flag.
Useful? React with 👍 / 👎.
| History must replicate a source fence containing the workflow first-execution identity, migration | ||
| ID, admission phase, and final accepted-event watermark. CHASM must persist the same migration ID, | ||
| snapshot revision, covered watermark, and staged/active phase. Workflow state must preserve the | ||
| migration ID and last acknowledged revision across continue-as-new. |
There was a problem hiding this comment.
A user can call ResetWorkflowExecution with an explicit scheduler run after migration completes, rebuilding a new run from a workflow task before the migration ID and frozen state were recorded. Preserving state only across continue-as-new does not cover this branch, and the described History fence blocks mutation admission rather than timer processing or the V1 StartWorkflow activity, so the reset V1 run can execute actions concurrently with the active CHASM scheduler. The protocol must reject resets for a fenced chain or force every reset branch to inherit the terminal migration fence.
Useful? React with 👍 / 👎.
| The stage operation is idempotent for an equal ID and revision, replaces only an older revision | ||
| for the same inactive ID, and rejects a different ID. Staging starts no generator, invoker, | ||
| backfiller, or callback task. Finalize atomically activates only the matching staged revision. | ||
| Abort removes only a matching inactive stage and can never deactivate an active scheduler. |
There was a problem hiding this comment.
While a staged CHASM execution exists under the schedule's normal business ID, CHASM-first frontend routing can send PatchSchedule or UpdateSchedule to it before activation. The requirements only prohibit scheduler tasks, so an implementation may acknowledge and persist that mutation; step 4 can then replace the stage with a higher V1 snapshot revision and silently discard it. Require inactive stages to reject or redirect every external mutation, with only migration reconciliation, finalize, abort, and deletion allowed to access them.
Useful? React with 👍 / 👎.
| 2. Every API operation acknowledged before cutover is represented exactly once at the active | ||
| destination or remains durably pending there. |
There was a problem hiding this comment.
If a V1 PatchSchedule signal is accepted just before sealing but its response is lost, the signal is included in the final snapshot while a client retry with the same request ID can reach CHASM after activation. History's signal-request-ID deduplication is not part of the listed migrated state, and the CHASM patch handler currently performs an ordinary component update without attaching the frontend request ID, so a trigger or backfill can be applied twice despite this exactly-once invariant. Transfer the accepted request-ID deduplication state or retain a routing-side dedupe record across the handoff.
Useful? React with 👍 / 👎.
| 5. Delete and reverse migration fence older migration IDs so an ambiguous retry cannot resurrect | ||
| an earlier schedule incarnation. |
There was a problem hiding this comment.
A delete can reach CHASM before the concurrent stage write and get NotFound, then terminate V1 successfully, after which the in-flight stage commits with no workflow left to abort it. The current frontend deliberately invokes CHASM deletion before V1 termination, and the required durable state contains no tombstone or ordered incarnation generation that lets a later stage distinguish this delete from an earlier one; an opaque migration ID alone cannot be ordered. Require deletion to durably fence the source migration ID even when the destination is absent, otherwise an acknowledged delete can leave an orphan stage that blocks recreation of the schedule ID.
Useful? React with 👍 / 👎.
| 3. History seals mutation admission for that source incarnation and returns the last accepted | ||
| event watermark. Later callers receive a retryable redirect and are never acknowledged by V1. | ||
| 4. V1 drains through that watermark. If state changed, it stages a higher revision. |
There was a problem hiding this comment.
The workflow cannot currently prove that it has drained through the event ID returned by History: SDK signal channels expose only the decoded signal payload, not the source history event ID, and the patch/update payloads contain no admission sequence. Consequently V1 can empty the channels it sees and label a revision with the sealed watermark without knowing whether the watermark event has actually been delivered, recreating the loss window this protocol is intended to close. Require an SDK-visible sequence on each mutation or a deterministic barrier event that is delivered only after every event through the watermark.
Useful? React with 👍 / 👎.
| Protocol support must reach every possible active and failover History host before a new scheduler | ||
| workflow version can emit fence commands. Older create requests without migration fields retain | ||
| legacy behavior for existing histories, but the rollout flag must not select them for safe | ||
| cutover. An older host that cannot enforce the ingress fence is a capability-gating failure. |
There was a problem hiding this comment.
If migration was previously enabled, an old V1 local activity can already have passed its rollout guard and still be issuing a legacy create while CHASM phase support is deployed first. Because this compatibility rule keeps requests without migration fields on legacy behavior, that in-flight request can immediately activate CHASM before the History fence exists, preserving the acknowledged-signal loss race despite disabling new selections. The deployment protocol must disable migration, wait out or reconcile all pending legacy attempts, and only then install handling that can accept legacy migration creates.
Useful? React with 👍 / 👎.
1 participant
Decision
Workflow-to-CHASM migration must remain disabled until History can durably fence mutation
admission for the exact source incarnation. Scheduler-only staging, token checks, retries, and
channel drains cannot create a lossless terminal ownership boundary.
Required protocol
The protocol uses an immutable migration ID, staged snapshot revisions, a replicated History
admission fence with a final accepted-event watermark, and atomic destination activation. Staging
starts no scheduler work. Finalize activates only the matching revision after V1 drains through
the sealed watermark. Abort removes only an inactive matching stage.
The complete design, mixed-version deployment order, crash/failover behavior, 10x-load assessment,
and rejected partial fixes are in the commit-pinned
protocol note.
Why no production code is included
History can acknowledge a signal and deliver it into an SDK channel while the migration local
activity is pending. A successful workflow completion can then leave that operation unread. A
drain followed by another stage/finalize RPC yields again and recreates the race. An indefinitely
retained forwarder has no finite retirement point and develops an unbounded outbox whenever
source input rate meets or exceeds target drain capacity.
Shipping only migration IDs or staged CHASM state would reduce some failure windows while still
permitting acknowledged operation loss. This layer therefore records the precise prerequisite
rather than presenting a partial mechanism as a fix.
Compatibility
CHASM phase support must deploy first, followed by History fence replication and frontend
redirect handling. Only after every active and failover History host advertises support may a new
recorded scheduler workflow version activate the protocol. Old histories keep legacy behavior but
must not be selected for safe migration.
Stack