docs: known limitations of the state machines

content/en/documentation/for-adopters/control-plane/state-machines/_index.md

@@ -0,0 +1,98 @@
+---
+title: State Machines Known Limitations
+description: Accepted limitations of the poll-based and task-based state machines of the control plane, their impact and how to mitigate them.
+weight: 30
+---
+
+<!-- TOC -->
+  * [Common to both state machines](#common-to-both-state-machines)
+  * [Poll-based state machine](#poll-based-state-machine)
+  * [Task-based state machine (NATS)](#task-based-state-machine-nats)
+<!-- TOC -->
+
+The control plane drives transfer processes, contract negotiations and policy monitor entries through one of two state
+machines:
+
+- the **poll-based** state machine, which periodically leases a batch of entities per state from the store;
+- the **task-based** state machine, which processes a task published to NATS for every state change.
+
+Both were reviewed for robustness and for running several replicas against the same database ([#6105] and [#6116]).
+The issues found were either fixed or, when their impact is low or a fix is not worth its cost, accepted. This page
+lists the accepted ones: what happens, when, the impact, and how to mitigate it where possible. See also the
+[stateful entity versioning] decision record, which describes how leases and versions protect the entities, and the
+[Tuning](../../tuning/_index.md#state-machines) page for the settings mentioned here.
+
+## Common to both state machines
+
+- **Synchronized clocks.** Lease expiry and retry delays compare timestamps written by one replica with the clock of
+  another. Replicas are expected to keep their clocks synchronized (e.g. with NTP).
+- **Duplicate protocol messages.** A step can run twice: on the poll-based state machine, when the lease of an entity
+  expires while it is still processed; on the task-based one, when a task is delivered again after a failure. The
+  counter-party can then receive the same message twice, with a different message id. Counter-parties are expected
+  to handle duplicate messages.
+- **No timeout for waiting states.** A transfer process or a contract negotiation waiting for a message of the
+  counter-party (e.g. REQUESTED, AGREED) waits indefinitely.
+- **Agreement rebuilt on every attempt.** Until the contract agreement is saved, the provider builds a new one (new
+  id, new signing date) at every attempt to send it. If an attempt reaches the consumer but is seen as failed, the
+  consumer receives a second agreement: it replaces the first one while in AGREED (the `agreed` event fires twice),
+  or rejects it if it has moved on, which terminates the negotiation.
+- **Retirement reverted before finalization.** Retiring an agreement does not change the version of its negotiation,
+  and saving a negotiation writes the retirement fields of its agreement. A retirement done while the negotiation is
+  between AGREED and FINALIZED can be reverted by a concurrent save of an older copy. Once the negotiation is
+  FINALIZED, nothing saves it anymore.
+- **Saving a deleted entity re-inserts it.** The store upsert inserts a missing row whatever the entity version is.
+  This is not reachable today: negotiations are deleted only in TERMINATED without agreement, which nothing saves
+  anymore, and transfer processes are never deleted. Splitting `save` into a create and an update would remove it.
+- **State changed before a step that can fail.** `processProviderInitial` transitions the transfer process before
+  storing the data address of the data plane; a retryable failure of that store would make the retry transition fail.
+  It is not reachable with `VaultDataAddressStore`, whose failures are fatal.
+
+## Poll-based state machine
+
+- **Processing longer than the lease.** When the processing of an entity lasts longer than its lease, including the
+  HTTP timeouts and retries of a protocol message, the entity is processed again, by another replica or by the same
+  one. Size the `lease-duration` settings accordingly (see
+  [Tuning](../../tuning/_index.md#lease-duration)).
+- **Synchronous failures are not retries.** An entity whose processing throws before anything is saved (e.g. an
+  unavailable vault, inconsistent data) does not increase its state count: it is not subject to the retry limit or
+  to the retry delay, and it is processed again at every iteration, with a severe log each time. This costs load and
+  logs, not correctness.
+- **Retry delays block the batch.** Entities waiting for their retry delay are leased first, ordered by state
+  timestamp, and fill the batch: ready entities in the same state are not processed until the delays end (up to a few
+  minutes with the defaults), and every waiting entity costs a lease and its release at every iteration. It shows when
+  at least `batch-size` entities in one state are waiting, e.g. during the outage of a counter-party, which then delays
+  the processing of the others. A fix would persist the due time on the entity and filter on it.
+- **Scalability.**
+  - **Idle polling:** an idle replica runs one query per processor (state) at every iteration, with no idle backoff.
+  - **Leasing a batch:** it takes one UPDATE per entity.
+  - **No backpressure:** nothing limits the number of dispatches in flight.
+
+## Task-based state machine (NATS)
+
+The provisioning of the NATS streams and consumers, the task store and the retry settings are described in
+[Tuning](../../tuning/_index.md#task-based-state-machine-nats).
+
+- **Lost tasks.** A task is published before its transaction commits. A replica that does not see it yet retries with
+  a growing delay and drops the message after `edc.nats.*.subscriber.max-retries` deliveries, so a commit slower than
+  that window loses the message. A task whose handling keeps failing is terminated after the retry limit, with a
+  severe log. In both cases, and if messages are lost on the NATS side (e.g. a purged stream), the task stays in the
+  task store, but nothing publishes it again.
+- **Transaction scope.** The handling of a task runs in a database transaction that holds a connection while the
+  remote call is made. The save that follows the call runs on the HTTP client thread, in its own transaction,
+  together with the creation of the next task. The two are not atomic, which is safe: a leftover task finds the entity
+  in another state and is skipped.
+- **Lease collisions.** A task whose save collides with the lease held by an incoming protocol message or an API
+  command is retried with the usual delay: on redelivery, the entity has either moved on, and the task is skipped, or
+  the remote call is made again. These retries do not count toward the retry limit. Such leases are held only for the
+  duration of the message or command, so collisions are rare and short.
+- **Head-of-line blocking.** A replica processes the tasks of an entity type with one subscriber thread, one message
+  at a time, and tasks are not scoped by participant: a slow counter-party delays the tasks behind it on that replica.
+  To be addressed by processing the batch with a pool of workers.
+- **Stale tasks.** A task carries only the entity id and its state. A task waiting for a retry delay while its entity
+  leaves the state and enters it again is processed for the new visit, which has its own task: at worst, a remote
+  call is made twice. The entity version cannot tell them apart, since every save changes it, including the one of a
+  failed attempt.
+
+[#6105]: https://github.com/eclipse-edc/Connector/issues/6105
+[#6116]: https://github.com/eclipse-edc/Connector/issues/6116
+[stateful entity versioning]: https://github.com/eclipse-edc/Connector/tree/main/docs/developer/decision-records/2026-10-14-stateful-entity-versioning

content/en/documentation/for-adopters/tuning/_index.md

@@ -8,6 +8,10 @@ weight: 80
   * [State Machines](#state-machines)
     * [Settings](#settings)
     * [How to tune them](#how-to-tune-them)
+    * [State machine nuances](#state-machine-nuances)
+    * [Lease duration](#lease-duration)
+    * [Task-based state machine (NATS)](#task-based-state-machine-nats)
+    * [Known limitations](#known-limitations)
 <!-- TOC -->
 
 Out of the box the EDC provides a set of default configurations that aim to find a good balance for performances.
@@ -100,3 +104,42 @@ this state machine manages the data-flows so based on how many data flows are ex
 _tuning suggestion:_ \
 `state-machine.iteration-wait`: in the order of 100/1000 milliseconds for many PUSH transfers, consider 1/10 seconds for few PULL transfers. \
 `state-machine.batch-size` in the order of 10/100 \
+
+### Lease duration
+
+When several connector replicas share the same database, an entity is leased by the replica that processes it. If the
+processing lasts longer than the lease, the lease expires and the entity is processed again, by another replica or by
+the same one. The lease must then last longer than the slowest processing step, including the HTTP timeouts and the
+retries of a protocol message.
+
+| Setting                                        | Default |
+|------------------------------------------------|---------|
+| `edc.transfer.state-machine.lease-duration`    | `PT60S` |
+| `edc.negotiation.state-machine.lease-duration` | `PT60S` |
+| `edc.policy.monitor.lease-duration`            | `PT60S` |
+
+### Task-based state machine (NATS)
+
+Instead of polling the store, the transfer processes and the contract negotiations can be driven by tasks published to
+NATS JetStream for every state change. The settings above about iterations and batches don't apply to it; the
+following ones do:
+
+- **Retries:** a retryable failure is delivered again after `edc.transfer.send.retry.base-delay.ms` /
+  `edc.negotiation.send.retry.base-delay.ms` (default 1000), doubled at every delivery, up to
+  `edc.transfer.send.retry.limit` / `edc.negotiation.send.retry.limit` (default 7), as on the poll-based state
+  machine. There is no maximum delay: a high limit leads to very long delays.
+- **Streams and consumers:** the ones created by the subscribers (`edc.nats.tp.subscriber.autocreate`,
+  `edc.nats.cn.subscriber.autocreate`) are meant for testing: memory storage, limits retention and the server
+  defaults. In production, provision them with persistent storage and the following settings:
+  - `MaxAckPending`: size it for the tasks that can wait at the same time. A task waiting for a retry delay counts
+    toward it, and the limit is shared by all the replicas, which use the same durable consumer. When it is reached,
+    new tasks are not delivered until some delays expire.
+  - Ack wait: the subscriber marks the messages it holds as in progress at a third of the ack wait, so the ack wait
+    does not need to cover the processing time.
+- **Task store:** the default in-memory task store does not work with several replicas: use the SQL task store
+  (`task-store-sql`).
+
+### Known limitations
+
+The accepted limitations of both state machines, their impact and how to mitigate them are listed in
+[State Machines Known Limitations](../control-plane/state-machines/_index.md).