docs: NATS subscriber workers and prefetch

content/en/documentation/for-adopters/control-plane/state-machines/_index.md

@@ -85,9 +85,11 @@ The provisioning of the NATS streams and consumers, the task store and the retry
   command is retried with the usual delay: on redelivery, the entity has either moved on, and the task is skipped, or
   the remote call is made again. These retries do not count toward the retry limit. Such leases are held only for the
   duration of the message or command, so collisions are rare and short.
-- **Head-of-line blocking.** A replica processes the tasks of an entity type with one subscriber thread, one message
-  at a time, and tasks are not scoped by participant: a slow counter-party delays the tasks behind it on that replica.
-  To be addressed by processing the batch with a pool of workers.
+- **Head-of-line blocking.** A replica handles the tasks of an entity type with a pool of workers (one by default),
+  and a worker is blocked for the duration of the remote call of a task: once all the workers wait for slow
+  counter-parties, the tasks behind them wait too on that replica. Tasks are not scoped by participant, so a
+  participant context with many slow tasks can take all the workers. Raise the number of workers (see
+  [Tuning](../../tuning/_index.md#task-based-state-machine-nats)).
 - **Stale tasks.** A task carries only the entity id and its state. A task waiting for a retry delay while its entity
   leaves the state and enters it again is processed for the new visit, which has its own task: at worst, a remote
   call is made twice. The entity version cannot tell them apart, since every save changes it, including the one of a

content/en/documentation/for-adopters/tuning/_index.md

@@ -124,6 +124,16 @@ Instead of polling the store, the transfer processes and the contract negotiatio
 NATS JetStream for every state change. The settings above about iterations and batches don't apply to it; the
 following ones do:
 
+- **Workers:** a subscriber handles the messages it fetches with `edc.nats.tp.subscriber.workers` /
+  `edc.nats.cn.subscriber.workers` workers (default 1), i.e. the number of messages handled concurrently. With a single
+  worker, a slow counter-party delays all the tasks behind it on that replica.
+- **Prefetch:** a subscriber holds at most `workers + prefetch` messages, with `edc.nats.tp.subscriber.prefetch` /
+  `edc.nats.cn.subscriber.prefetch` (default 10), and fetches again once half of them are completed, up to
+  `edc.nats.tp.subscriber.batch-size` / `edc.nats.cn.subscriber.batch-size` messages per fetch (default 100). The
+  messages held by a replica can't be taken by another one, so keep the prefetch small.
+- **Database connections:** a worker holds a database connection while it makes the remote call of a task. Size the
+  datasource pool for the workers of both subscribers (transfer processes and contract negotiations), on top of the
+  API traffic.
 - **Retries:** a retryable failure is delivered again after `edc.transfer.send.retry.base-delay.ms` /
   `edc.negotiation.send.retry.base-delay.ms` (default 1000), doubled at every delivery, up to
   `edc.transfer.send.retry.limit` / `edc.negotiation.send.retry.limit` (default 7), as on the poll-based state
@@ -133,7 +143,8 @@ following ones do:
   defaults. In production, provision them with persistent storage and the following settings:
   - `MaxAckPending`: size it for the tasks that can wait at the same time. A task waiting for a retry delay counts
     toward it, and the limit is shared by all the replicas, which use the same durable consumer. When it is reached,
-    new tasks are not delivered until some delays expire.
+    new tasks are not delivered until some delays expire. It must also cover the `workers + prefetch` messages held by
+    every replica.
   - Ack wait: the subscriber marks the messages it holds as in progress at a third of the ack wait, so the ack wait
     does not need to cover the processing time.
 - **Task store:** the default in-memory task store does not work with several replicas: use the SQL task store