docs(integration-toolkit): poll delivery payload mapping and pollable outbound hardening

docs/integrations/integration-toolkit/configuration.md

@@ -185,16 +185,38 @@ curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integra
   }'
 ```
 
+#### Event Filter
+
+`event_filter` is an optional JSONata predicate on the use case configuration, next to `event_catalog_event`. It narrows which events of that name the use case handles — for example, only tickets with a certain purpose, or only certain contract types:
+
+```json
+{
+  "event_catalog_event": "CustomerRequestSubmitted",
+  "event_filter": "$count(ticket._purpose[$ = $env.move_request_purpose]) > 0",
+  "mappings": [ … ]
+}
+```
+
+- **Input:** the full hydrated event-catalog event, so relation nodes such as `ticket` and `contact` are populated.
+- **Bindings:** `$env` (the organization's non-secret environment variables, including [Key/Value Maps](./key-value-maps.md)), `$mapValue` and `$mapKey`. Referencing `$env` instead of hard-coding organization-specific values such as entity IDs keeps the filter portable between organizations, for example in a blueprint.
+- **Result:** the use case handles the event only when the filter evaluates truthy. When `event_filter` is absent, every event of the configured name is handled.
+- **Errors:** a filter that throws while evaluating is treated as **no match** and logged, so one malformed filter cannot stop the other use cases subscribed to the same event.
+- **Validation on save:** the filter must be a non-empty string, valid JSONata, and use no bindings other than `$env`, `$mapValue` and `$mapKey` (names the expression binds itself with `:=` are allowed).
+
+An event the filter rejects is not processed by the use case at all: no [Pollable Outbound](./pollable-outbound.md) queue item and no [file delivery](./outbound-file-delivery.md). `event_filter` currently applies to poll and file proxy deliveries; it is not evaluated for webhook deliveries.
+
 #### Mapping Properties
 
 | Property | Type | Required | Description |
 |----------|------|----------|-------------|
 | `id` | string (UUID) | No | Unique identifier for the mapping; generated when omitted |
 | `name` | string | Yes | Display name for the mapping |
 | `enabled` | boolean | Yes | Whether this mapping is active |
-| `jsonata_expression` | string | For `webhook` delivery | JSONata expression to transform the event payload. Required for `webhook`, ignored for `poll`, and rejected for `file_proxy` delivery |
+| `jsonata_expression` | string | For `webhook` delivery | JSONata expression to transform the event payload. Required for `webhook` (evaluated by the webhook service). Optional for `poll`: evaluated at enqueue time against the standardized event-catalog event with `$env` / `$mapValue` / `$mapKey`, must return a JSON object, and an empty value delivers the raw event — see [Payload Mapping](./pollable-outbound.md#payload-mapping). Rejected for `file_proxy` delivery |
 | `delivery` | object | Yes | How the event is delivered — discriminated on `type`: `webhook`, `poll`, or `file_proxy` |
 
+Outbound configurations are validated on save. The v1 use case endpoints require a `jsonata_expression` on every `webhook` mapping. The v2 integration upsert (`POST` / `PUT /v2/integrations`) validates an outbound use case only when it is new or its configuration changed, so configurations stored before a rule existed can be re-sent unchanged. It also accepts a `webhook` mapping with **no** `jsonata_expression` at all, for configurations that predate that requirement — such a mapping never enables or updates its webhook. An empty expression and invalid JSONata are rejected on both versions.
+
 #### Delivery Types
 
 **Webhook delivery (push):** the event payload is transformed with the mapping's `jsonata_expression` and pushed to a pre-configured webhook (epilot Webhooks):
@@ -211,7 +233,7 @@ curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integra
 | `webhook_id` | string | Yes | Reference to the webhook configuration in epilot Webhooks |
 | `webhook_name` | string | No | Cached webhook name for display purposes |
 
-**Poll delivery (pull):** for ERPs that cannot expose an inbound HTTP endpoint (firewalled, on-prem, batch systems). Items are placed on a pull-based queue that your system fetches and acknowledges. Poll items carry the **raw standardized event payload** — no JSONata transform is applied. See [Pollable Outbound](./pollable-outbound.md) for the full feature documentation (polling API, ordering guarantees, dead-letter handling, monitoring):
+**Poll delivery (pull):** for ERPs that cannot expose an inbound HTTP endpoint (firewalled, on-prem, batch systems). Items are placed on a pull-based queue that your system fetches and acknowledges. Poll items carry the **raw standardized event payload**, unless the mapping sets a `jsonata_expression` — then they carry its output, evaluated once at enqueue time (see [Payload Mapping](./pollable-outbound.md#payload-mapping)). See [Pollable Outbound](./pollable-outbound.md) for the full feature documentation (polling API, payload mapping, ordering guarantees, dead-letter handling, monitoring):
 
 ```jsonc
 // DeliveryConfig — poll variant
@@ -234,7 +256,7 @@ curl -X POST 'https://integration-toolkit.sls.epilot.io/v1/integrations/{integra
 - A `poll` delivery must not carry webhook fields (`webhook_id`, `webhook_name`), and a `webhook` delivery must not carry poll fields (`retention_days`, `poison_policy`, `max_delivery_attempts`).
 :::
 
-Everything beyond the configuration contract — the polling and acknowledgement API, lease and ordering semantics, retention and expiry behavior, the dead-letter queue and operator actions, and poll-mode monitoring — is documented on the dedicated [Pollable Outbound](./pollable-outbound.md) page.
+Everything beyond the configuration contract — the polling and acknowledgement API, payload mapping and its preview endpoint, lease and ordering semantics, retention and expiry behavior, the dead-letter queue and operator actions, and poll-mode monitoring — is documented on the dedicated [Pollable Outbound](./pollable-outbound.md) page.
 
 **File proxy delivery (push):** points to an upload-direction `file_proxy` use case in the same integration. The referenced recipe owns fan-out, payload mapping, authentication, and HTTP steps. `jsonata_expression` is rejected on this mapping type, and the use case's `event_catalog_event` must declare `event_attachments`. See [Outbound File Delivery](./outbound-file-delivery.md) for the complete setup and runtime behavior.
 

docs/integrations/integration-toolkit/key-value-maps.md

@@ -51,13 +51,14 @@ Rules worth knowing:
 - Lookup keys are compared as strings — `$mapValue($env.salutation, 1)` and `$mapValue($env.salutation, "1")` are the same lookup.
 - `$mapKey` compares values with strict equality: a map value `"1"` does not match the number `1`. Coerce with `$string()` when the source field is numeric.
 - When no `default` is given and nothing matches, the result is `undefined` and the mapped attribute is simply omitted — the same behaviour as any other undefined JSONata result.
-- If the first argument is not an object (for example the environment variable does not exist yet), the expression fails with `$mapValue: first argument must be an object` / `$mapKey: …`. In inbound use cases this surfaces as a mapping error in monitoring; in webhooks the delivery fails.
+- If the first argument is not an object (for example the environment variable does not exist yet), the expression fails with `$mapValue: first argument must be an object` / `$mapKey: …`. In inbound use cases this surfaces as a mapping error in monitoring; in webhooks the delivery fails; in pollable outbound the queue item is marked as a [mapping failure](./pollable-outbound.md#mapping-failures).
 - Values are read through the environments cache, so a change to a map becomes visible to running integrations within about 60 seconds.
 
 ### Where `$env`, `$mapValue` and `$mapKey` are available
 
 - Inbound use cases — every `jsonataExpression` field mapping and entity-level `jsonata` expression, including the mapping simulation endpoint.
 - Outbound webhooks — the payload transformation and multipart form-field expressions.
+- Pollable outbound — the poll mapping's [payload transform](./pollable-outbound.md#payload-mapping), evaluated at enqueue time, including its preview endpoint. See the [multi-organization example](./pollable-outbound.md#example-one-shape-for-a-multi-organization-middleware) for maps that give many organizations one payload shape.
 - Outbound file proxy — request body templates and delivery expressions.
 
 ## Recommended shape: one map, both directions

docs/integrations/integration-toolkit/monitoring/acks.md

@@ -81,11 +81,12 @@ filterable in the Monitoring tab:
 | Code | Level | When |
 |---|---|---|
 | `ACK_PENDING` | info | The event was delivered and epilot is waiting for the acknowledgement |
-| `ACK_CONFIRMED` | info | Your acknowledgement arrived |
+| `ACK_CONFIRMED` | success | Your acknowledgement arrived |
 | `ACK_TIMEOUT` | warning | No acknowledgement within the timeout window |
 
-`ACK_PENDING` and `ACK_CONFIRMED` are **info**-level: they are lifecycle markers, not
-outcomes, so they are counted in total events but deliberately excluded from the
+`ACK_PENDING` is **info**-level: a lifecycle marker, not an outcome, so it is counted
+in total events but deliberately excluded from the success rate. `ACK_CONFIRMED` is a
+**success** — your system confirmed it processed the event, so it counts towards the
 success rate. `ACK_TIMEOUT` is a **warning** — the delivery itself worked, so it is
 not an error on epilot's side, but something on yours needs attention.
 

docs/integrations/integration-toolkit/monitoring/codes.md

@@ -15,7 +15,7 @@ Every monitoring event carries a **code** and a **level**. The code says what
 happened; the level says how much you should care. Both are filterable in the
 Integration Hub's [Monitoring tab](./overview.md) and through the events API.
 
-There are 80 codes. You are most likely here because you saw one in a failed
+There are 85 codes. You are most likely here because you saw one in a failed
 event — find it below.
 
 :::tip
@@ -31,6 +31,7 @@ Something failed and the event did not do what it was meant to do. These are wha
 |---|---|
 | `ATTACHMENT_NOT_FOUND` | The file no longer exists — it was removed between the event and the delivery |
 | `ATTRIBUTE_TYPE_MISMATCH` | An attribute value did not match the type declared in the entity schema |
+| `CONDITIONAL_VARIANT_WRITE_FAILED` | Conditional pricing refused one variant write. The item is dropped and not retried; the rest of the batch and the run continue. details.code names the reason where pricing gave one — UNKNOWN_ERROR means it did not |
 | `DEPRECATED_ENDPOINT` | This endpoint version is deprecated |
 | `DIRECT_ENTITY_NOT_ALLOWED` | The entity is not permitted by the use case entity allowlist |
 | `DIRECT_PAYLOAD_INVALID` | The direct mode payload failed validation against the versioned payload schema |
@@ -53,11 +54,14 @@ Something failed and the event did not do what it was meant to do. These are wha
 | `METER_READING_GROUP_RETRYING` | A batch write of meter readings failed and will be retried automatically — one event per attempt covering the whole group (reading_count and external_ids in details) |
 | `MISSING_REQUIRED_PARAM` | A required parameter is missing from the request |
 | `MISSING_UNIQUE_IDENTIFIERS` | The event is missing the unique identifier field(s) required to match an entity |
+| `MSG_DEAD_LETTERED` | Outbound message moved to the dead-letter queue after exhausting delivery attempts, or via an operator skip |
+| `MSG_EXPIRED_UNPOLLED` | Outbound message expired before being consumed — retention elapsed without a successful poll |
+| `MSG_HEAD_BLOCKED` | Outbound stream halted by a poison head message (block policy) — requires operator unblock or consumer acknowledgement |
 | `OAUTH2_TOKEN_FAILURE` | Failed to obtain an OAuth2 access token |
 | `PAYLOAD_TOO_LARGE` | The payload exceeded the maximum size accepted by the receiving system |
 | `PRUNE_SCOPE_PARTIAL_FAILURE` | Scope pruning completed with some failures |
 | `RECURSION_DEPTH_EXCEEDED` | Maximum recursion depth was exceeded during processing |
-| `RELATION_REF_ITEM_NOT_FOUND` | The relation_ref target entity exists but the referenced item/value could not be matched — skipped as non-retryable. Check the mapping configuration and the entity data. |
+| `RELATION_REF_ITEM_NOT_FOUND` | The relation_ref value could not be matched on the target entity (invalid mapped value, or still no match after writing it to the target) — skipped as non-retryable. Check the mapping configuration and the entity data. |
 | `RELATION_REF_VALUE_UNDEFINED` | A relation_ref mapping value resolved to undefined — check the mapping expression |
 | `REQUIRED_PARAM_MISSING` | A param the use case marks as required resolved to nothing, so the delivery was stopped before anything was sent — see param_name |
 | `SECURE_PROXY_DISABLED` | The secure proxy use case is disabled |
@@ -75,6 +79,7 @@ Something failed and the event did not do what it was meant to do. These are wha
 | `SIGNATURE_VERIFICATION_UNAVAILABLE` | The file service could not be reached to verify the request signature |
 | `STEP_DISABLED` | A request step's "run this step when" expression returned false, so this step and every step after it were skipped. This is the configuration working as written, not a fault. |
 | `TIMEOUT` | The operation timed out |
+| `UNIQUE_ID_LOOKUP_UNRESOLVABLE` | A related entity created for this event still could not be found by its unique ID, so the relation was skipped instead of creating a duplicate. Check that the unique ID value type matches the entity schema. |
 | `UNIQUE_ID_MULTIPLE_MATCHES` | Multiple entities matched the unique ID |
 | `UNIQUE_ID_NOT_IN_SCHEMA` | The unique ID attribute is not defined in the entity schema |
 | `UNKNOWN_ERROR` | An unexpected error occurred during processing |
@@ -90,9 +95,11 @@ Processing continued, but something needs a human eye — often a retry in fligh
 | Code | What it means |
 |---|---|
 | `ACK_TIMEOUT` | Acknowledgement timed out waiting for the ERP system |
+| `CONDITIONAL_VARIANT_WRITE_WARNING` | One variant write succeeded with a warning. The variant is stored and the rest of the batch and the run continue. details.code names the warning |
 | `EXTERNAL_WARNING` | A warning span pushed by an external system via the external monitoring events endpoint. |
 | `FILE_PROXY_UPLOAD_RETRYING` | A file upload failed with a retryable error and will be retried automatically — one event per attempt |
 | `LOOKUP_UNMAPPED` | A value was not listed in a lookup table and its fallback was used — see lookup_name and lookup_key for the gap |
+| `MSG_LATE_ARRIVAL` | An event arrived after the poll consumer had already received later events, so it was placed at the end of the stream instead of at its event time — it is delivered, but out of event-time order |
 | `SOFT_DELETED_ENTITY_MATCHED` | A soft-deleted entity matched the unique ID — it will be resurrected on upsert, or referenced as-is by a relation. Investigate why the ERP source is sending events for a deleted entity. |
 
 ## Success
@@ -101,6 +108,8 @@ The event did what it was meant to do. Useful for confirming a sync actually lan
 
 | Code | What it means |
 |---|---|
+| `ACK_CONFIRMED` | Acknowledgement was confirmed by the ERP system |
+| `CONDITIONAL_VARIANTS_WRITTEN` | Conditional price variants were written for one imported entity: emitted once per chunk, with a per-outcome count and the variant ids in details |
 | `ENTITY_CREATED` | A new entity was created in epilot |
 | `ENTITY_DELETED` | An entity was deleted from epilot |
 | `ENTITY_NO_OP` | No changes were needed for the entity |
@@ -110,6 +119,7 @@ The event did what it was meant to do. Useful for confirming a sync actually lan
 | `FILE_PROXY_UPLOADED` | The external system accepted the file |
 | `METER_READING_DELETED` | One or more meter readings were deleted — emitted once per batch, not per reading (reading_count and external_ids in details) |
 | `METER_READING_UPSERTED` | One or more meter readings were created or updated — emitted once per batch, not per reading (reading_count and external_ids in details) |
+| `MSG_ACKED` | Outbound message delivered: the polling consumer acknowledged it and it was removed from the queue |
 | `PRUNE_SCOPE_COMPLETED` | Scope pruning completed successfully |
 | `WEBHOOK_DELIVERED` | Webhook was delivered successfully |
 
@@ -119,17 +129,12 @@ Lifecycle markers rather than outcomes: a message was queued, a duplicate was ig
 
 | Code | What it means |
 |---|---|
-| `ACK_CONFIRMED` | Acknowledgement was confirmed by the ERP system |
 | `ACK_PENDING` | Acknowledgement is pending from the ERP system |
 | `DUPLICATE_EVENT` | This event was already processed (duplicate) |
 | `EXTERNAL_INFO` | An informational span pushed by an external system via the external monitoring events endpoint. |
 | `FAN_OUT_EMPTY` | The split expression returned an empty list, so nothing was sent — expected for events that carry no relevant items |
 | `FILE_PROXY_UPLOAD_ENQUEUED` | A per-file upload was accepted for delivery during fan-out |
-| `MSG_ACKED` | Outbound message acknowledged by the polling consumer and removed from the queue |
-| `MSG_DEAD_LETTERED` | Outbound message moved to the dead-letter queue after exhausting delivery attempts, or via an operator skip |
 | `MSG_ENQUEUED` | Outbound message enqueued to the poll queue, awaiting consumption by the ERP |
-| `MSG_EXPIRED_UNPOLLED` | Outbound message expired before being consumed — retention elapsed without a successful poll |
-| `MSG_HEAD_BLOCKED` | Outbound stream halted by a poison head message (block policy) — requires operator unblock or consumer acknowledgement |
 ## Status-code families
 
 Some codes are generated from an upstream response rather than drawn from the fixed list above.

docs/integrations/integration-toolkit/overview.md

@@ -120,6 +120,7 @@ See the [Configuration Guide](./configuration.md#secure-proxy-use-cases) for set
 
 - Inbound event processing (ERP to epilot entity mapping)
 - Outbound webhook payloads (epilot event to ERP format)
+- Pollable outbound payloads, transformed at enqueue time ([Payload Mapping](./pollable-outbound.md#payload-mapping))
 - The Map Data flow building block
 
 ### Monitoring and Alerting