Sandbox mode isolated: the parent brokers db_lookup and fhir_lookup (BACKLOG #3131, slice A)

changelog.d/3131.added.md

@@ -0,0 +1,14 @@
+- **`[sandbox].mode = "isolated"`: the subprocess sandbox worker, with `db_lookup` and
+  `fhir_lookup` brokered through the engine.** A Handler's lookup is sent to the engine over the
+  worker pipe and answered by the same lookup runner `mode = "off"` uses, so it meets the same
+  checks. The engine refuses a brokered lookup bound to a Router or an `accepts=` predicate, and one
+  whose `[egress].allowed_db` or `[egress].allowed_http` list is empty, even under the audited
+  opt-out `[security].block_unlisted_outbound = false`. `wall_seconds` bounds the whole call,
+  the time the engine spends serving its lookups included.
+  This mode adds no operating-system confinement yet: an `isolated` worker can open files and
+  sockets exactly as a `subprocess` worker can. See [CONFIGURATION.md](../docs/CONFIGURATION.md),
+  section `[sandbox]`. (vault `BACKLOG #3131`)
+- **Every sandbox worker mode now disables `[pipeline].fuse_thread_hops`,** not only
+  `subprocess`. Under `isolated`, a runner whose graph has no config directory refuses at
+  construction, rather than run its Handlers in the engine process with unbrokered lookups.
+  (vault `BACKLOG #3131`)

docs/CONFIGURATION.md

@@ -1311,6 +1311,51 @@ matters most in the other direction: setting `mode="off"` because one Handler ne
 takes every other Router and Handler out of the sandbox too. To isolate some feeds and not others you
 need separate engine processes.
 
+**`mode="isolated"` is `subprocess` plus a lookup broker** ([ADR
+0147](adr/0147-hardened-runtime-isolation-for-router-handler-code-ipc-brokered-sandbox-extends-adr-0087.md),
+vault BACKLOG #3131). It runs the same per-inbound worker, under the same caps and the same codec.
+It differs from `subprocess` in at least these ways:
+
+- **A Handler's `db_lookup`/`fhir_lookup` works.** The worker sends the call to the engine over the
+  worker pipe, and the engine answers it with the same lookup runner it uses at `mode="off"`, against
+  the same `DatabaseLookup`/`FhirLookup` connections. So a lookup there meets every check it meets
+  in-process, at least the `[egress]` host check made when the connection is built, the read-only
+  statement test, `max_rows` and the result timeout. Its result, or its error, reaches the Handler as
+  it would at `mode="off"`, so a Handler that catches `DbLookupError` or `FhirLookupError` still can.
+  A `Decimal` column value arrives as a `Decimal`. A value outside the worker pipe's types, such as a
+  `UUID`, fails that lookup with an error that says so.
+- **A lookup whose `[egress]` list is empty is refused,** `[egress].allowed_db` for `db_lookup` and
+  `[egress].allowed_http` for `fhir_lookup`. **That holds under the audited opt-out
+  `[security].block_unlisted_outbound = false` too,** where an empty list would otherwise mean
+  unrestricted, so the opt-out cannot open a lookup from the worker to an unlisted host. The first
+  refusal per inbound and list logs a `WARNING`; each refused call fails like any lookup error.
+  A call made where no lookup runner exists fails with the `mode="off"` error first, so a graph
+  that declares no `DatabaseLookup` is never told to edit `[egress]`.
+- **It is still a Handler-only call.** A Router or an `accepts=` predicate that calls either lookup
+  fails as it does at `mode="off"`, and the engine refuses a brokered call bound to either of them.
+  The dry run has no lookup runner, so it fails there as in every mode.
+- **`wall_seconds` counts the lookups.** The cap bounds the whole call, including the time the engine
+  spends serving its lookups, so raise it for a feed whose lookups are slow. When the cap runs out
+  during a lookup, the call fails at once and its worker is recycled. A lookup that has already
+  reached its runner runs on to its own timeout in the engine, and its result is discarded. One that
+  has not reached it is normally not started. One that was a moment from its runner call when the
+  cap ran out can still start; it is then recorded and holds the connection like any other. While a
+  lookup runs on this way, a later lookup on the same connection from that inbound waits for it,
+  until a quarter of its own call's cap (at most 2 seconds) is left. It is refused if it is still
+  waiting then, which leaves that quarter for the Handler to handle the refusal. This rule holds
+  per inbound, within one sandbox session: the engine starts an inbound's session afresh at a
+  reload or a stop, and a lookup still running from before then is not waited for. Two inbounds can
+  each run a lookup on one connection. It does not bound the database either: a `db_lookup` its
+  runner gave up on can still be running there.
+- **It needs a config directory.** A graph built in code with none cannot start a worker. Under
+  `subprocess` such a graph quietly runs in-process; under `isolated` the runner refuses to be built
+  instead, because in-process its lookups would not go through the broker.
+
+**What `isolated` does not do yet:** confine the worker at the operating-system level. ADR 0147's
+Landlock and seccomp confinement on Linux and its AppContainer token on Windows are not built, so an
+`isolated` worker can still open files and sockets as the service account, exactly as a `subprocess`
+worker can. *What this boundary does NOT cover*, below, holds for both modes.
+
 **What turning it on costs you.** At least these things change relative to `mode="off"`:
 
 | | At `mode="subprocess"` |
@@ -1367,8 +1412,8 @@ under `serve`. And config code there sees the full environment, not the worker's
 of these. The dry run has no lookup runner, so the call raises there in every mode. Unless the
 Handler catches that error, the fixture records `ERROR`. The gate then fails unless its `.expect`
 file declares `ERROR`. Under `subprocess` that matches `serve`, which refuses the lookup; at
-`mode="off"` it does not, because `serve` runs the lookup there. And
-`[pipeline].fuse_thread_hops` is **hard-disabled** whenever `mode="subprocess"`: fusion runs
+`mode="off"` and `mode="isolated"` it does not, because `serve` runs the lookup there. And
+`[pipeline].fuse_thread_hops` is **hard-disabled** whenever `mode` is not `off`: fusion runs
 Router/Handler code in-process on an executor hop, so honouring both would silently unsandbox the code
 you asked to isolate. The runner fails closed to the async sandboxed path and logs it. To get fusion you
 must leave `mode="off"`.
@@ -1425,7 +1470,7 @@ on a live feed; a Router that only reads is unaffected. Grep your config for `os
 
 | Key | Type | Default | Notes |
 |---|---|---|---|
-| `mode` | enum | `off` | `off` (in-process, byte-identical, no subprocess) or `subprocess` (persistent per-inbound worker child). |
+| `mode` | enum | `off` | `off` (in-process, byte-identical, no subprocess), `subprocess` (persistent per-inbound worker child), or `isolated` (the same worker, with `db_lookup`/`fhir_lookup` brokered through the engine; see *`mode="isolated"`* above). Any other value refuses to load. |
 | `wall_seconds` | float (>0) | 5.0 | **Authoritative** wall-clock cap per Router/Handler call on **every** platform — the parent kills a worker that overruns it, so a pathological busy-loop can't wedge intake. The worker sets no `RLIMIT_CPU`, so CPU that admin code spends outside a call, such as a thread left running or a grandchild that leaves the worker's process group, has no bound at all ([ADR 0087](adr/0087-sandbox-subprocess-isolation.md), amendment of 2026-10-07). |
 | `mem_mb` | int (≥1) or null | 512 | POSIX-only memory cap (MiB) inside the child (no-op on Windows). `null` disables it. **On Linux it is headroom above the loaded graph:** the worker measures its own `VmData` once `load_config` has built the graph, and sets `RLIMIT_DATA` to that plus `mem_mb`. So a graph bigger than `mem_mb` still starts, and address space a thread only reserves does not count. The worker also starts with `MALLOC_ARENA_MAX=2` unless you pass your own value through `pass_environment`, so threads cannot use up the headroom for the worker's whole life. On other POSIX systems it is an absolute `RLIMIT_AS` cap on the whole worker, and the worker logs a line saying so. A call that runs out of memory dead-letters with an error naming the memory cap, and its worker exits so the next message gets a fresh one ([ADR 0087](adr/0087-sandbox-subprocess-isolation.md), amendment of 2026-10-11). |
 | `pass_environment` | list of names | `[]` | Extra environment variable **names** the worker is given, beyond its allowlist. Use it for a variable your config code reads. Names only; the value comes from the engine's own environment at worker start. **Refused at load** if a name is one of the engine's own: any `MEFOR_*` name, and `VAULT_TOKEN` or `PGPASSWORD`. Give a variable your config reads a name outside `MEFOR_`. Whatever you list is readable by every Router and Handler, so do not list a secret of your own either. Env: `MEFOR_SANDBOX_PASS_ENVIRONMENT`, comma-separated. |

docs/adr/0147-hardened-runtime-isolation-for-router-handler-code-ipc-brokered-sandbox-extends-adr-0087.md

@@ -174,8 +174,32 @@ unresearched.
 - [ ] The Landlock **network-allowlist kernel floor** (≥6.7 for net rules) + the seccomp profile; the
       degrade path on an older kernel (FS-only Landlock, or refuse `mode=isolated`).
 - [ ] A **throughput benchmark** of the broker round-trip + confinement vs the ADR 0087 subprocess baseline.
-- [ ] Whether this ships as `[sandbox].mode=isolated` (a third mode above `subprocess`) or a sub-flag on
-      `subprocess`.
+- [x] Whether this ships as `[sandbox].mode=isolated` (a third mode above `subprocess`) or a sub-flag on
+      `subprocess`. **Settled 2026-10-27: a third mode, `isolated`.** Decided by a Manager seat under
+      the owner's delegation, with two companion rulings: every check meaning "runs in a sandbox
+      worker" reads `is not SandboxMode.OFF`, and a brokered lookup refuses while its `[egress]` list
+      is empty, even under `[security].block_unlisted_outbound = false` (Amendment A's open AC-2
+      question). See *Build status* below.
 - [ ] (Added 2026-10-07, Amendment A.) How the AppContainer spawn fits the worker's suspended start and
       kill-on-close job (vault BACKLOG #3065).
 - [ ] A **WASI / CPython-on-WASM feasibility spike** (C-extension deps, throughput) before re-scoring option 3.
+
+## Build status (2026-10-28, vault BACKLOG #3131 slice A)
+
+**Built:** the `isolated` mode and decision (1), the parent-held broker. A Handler's
+`db_lookup`/`fhir_lookup` in the worker writes a `lookup` frame in the closed tag set. The frame is
+bound to the dispatch's `(id, phase, name)` triple and a per-dispatch sequence number (AC-6, for a
+per-inbound worker, whose lane the session already is). The parent answers it on a helper thread, in
+the dispatch's context, through the same accessor and runner the in-process path uses (AC-1, AC-2).
+It refuses one bound to a Router or `accepts=` dispatch, and one whose `[egress].allowed_db` or
+`[egress].allowed_http` is empty. A refused or failed lookup is the Handler's error, routed to
+`ERROR`/dead-letter post-ACK (AC-5, for the broker). `wall_seconds` bounds the whole dispatch, the
+brokered lookups and the writes of their replies included. What happens to a lookup the cap cuts
+off, and to a later lookup on the same connection, is stated once, with the limits of that bound,
+in [CONFIGURATION.md, `[sandbox]`](../CONFIGURATION.md#sandbox), under `mode="isolated"`.
+
+**Not built:** decision (2), the OS confinement: AC-3 (Landlock and seccomp), AC-4 (the AppContainer
+token), and AC-7 (no opening or signalling the parent or a sibling). The first two open boxes above,
+the #3065 box and the throughput benchmark still gate them. AC-6's scoping for a pooled worker
+(ADR 0187) is not built either, because no pooled worker exists yet. Until AC-3 and AC-4 land, an
+`isolated` worker is confined exactly as a `subprocess` worker is.

messagefoundry/config/db_lookup.py

@@ -40,6 +40,7 @@
     "DbLookupError",
     "LookupRunner",
     "db_lookup",
+    "is_available",
     "set_active",
     "reset",
     "activated",
@@ -93,6 +94,13 @@ def activated(runner: LookupRunner | None) -> Iterator[None]:
         _active.reset(token)
 
 
+def is_available() -> bool:
+    """Whether a lookup runner is published here, so :func:`db_lookup` would run rather than raise
+    its "unavailable here" error. The ``mode=isolated`` broker asks this before any refusal of its
+    own, so a call that would fail that way at ``mode=off`` fails that way there too (ADR 0147)."""
+    return _active.get() is not None
+
+
 def db_lookup(
     connection: str, statement: str, params: Mapping[str, Any] | None = None
 ) -> list[dict[str, Any]]:

messagefoundry/config/fhir_lookup.py

@@ -53,6 +53,7 @@
     "FhirLookupError",
     "FhirLookupRunner",
     "fhir_lookup",
+    "is_available",
     "set_active",
     "reset",
     "activated",
@@ -115,6 +116,13 @@ def activated(runner: FhirLookupRunner | None) -> Iterator[None]:
         _active.reset(token)
 
 
+def is_available() -> bool:
+    """Whether a FHIR-lookup runner is published here, so :func:`fhir_lookup` would run rather than
+    raise its "unavailable here" error. The ``mode=isolated`` broker asks this before any refusal of
+    its own (ADR 0147)."""
+    return _active.get() is not None
+
+
 def fhir_lookup(
     connection: str,
     query: str,

messagefoundry/config/settings.py

@@ -2498,7 +2498,7 @@ class PipelineSettings(_Section):
     # NOT re-read it — restart to change, exactly like claim_mode). Harness A/B via
     # MEFOR_PIPELINE_FUSE_THREAD_HOPS.
     # ONE OTHER SETTING CAN CANCEL THIS ONE, and you must be told here rather than in that knob's
-    # docs: the runner HARD-DISABLES fusion whenever [sandbox].mode is subprocess (it ships off, so
+    # docs: the runner HARD-DISABLES fusion whenever [sandbox].mode is not off (it ships off, so
     # this bites only a site that turned the sandbox on). Fusion runs Router/Handler/accepts= code
     # in-process on an executor hop, so honouring both would silently run unsandboxed the code a
     # config asked to isolate; the runner fails CLOSED to the async sandboxed path and logs it. To
@@ -2595,13 +2595,15 @@ class SandboxSettings(_Section):
 
     * **Live enrichment is refused.** ``db_lookup``/``fhir_lookup`` re-enter the event loop, which a
       process boundary breaks, so they fail closed inside the child. **A Handler needing either must
-      run ``mode="off"``** — that escape is supported and is not going away. ``[sandbox]`` is a
-      single **engine-wide** section: :meth:`messagefoundry.pipeline.engine.Engine.add_registry`
+      run ``mode="off"``, or ``mode="isolated"``** — the off escape is supported and is not going
+      away. ``mode="isolated"`` (ADR 0147, vault BACKLOG #3131) is the same worker with both lookups
+      brokered through the engine; ``docs/CONFIGURATION.md``, section ``[sandbox]``, says what it
+      does and does not do. ``[sandbox]`` is a single **engine-wide** section: :meth:`messagefoundry.pipeline.engine.Engine.add_registry`
       renders ONE ``SandboxPolicy`` for the whole graph and a connection carries no per-connection
       sandbox field, so ``mode="off"`` set for one Handler runs **every** Router and Handler in the
       process in-process, not just that one.
     * **``wall_seconds`` starts being enforced.** At ``mode="off"`` there is no timeout at all; at
-      ``mode="subprocess"`` the parent kills a worker that overruns and dead-letters that message
+      ``mode="subprocess"`` or ``mode="isolated"`` the parent kills a worker that overruns and dead-letters that message
       post-ACK. A busy-loop can no longer wedge intake, **and** a legitimately slow Handler that used
       to finish now dead-letters. ``startup_seconds`` and the POSIX ``mem_mb`` arm with it.
     * **Throughput** ~0.19 ms per dispatch with no reference view; a 20k-entry crosswalk ~4.5 ms
@@ -2620,9 +2622,10 @@ class SandboxSettings(_Section):
     **restart to change**, exactly like ``claim_mode``)."""
 
     # off (default, byte-identical, no subprocess — and the supported escape for a Handler needing
-    # live enrichment) | subprocess (persistent per-inbound worker child). ENGINE-WIDE, not per
-    # connection: one policy is rendered for the whole graph.
-    mode: Literal["off", "subprocess"] = Field(default="off")
+    # live enrichment) | subprocess (persistent per-inbound worker child) | isolated (that worker,
+    # with db_lookup/fhir_lookup brokered through the engine; ADR 0147). Any other value refuses to
+    # load. ENGINE-WIDE, not per connection: one policy is rendered for the whole graph.
+    mode: Literal["off", "subprocess", "isolated"] = Field(default="off")
     # Authoritative wall-clock cap (seconds) per Router/Handler call on EVERY platform: the parent
     # kills a worker that overruns it, so a pathological busy-loop can never wedge intake. Floor > 0.
     # There is no RLIMIT_CPU, so CPU spent outside a call has no bound (ADR 0087, 2026-10-07).

messagefoundry/fhirsearch.py

@@ -137,6 +137,19 @@ class FhirRaw:
     value: str
 
 
+@final
+@dataclass(frozen=True, slots=True)
+class ForeignSearchValue:
+    """A search value of a kind :func:`fhir_lookup` does not take, carried by its type NAME only.
+
+    Not a kind an author writes, and not in ``__all__``: it is the sandbox codec's carrier. The
+    ``mode=isolated`` sandbox (ADR 0147) sends a Handler's ``params`` to the engine, and a value of
+    any other type cannot cross the pipe, so it crosses as this. :func:`resolve_search_pairs` refuses it with the words it gives the original value, in the
+    engine and after the engine's own checks, so the Handler sees what ``mode=off`` shows it."""
+
+    type_name: str
+
+
 #: One search value: data (``str``), a token whose halves are declared (``FhirToken``), or author
 #: syntax (``FhirRaw``).
 FhirSearchValue = str | FhirToken | FhirRaw
@@ -163,7 +176,7 @@ def _screen(key: str, value: str, *, half: str) -> str:
     )
 
 
-def _resolve_one(key: str, value: FhirSearchValue) -> str:
+def _resolve_one(key: str, value: object) -> str:
     """One ``params`` value to the string that goes on the URL, before percent-encoding."""
     if isinstance(value, str):
         return _screen(key, value, half="value")
@@ -173,10 +186,11 @@ def _resolve_one(key: str, value: FhirSearchValue) -> str:
         return f"{value.system}|{_screen(key, value.code, half='code half of the FhirToken')}"
     if isinstance(value, FhirRaw):
         return value.value
+    got = value.type_name if isinstance(value, ForeignSearchValue) else type(value).__name__
     raise ValueError(
         f"FHIR search parameter {key!r}: a search value must be a str (data), a "
         f"FhirToken(system, code), or a FhirRaw(...) author-syntax value -- got "
-        f"{type(value).__name__}. A number is not a kind here: wrap it, str(value). "
+        f"{got}. A number is not a kind here: wrap it, str(value). "
         f"(The value is withheld: it may carry PHI.)"
     )