diff --git a/docs/adr/0030-anonymization-test-harness-tee.md b/docs/adr/0030-anonymization-test-harness-tee.md index d097bf739f..0ee1f7fe70 100644 --- a/docs/adr/0030-anonymization-test-harness-tee.md +++ b/docs/adr/0030-anonymization-test-harness-tee.md @@ -62,6 +62,32 @@ characters of any one key or value, and of the list of unknown keys. **This changes none of the §2 ratified choices:** the overlay still only maps a field to an existing kind. [PHI.md](../PHI.md) §9 is the long form for the retarget. +- **AMENDMENT PROPOSED 2026-10-27, AWAITING THE OWNER'S SIGN-OFF (BACKLOG #1192, + [ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) + section 5 and slice 5): the package's home moves out of the engine.** ADR 0201 was accepted on + 2026-09-29 and designs this amendment. Its section 5 also says that changing this ADR's ratified + home needs the owner's sign-off on its own, in its own pull request. So the four items below are + proposed, not ratified, until that sign-off is recorded in this bullet. This builds nothing either + way: the package is still at `messagefoundry/anon/`, and every path in this ADR describes the + tree until ADR 0201 slice 6 moves it. That slice re-points the guards and tests that pin the + path, in the same change. + 1. **The home becomes `messagefoundry_toolkit/anon/`**, in the `messagefoundry-toolkit` + distribution rather than the engine wheel. This replaces the §1 *"engine-owned"* home *"sitting + beside `parsing/`"*, one of the forks ratified before the build. + 2. **The rule table stays the single authority, and the tee still vendors a copy at + `tee/anon/`.** ADR 0201 section 5 calls it a byte-identical copy, as §1 does. Not every file + is: [`tests/test_anon_parity.py`](../../tests/test_anon_parity.py) records which are. + 3. **The parity check re-points to the new path**, in slice 6. + 4. **The carve-out extension is withdrawn.** The Related line and §1 say acceptance extends the + CLAUDE.md §4 pure-library carve-out from `parsing/` to `anon/`, and §8 and *To resolve on + acceptance* list that as a doc flip. That flip never landed: CLAUDE.md §4 names `parsing/` + alone. Once `anon/` is not an engine package there is nothing for the carve-out to cover, so + the flip is withdrawn rather than left pending. Each of those four places carries a marker. + + What the package does, its rules, its fail-closed behaviour and its limits are unchanged. + **One correction here needs no sign-off**, because it records what the tree and an owner ruling + already say: §1 claimed the harness imports the package, and none does. The correction stands + where that sentence stood, with markers on the two other places that read as permission. - **Decision in one line:** ship a **pure-stdlib, dependency-free `anon` package** that turns real, messy HL7 v2 into structurally-faithful **PHI-free** datasets via a **two-layer rule model — a declarative field-*selection* map (data) over a code registry of pure surrogate *functions* (logic)** — @@ -78,8 +104,8 @@ [PHI.md](../PHI.md) §9 (de-identification `[ROADMAP]` — on landing this *would be* the first concrete slice that flips it) / §7 (redaction is *not* de-id); [CLAUDE.md](../../CLAUDE.md) §9 (de-id planned-not- built, centralize the rules) / §8 (read separators from MSH, never string-slice, parse defensively) / §4 - (`parsing/` is the one pure carve-out lib; this ADR extends that carve-out to `anon/` on acceptance — - dependency direction one-way) / §12 (don't drift to declarative *logic*). + (`parsing/` is the one pure carve-out lib; this ADR extends that carve-out to `anon/` on acceptance + (*withdrawal proposed 2026-10-27, see the status banner*) — dependency direction one-way) / §12 (don't drift to declarative *logic*). ## Context @@ -102,6 +128,8 @@ verbatim is *"conservative redaction, **not** de-identification."* This ADR must The constraint that shapes everything is a **dependency-split asymmetry**: - The **harness** freely imports the engine (`messagefoundry.parsing`, `.generators`, …) — it already imports engine packages directly, so a new `messagefoundry.anon` is an ordinary import for it. + *(Corrected 2026-10-27: no harness file imports this package; see the correction under §1's + vendored-copy paragraph.)* - The **tee** is **deliberately `messagefoundry`-free** (verified: zero `from messagefoundry` imports). It **vendors** [`tee/hl7_fields.py`](../../tee/hl7_fields.py) and `tee/mllp.py` precisely so the cutover relay can sit on the Epic/Corepoint boundary without dragging in the engine or its attack surface. @@ -122,7 +150,10 @@ The anonymizer ships as **`messagefoundry/anon/`**: a small, **pure-stdlib** pac `parsing/` it is a pure, side-effect-free library, but the §4 carve-out is **verbatim scoped to `parsing/`** and does **not** automatically extend to a new sibling. Accepting this ADR **amends CLAUDE.md §4** to add `anon/` to the console carve-out (listed in *To resolve on acceptance*); until then the console may not import -it. The harness is unaffected — it imports engine packages freely. Files: +it. *(Withdrawal proposed 2026-10-27: the flip never landed, and ADR 0201 moves `anon/` out of the +engine; see the status banner.)* The harness is unaffected — it imports engine packages freely. +*(Corrected 2026-10-27: not this package; see the correction under the vendored-copy paragraph +below.)* Files: - `anon/rules.py` — the **default field-selection map** (data; §2): `FieldRule(path, SurrogateKind)` rows naming *which* HL7 field gets *which kind* of surrogate. No transform expressions — just a field→kind map. @@ -165,9 +196,19 @@ SURROGATES: dict[SurrogateKind, Callable[[str, DatasetContext], str]] = { ... } **The tee vendors a byte-identical copy** at `tee/anon/` (mirroring how `tee/hl7_fields.py`/`tee/mllp.py` were vendored), kept in sync by a CI parity check — the same discipline the tee already lives by. The -**harness imports `messagefoundry.anon` directly** (it already imports `parsing`/`generators`). This is the -**asymmetric resolution**: both sides run the *same rules*, the harness by import, the tee by a -parity-pinned vendor copy. +harness's [`CaptureSink`](../../harness/reconcile/capture.py) anonymizes only when its caller passes it +an anonymizer function, and no harness file imports the package. The shipped +`python -m harness.reconcile capture` passes none, so it writes the bodies it receives as they +arrive. The tee runs the rules by a parity-pinned vendor copy. + +**CORRECTED 2026-10-27 (BACKLOG #1192; see the 2026-10-27 status-banner bullet):** the paragraph +read, after its first sentence, *"The **harness imports `messagefoundry.anon` directly** (it already +imports `parsing`/`generators`). This is the **asymmetric resolution**: both sides run the *same +rules*, the harness by import, the tee by a parity-pinned vendor copy."* No harness file imports the +package. The owner ruled on 2026-10-12 (BACKLOG #3006 part 3) that such an import gives way to the +record's claim that no file outside the package imports it, and an author who needs the package +waits for its move. `tests/test_dependency_boundaries.py` refuses an import of it from the source +trees it walks, the harness among them. ### 2. The rule model: field-*selection* is data, surrogate *production* is code (the load-bearing call) @@ -366,8 +407,8 @@ re-identification/linkage tooling. On landing, this **would become** the de-id framework's **first bounded slice** — a *test-data* anonymizer, not the full enterprise pipeline. **While this ADR is Proposed nothing is built**, so it does **not** yet flip -[PHI.md](../PHI.md) §9 from `[ROADMAP]` to built — that flip, and the lockstep CLAUDE.md §9 / §4 (carve-out) -and PHI.md §12/§13 edits, happen **only when the code lands**, never on acceptance (CLAUDE.md §9: "don't +[PHI.md](../PHI.md) §9 from `[ROADMAP]` to built — that flip, and the lockstep CLAUDE.md §9 / §4 (carve-out; +*withdrawal proposed 2026-10-27, see the status banner*) and PHI.md §12/§13 edits, happen **only when the code lands**, never on acceptance (CLAUDE.md §9: "don't reference a framework that doesn't exist"). The rule table is **designed to be reusable** as the future AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on landing). @@ -455,7 +496,7 @@ AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on `tee export`; requires `--capture-bodies`; errors when no captures exist; no raw bodies to stdout/CI), harness `CaptureSink._write` hook, and the file-backed corpus source. - [ ] Approve the **doc lockstep, gated on code landing (not acceptance):** add `anon/` to the CLAUDE.md §4 - console carve-out; flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13. + console carve-out (*withdrawal proposed 2026-10-27, see the status banner*); flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13. --- diff --git a/docs/adr/README.md b/docs/adr/README.md index 3e8c3addb6..0c6961a8ad 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -64,7 +64,7 @@ what is withheld and what you can request. | [0027](0027-per-connection-retention.md) | Per-connection retention / pruning windows — a per-connection `messages_days` (inbound) / `dead_letter_days` (outbound) override layered over the global `[retention]` default (`None` = inherit), reusing the #46 per-connection-override plumbing + ADR 0007 `connections.toml`; threads a `{connection → cutoff}` map through the existing purge SQL on all three backends; one `retention_purge` audit row per pass; null-body-keep-metadata + the in-flight guard preserved (#34) | Accepted (built, 0.2.9) | | [0028](0028-base64-binary-carriage-codec.md) | base64 binary-carriage codec (+ HL7 OBX-5 ED embedding) — carry arbitrary bytes over the str/TEXT ingress+store as `mfb64:v1:` unbroken base64; pure stdlib `parsing/binary.py` + `RawMessage.from_bytes`/`.raw_bytes`. Supersedes ADR 0025's latin-1 round-trip (NUL-unsafe across the store) | Accepted (2026-06-20) — Amended 2026-07-13: ERROR-path + happy-path NUL carriage (INGEST-4) — built (pure `parsing/binary.py` codec, the four `RawMessage` additions, OBX-5 ED helpers) | | [0029](0029-email-smtp-destination.md) | Email (SMTP-send) outbound destination — a stdlib `smtplib`/`email` `DestinationConnector` + new `ConnectorType.EMAIL` + a code-first `Email()`/`SMTP()` factory; STARTTLS-by-default with the existing insecure-TLS escape; a new `[egress].allowed_smtp` deny-by-default allowlist arm; `DeliveryError` → staged-queue retry (transform stays pure, SMTP is the side effect, at-least-once like other one-way destinations). IMAP/POP inbound read + XOAUTH2 deferred to Phase 2 (#23; was earmarked 0024 before SMART claimed it) | Accepted (2026-06-27, built) | -| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) | +| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) -- amended since, at least five times; the ADR's status banner lists each. An amendment proposed 2026-10-27, awaiting the owner's sign-off, moves its home to `messagefoundry_toolkit/anon/` ([ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) slice 6, not built) | | [0031](0031-startup-connection-fault-isolation.md) | Startup connection fault isolation — a single connection that fails to build/bind at startup is isolated (logged + alerted + reported `failed`), not fatal: the engine starts the rest of the graph and serves the API; a failed outbound retries (never drops) and self-heals on reload/restart. Reload stays fail-fast. Amended 2026-10-06 (vault #2371): a refused outbound or `Ftp` poller CA degrades its lane at start; a reload refuses only for a lane it builds or keeps running; a `FhirLookup` CA and an inbound listener CA still refuse | Accepted (2026-06-21) | | [0032](0032-console-desktop-launch.md) | Console desktop launch — a windowed `gui-script` (`messagefoundry-console`) + window/taskbar icon + Desktop/Start-Menu shortcut installer (Phase A); frozen installer (Phase B) retired. **RETIRED (2026-07-13, BACKLOG #103):** the whole PySide6 desktop console is removed — superseded by the browser web console (ADR 0065) + the `apiclient`/`messagefoundry service` extraction (ADR 0088) | RETIRED (2026-07-13) | | [0033](0033-gui-manageable-code-sets.md) | GUI-manageable code sets (translation tables) — a `messagefoundry codeset` CLI (mirrors `connection`: owns validate-before-write + atomic temp+replace + owner-only perms + post-write re-load) + a VS Code webview grid editor that shells it, CSV-first for v1 (TOML sets summarized/shown read-only). Reference sets (ADR 0006) + `db_lookup` (ADR 0010) are out of scope (externally-owned data); a rename/remove can dangle a `code_set(...)` reference, caught by `messagefoundry check`'s dry-run | Accepted (built, 0.2.3, #540) | diff --git a/tests/test_dependency_boundaries.py b/tests/test_dependency_boundaries.py index 699b75257e..4335ebda54 100644 --- a/tests/test_dependency_boundaries.py +++ b/tests/test_dependency_boundaries.py @@ -1557,9 +1557,9 @@ def test_the_toolkit_search_sees_a_planted_name(tmp_path: Path, plant: str) -> N # accident, not against an evasion. # # A REAL import is a different matter from a line that only looks like one. ADR 0030 section 1 -# allows the harness to import the package directly, and nothing does yet. An owner ruling says -# which side gives way then, and `_REAL_ANON_IMPORT_ADVICE` cites it in every failure message -# that may be a real import. +# said the harness imports the package directly; nothing does, and its 2026-10-27 correction says +# so. An owner ruling says which side gives way, and `_REAL_ANON_IMPORT_ADVICE` cites it in every +# failure message that may be a real import. # # THE PACKAGE'S OWN FILES. The claim is about files outside `messagefoundry/anon/`, and the package # imports its own modules, so the parse reports nothing there. It counts those imports instead, as @@ -1646,8 +1646,9 @@ def test_the_toolkit_search_sees_a_planted_name(tmp_path: Path, plant: str) -> N #: Shared by every failure that may be a real import, and so put first in each. _REAL_ANON_IMPORT_ADVICE = ( - "If a hit is a REAL import, the import must go. ADR 0030 section 1 allows the harness to " - "import the package. The record's absence claim says no file outside the package imports it. " + "If a hit is a REAL import, the import must go. ADR 0030 section 1 once said the harness " + "imports the package, and its correction says none does. The record's absence claim says no " + "file outside the package imports it. " "The owner ruled that the import gives way to the claim (owner ruling 2026-10-12, BACKLOG " "#3006 part 3). An author who needs the package waits for its move, BACKLOG #1192. Do not " "hide the import from these searches by rewording it or building its name at run time."