From 662b32a1bc19e9b58ef7eb0e39d151a46d860844 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 27 Oct 2026 09:13:53 -0500 Subject: [PATCH 1/3] ADR 0201 slice 5: amend ADR 0030 for the planned anon/ move Documentation only. ADR 0030 gains a dated status-banner amendment recording ADR 0201 section 5: the package's home is planned to become messagefoundry_toolkit/anon/, the rule table stays the single authority with a tee copy, the parity check re-points in slice 6, and the CLAUDE.md section 4 carve-out extension to anon/ is withdrawn. Nothing moves; every path in ADR 0030 still describes the tree until slice 6. Corrects ADR 0030 section 1, which said the harness imports the package directly. No harness file does, per the owner ruling of 2026-10-12 (BACKLOG #3006 part 3); the old sentence is kept as a record. The ADR index row for 0030 records the amendment. No new ADR number. BACKLOG #1192 Proposed PR title: ADR 0201 slices 3 and 5: the toolkit name is published, and ADR 0030 records the planned anon/ move (BACKLOG #1192) Proposed ledger banner: #1192: ADR 0201 slices 3 and 5 built (the install-provenance guard lists messagefoundry-toolkit as published after a PyPI probe with a control; ADR 0030 amended for the planned anon/ move, with the harness-import sentence corrected). Slices 4, 6, 7 and the release-gated 15.2.3 re-score (slice 8) remain. --- .../0030-anonymization-test-harness-tee.md | 40 +++++++++++++++++-- docs/adr/README.md | 2 +- 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/docs/adr/0030-anonymization-test-harness-tee.md b/docs/adr/0030-anonymization-test-harness-tee.md index d097bf739..ee615ed9e 100644 --- a/docs/adr/0030-anonymization-test-harness-tee.md +++ b/docs/adr/0030-anonymization-test-harness-tee.md @@ -62,6 +62,31 @@ characters of any one key or value, and of the list of unknown keys. **This changes none of the §2 ratified choices:** the overlay still only maps a field to an existing kind. [PHI.md](../PHI.md) §9 is the long form for the retarget. +- **AMENDED 2026-10-27 (BACKLOG #1192, [ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) + section 5 and slice 5): the package's home is planned to move out of the engine.** This records a + planned move and builds nothing. The package is still at `messagefoundry/anon/`, and every path + in this ADR still describes the tree until ADR 0201 slice 6 moves it. That slice re-points the + guards and tests that pin the path, in the same change. ADR 0201 was accepted on 2026-09-29, and + this amendment records the §1 changes it designs: + 1. **The home becomes `messagefoundry_toolkit/anon/`**, in the `messagefoundry-toolkit` + distribution rather than the engine wheel. This replaces the §1 *"engine-owned"* home *"sitting + beside `parsing/`"*, one of the forks ratified before the build. + 2. **The rule table stays the single authority, and the tee still vendors a copy at + `tee/anon/`.** Which of its files are byte-identical is recorded by + [`tests/test_anon_parity.py`](../../tests/test_anon_parity.py), not by this ADR. + 3. **The parity check re-points to the new path**, in slice 6. + 4. **The carve-out language is withdrawn.** The Related line and §1 say acceptance extends the + CLAUDE.md §4 pure-library carve-out from `parsing/` to `anon/`, and §8 and *To resolve on + acceptance* list that as a doc flip. That flip never landed: CLAUDE.md §4 names `parsing/` + alone. Once `anon/` is not an engine package there is nothing for the carve-out to cover, so + the flip is withdrawn rather than left pending. + + What the package does, its rules, its fail-closed behaviour and its limits are unchanged. + **Correction in the same amendment:** §1 said the harness imports the package directly. No + harness file does, and the owner ruled on 2026-10-12 (BACKLOG #3006 part 3) that the import gives + way to the record's claim that no file outside the package imports it. §1 carries the correction + where the sentence stood. The Context bullet and §1's *"The harness is unaffected"* describe what + the harness may import from the engine in general, not an import of this package. - **Decision in one line:** ship a **pure-stdlib, dependency-free `anon` package** that turns real, messy HL7 v2 into structurally-faithful **PHI-free** datasets via a **two-layer rule model — a declarative field-*selection* map (data) over a code registry of pure surrogate *functions* (logic)** — @@ -165,9 +190,18 @@ SURROGATES: dict[SurrogateKind, Callable[[str, DatasetContext], str]] = { ... } **The tee vendors a byte-identical copy** at `tee/anon/` (mirroring how `tee/hl7_fields.py`/`tee/mllp.py` were vendored), kept in sync by a CI parity check — the same discipline the tee already lives by. The -**harness imports `messagefoundry.anon` directly** (it already imports `parsing`/`generators`). This is the -**asymmetric resolution**: both sides run the *same rules*, the harness by import, the tee by a -parity-pinned vendor copy. +harness's [`CaptureSink`](../../harness/reconcile/capture.py) takes the anonymizer as a function its +caller passes in, and no harness file imports the package. This is the **asymmetric resolution**: +both sides run the *same rules*, the harness through the anonymizer its caller hands the sink, the +tee by a parity-pinned vendor copy. + +**CORRECTED 2026-10-27 (BACKLOG #1192; see the 2026-10-27 status-banner amendment):** the second +sentence above read *"The **harness imports `messagefoundry.anon` directly** (it already imports +`parsing`/`generators`)."*, and the third ended *"the harness by import"*. No harness file imports +the package. The owner ruled on 2026-10-12 (BACKLOG #3006 part 3) that such an import gives way to +the record's claim that no file outside the package imports it, and an author who needs the package +waits for its move. `tests/test_dependency_boundaries.py` refuses an import of it from the source +trees it walks, the harness among them. ### 2. The rule model: field-*selection* is data, surrogate *production* is code (the load-bearing call) diff --git a/docs/adr/README.md b/docs/adr/README.md index 3e8c3addb..cc847c6cd 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -64,7 +64,7 @@ what is withheld and what you can request. | [0027](0027-per-connection-retention.md) | Per-connection retention / pruning windows — a per-connection `messages_days` (inbound) / `dead_letter_days` (outbound) override layered over the global `[retention]` default (`None` = inherit), reusing the #46 per-connection-override plumbing + ADR 0007 `connections.toml`; threads a `{connection → cutoff}` map through the existing purge SQL on all three backends; one `retention_purge` audit row per pass; null-body-keep-metadata + the in-flight guard preserved (#34) | Accepted (built, 0.2.9) | | [0028](0028-base64-binary-carriage-codec.md) | base64 binary-carriage codec (+ HL7 OBX-5 ED embedding) — carry arbitrary bytes over the str/TEXT ingress+store as `mfb64:v1:` unbroken base64; pure stdlib `parsing/binary.py` + `RawMessage.from_bytes`/`.raw_bytes`. Supersedes ADR 0025's latin-1 round-trip (NUL-unsafe across the store) | Accepted (2026-06-20) — Amended 2026-07-13: ERROR-path + happy-path NUL carriage (INGEST-4) — built (pure `parsing/binary.py` codec, the four `RawMessage` additions, OBX-5 ED helpers) | | [0029](0029-email-smtp-destination.md) | Email (SMTP-send) outbound destination — a stdlib `smtplib`/`email` `DestinationConnector` + new `ConnectorType.EMAIL` + a code-first `Email()`/`SMTP()` factory; STARTTLS-by-default with the existing insecure-TLS escape; a new `[egress].allowed_smtp` deny-by-default allowlist arm; `DeliveryError` → staged-queue retry (transform stays pure, SMTP is the side effect, at-least-once like other one-way destinations). IMAP/POP inbound read + XOAUTH2 deferred to Phase 2 (#23; was earmarked 0024 before SMART claimed it) | Accepted (2026-06-27, built) | -| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) | +| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) -- Amended 2026-10-27: the package's home is planned to move to `messagefoundry_toolkit/anon/` ([ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) slice 5; the move itself is slice 6, not built), the `parsing/` carve-out extension is withdrawn, and §1 no longer says the harness imports it | | [0031](0031-startup-connection-fault-isolation.md) | Startup connection fault isolation — a single connection that fails to build/bind at startup is isolated (logged + alerted + reported `failed`), not fatal: the engine starts the rest of the graph and serves the API; a failed outbound retries (never drops) and self-heals on reload/restart. Reload stays fail-fast. Amended 2026-10-06 (vault #2371): a refused outbound or `Ftp` poller CA degrades its lane at start; a reload refuses only for a lane it builds or keeps running; a `FhirLookup` CA and an inbound listener CA still refuse | Accepted (2026-06-21) | | [0032](0032-console-desktop-launch.md) | Console desktop launch — a windowed `gui-script` (`messagefoundry-console`) + window/taskbar icon + Desktop/Start-Menu shortcut installer (Phase A); frozen installer (Phase B) retired. **RETIRED (2026-07-13, BACKLOG #103):** the whole PySide6 desktop console is removed — superseded by the browser web console (ADR 0065) + the `apiclient`/`messagefoundry service` extraction (ADR 0088) | RETIRED (2026-07-13) | | [0033](0033-gui-manageable-code-sets.md) | GUI-manageable code sets (translation tables) — a `messagefoundry codeset` CLI (mirrors `connection`: owns validate-before-write + atomic temp+replace + owner-only perms + post-write re-load) + a VS Code webview grid editor that shells it, CSV-first for v1 (TOML sets summarized/shown read-only). Reference sets (ADR 0006) + `db_lookup` (ADR 0010) are out of scope (externally-owned data); a rename/remove can dangle a `code_set(...)` reference, caught by `messagefoundry check`'s dry-run | Accepted (built, 0.2.3, #540) | From f44b0dfde0b0ee59bdae98610e5289c4358c3bfc Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 27 Oct 2026 09:21:09 -0500 Subject: [PATCH 2/3] ADR 0201 slice 5: one home for the harness correction, and carve-out markers /simplify findings on the ADR 0030 amendment. The harness-import correction was stated in the status banner and again in section 1; the banner now points to section 1. The Related line, section 1 and the To resolve item still read as extending the parsing/ carve-out to anon/, so each carries a short withdrawn marker. The ADR index row's amendment note is cut to one clause. BACKLOG #1192 Proposed PR title: ADR 0201 slices 3 and 5: the toolkit name is published, and ADR 0030 records the planned anon/ move (BACKLOG #1192) Proposed ledger banner: #1192: ADR 0201 slices 3 and 5 built (the install-provenance guard lists messagefoundry-toolkit as published after a PyPI probe with a control; ADR 0030 amended for the planned anon/ move, with the harness-import sentence corrected). Slices 4, 6, 7 and the release-gated 15.2.3 re-score (slice 8) remain. --- docs/adr/0030-anonymization-test-harness-tee.md | 16 +++++++--------- docs/adr/README.md | 2 +- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/adr/0030-anonymization-test-harness-tee.md b/docs/adr/0030-anonymization-test-harness-tee.md index ee615ed9e..2fd383178 100644 --- a/docs/adr/0030-anonymization-test-harness-tee.md +++ b/docs/adr/0030-anonymization-test-harness-tee.md @@ -81,12 +81,9 @@ alone. Once `anon/` is not an engine package there is nothing for the carve-out to cover, so the flip is withdrawn rather than left pending. - What the package does, its rules, its fail-closed behaviour and its limits are unchanged. - **Correction in the same amendment:** §1 said the harness imports the package directly. No - harness file does, and the owner ruled on 2026-10-12 (BACKLOG #3006 part 3) that the import gives - way to the record's claim that no file outside the package imports it. §1 carries the correction - where the sentence stood. The Context bullet and §1's *"The harness is unaffected"* describe what - the harness may import from the engine in general, not an import of this package. + What the package does, its rules, its fail-closed behaviour and its limits are unchanged. The same + amendment corrects §1's claim that the harness imports the package; the correction stands where + that sentence stood. - **Decision in one line:** ship a **pure-stdlib, dependency-free `anon` package** that turns real, messy HL7 v2 into structurally-faithful **PHI-free** datasets via a **two-layer rule model — a declarative field-*selection* map (data) over a code registry of pure surrogate *functions* (logic)** — @@ -104,7 +101,7 @@ slice that flips it) / §7 (redaction is *not* de-id); [CLAUDE.md](../../CLAUDE.md) §9 (de-id planned-not- built, centralize the rules) / §8 (read separators from MSH, never string-slice, parse defensively) / §4 (`parsing/` is the one pure carve-out lib; this ADR extends that carve-out to `anon/` on acceptance — - dependency direction one-way) / §12 (don't drift to declarative *logic*). + dependency direction one-way; *withdrawn 2026-10-27, see the status-banner amendment*) / §12 (don't drift to declarative *logic*). ## Context @@ -147,7 +144,8 @@ The anonymizer ships as **`messagefoundry/anon/`**: a small, **pure-stdlib** pac `parsing/` it is a pure, side-effect-free library, but the §4 carve-out is **verbatim scoped to `parsing/`** and does **not** automatically extend to a new sibling. Accepting this ADR **amends CLAUDE.md §4** to add `anon/` to the console carve-out (listed in *To resolve on acceptance*); until then the console may not import -it. The harness is unaffected — it imports engine packages freely. Files: +it. *(Withdrawn 2026-10-27: the flip never landed, and ADR 0201 moves `anon/` out of the engine; see +the status-banner amendment.)* The harness is unaffected — it imports engine packages freely. Files: - `anon/rules.py` — the **default field-selection map** (data; §2): `FieldRule(path, SurrogateKind)` rows naming *which* HL7 field gets *which kind* of surrogate. No transform expressions — just a field→kind map. @@ -489,7 +487,7 @@ AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on `tee export`; requires `--capture-bodies`; errors when no captures exist; no raw bodies to stdout/CI), harness `CaptureSink._write` hook, and the file-backed corpus source. - [ ] Approve the **doc lockstep, gated on code landing (not acceptance):** add `anon/` to the CLAUDE.md §4 - console carve-out; flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13. + console carve-out (*withdrawn 2026-10-27, see the status-banner amendment*); flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13. --- diff --git a/docs/adr/README.md b/docs/adr/README.md index cc847c6cd..ec098afb9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -64,7 +64,7 @@ what is withheld and what you can request. | [0027](0027-per-connection-retention.md) | Per-connection retention / pruning windows — a per-connection `messages_days` (inbound) / `dead_letter_days` (outbound) override layered over the global `[retention]` default (`None` = inherit), reusing the #46 per-connection-override plumbing + ADR 0007 `connections.toml`; threads a `{connection → cutoff}` map through the existing purge SQL on all three backends; one `retention_purge` audit row per pass; null-body-keep-metadata + the in-flight guard preserved (#34) | Accepted (built, 0.2.9) | | [0028](0028-base64-binary-carriage-codec.md) | base64 binary-carriage codec (+ HL7 OBX-5 ED embedding) — carry arbitrary bytes over the str/TEXT ingress+store as `mfb64:v1:` unbroken base64; pure stdlib `parsing/binary.py` + `RawMessage.from_bytes`/`.raw_bytes`. Supersedes ADR 0025's latin-1 round-trip (NUL-unsafe across the store) | Accepted (2026-06-20) — Amended 2026-07-13: ERROR-path + happy-path NUL carriage (INGEST-4) — built (pure `parsing/binary.py` codec, the four `RawMessage` additions, OBX-5 ED helpers) | | [0029](0029-email-smtp-destination.md) | Email (SMTP-send) outbound destination — a stdlib `smtplib`/`email` `DestinationConnector` + new `ConnectorType.EMAIL` + a code-first `Email()`/`SMTP()` factory; STARTTLS-by-default with the existing insecure-TLS escape; a new `[egress].allowed_smtp` deny-by-default allowlist arm; `DeliveryError` → staged-queue retry (transform stays pure, SMTP is the side effect, at-least-once like other one-way destinations). IMAP/POP inbound read + XOAUTH2 deferred to Phase 2 (#23; was earmarked 0024 before SMART claimed it) | Accepted (2026-06-27, built) | -| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) -- Amended 2026-10-27: the package's home is planned to move to `messagefoundry_toolkit/anon/` ([ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) slice 5; the move itself is slice 6, not built), the `parsing/` carve-out extension is withdrawn, and §1 no longer says the harness imports it | +| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) -- Amended 2026-10-27: home planned to move to `messagefoundry_toolkit/anon/` ([ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) slice 6, not built) | | [0031](0031-startup-connection-fault-isolation.md) | Startup connection fault isolation — a single connection that fails to build/bind at startup is isolated (logged + alerted + reported `failed`), not fatal: the engine starts the rest of the graph and serves the API; a failed outbound retries (never drops) and self-heals on reload/restart. Reload stays fail-fast. Amended 2026-10-06 (vault #2371): a refused outbound or `Ftp` poller CA degrades its lane at start; a reload refuses only for a lane it builds or keeps running; a `FhirLookup` CA and an inbound listener CA still refuse | Accepted (2026-06-21) | | [0032](0032-console-desktop-launch.md) | Console desktop launch — a windowed `gui-script` (`messagefoundry-console`) + window/taskbar icon + Desktop/Start-Menu shortcut installer (Phase A); frozen installer (Phase B) retired. **RETIRED (2026-07-13, BACKLOG #103):** the whole PySide6 desktop console is removed — superseded by the browser web console (ADR 0065) + the `apiclient`/`messagefoundry service` extraction (ADR 0088) | RETIRED (2026-07-13) | | [0033](0033-gui-manageable-code-sets.md) | GUI-manageable code sets (translation tables) — a `messagefoundry codeset` CLI (mirrors `connection`: owns validate-before-write + atomic temp+replace + owner-only perms + post-write re-load) + a VS Code webview grid editor that shells it, CSV-first for v1 (TOML sets summarized/shown read-only). Reference sets (ADR 0006) + `db_lookup` (ADR 0010) are out of scope (externally-owned data); a rename/remove can dangle a `code_set(...)` reference, caught by `messagefoundry check`'s dry-run | Accepted (built, 0.2.3, #540) | From e110c3a2c939987ef3ec5e12a61fbc159ac3a894 Mon Sep 17 00:00:00 2001 From: Scott Hall Date: Tue, 27 Oct 2026 09:50:33 -0500 Subject: [PATCH 3/3] ADR 0201 slice 5: code-review round 1 repairs ADR 0201 section 5 says changing ADR 0030's ratified home needs the owner's sign-off on its own, so the banner bullet is now a PROPOSED amendment awaiting that sign-off, and each carve-out marker says "withdrawal proposed" (section 8 gains one; the Related-line marker sits on the carve-out clause, not the one-way rule). The harness correction: section 1 now says CaptureSink anonymizes only when its caller passes a function, and the shipped capture command passes none; the Context bullet and "The harness is unaffected" carry markers. Item 2 notes that not every tee file is byte-identical. The index row says the ADR has earlier amendments. test_dependency_boundaries' advice and comment no longer cite section 1 as allowing the harness import. BACKLOG #1192 Proposed PR title: ADR 0201 slice 5: ADR 0030 proposes the anon/ move and corrects its harness-import sentence (BACKLOG #1192) Proposed ledger banner: #1192: ADR 0201 slice 5 built (ADR 0030 carries a proposed amendment for the anon/ move, awaiting the owner's sign-off on this PR, and its harness-import sentence is corrected). Slice 3 is on its own PR; slices 4, 6, 7 and the release-gated 15.2.3 re-score (slice 8) remain. --- .../0030-anonymization-test-harness-tee.md | 69 +++++++++++-------- docs/adr/README.md | 2 +- tests/test_dependency_boundaries.py | 11 +-- 3 files changed, 46 insertions(+), 36 deletions(-) diff --git a/docs/adr/0030-anonymization-test-harness-tee.md b/docs/adr/0030-anonymization-test-harness-tee.md index 2fd383178..0ee1f7fe7 100644 --- a/docs/adr/0030-anonymization-test-harness-tee.md +++ b/docs/adr/0030-anonymization-test-harness-tee.md @@ -62,28 +62,32 @@ characters of any one key or value, and of the list of unknown keys. **This changes none of the §2 ratified choices:** the overlay still only maps a field to an existing kind. [PHI.md](../PHI.md) §9 is the long form for the retarget. -- **AMENDED 2026-10-27 (BACKLOG #1192, [ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) - section 5 and slice 5): the package's home is planned to move out of the engine.** This records a - planned move and builds nothing. The package is still at `messagefoundry/anon/`, and every path - in this ADR still describes the tree until ADR 0201 slice 6 moves it. That slice re-points the - guards and tests that pin the path, in the same change. ADR 0201 was accepted on 2026-09-29, and - this amendment records the §1 changes it designs: +- **AMENDMENT PROPOSED 2026-10-27, AWAITING THE OWNER'S SIGN-OFF (BACKLOG #1192, + [ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) + section 5 and slice 5): the package's home moves out of the engine.** ADR 0201 was accepted on + 2026-09-29 and designs this amendment. Its section 5 also says that changing this ADR's ratified + home needs the owner's sign-off on its own, in its own pull request. So the four items below are + proposed, not ratified, until that sign-off is recorded in this bullet. This builds nothing either + way: the package is still at `messagefoundry/anon/`, and every path in this ADR describes the + tree until ADR 0201 slice 6 moves it. That slice re-points the guards and tests that pin the + path, in the same change. 1. **The home becomes `messagefoundry_toolkit/anon/`**, in the `messagefoundry-toolkit` distribution rather than the engine wheel. This replaces the §1 *"engine-owned"* home *"sitting beside `parsing/`"*, one of the forks ratified before the build. 2. **The rule table stays the single authority, and the tee still vendors a copy at - `tee/anon/`.** Which of its files are byte-identical is recorded by - [`tests/test_anon_parity.py`](../../tests/test_anon_parity.py), not by this ADR. + `tee/anon/`.** ADR 0201 section 5 calls it a byte-identical copy, as §1 does. Not every file + is: [`tests/test_anon_parity.py`](../../tests/test_anon_parity.py) records which are. 3. **The parity check re-points to the new path**, in slice 6. - 4. **The carve-out language is withdrawn.** The Related line and §1 say acceptance extends the + 4. **The carve-out extension is withdrawn.** The Related line and §1 say acceptance extends the CLAUDE.md §4 pure-library carve-out from `parsing/` to `anon/`, and §8 and *To resolve on acceptance* list that as a doc flip. That flip never landed: CLAUDE.md §4 names `parsing/` alone. Once `anon/` is not an engine package there is nothing for the carve-out to cover, so - the flip is withdrawn rather than left pending. + the flip is withdrawn rather than left pending. Each of those four places carries a marker. - What the package does, its rules, its fail-closed behaviour and its limits are unchanged. The same - amendment corrects §1's claim that the harness imports the package; the correction stands where - that sentence stood. + What the package does, its rules, its fail-closed behaviour and its limits are unchanged. + **One correction here needs no sign-off**, because it records what the tree and an owner ruling + already say: §1 claimed the harness imports the package, and none does. The correction stands + where that sentence stood, with markers on the two other places that read as permission. - **Decision in one line:** ship a **pure-stdlib, dependency-free `anon` package** that turns real, messy HL7 v2 into structurally-faithful **PHI-free** datasets via a **two-layer rule model — a declarative field-*selection* map (data) over a code registry of pure surrogate *functions* (logic)** — @@ -100,8 +104,8 @@ [PHI.md](../PHI.md) §9 (de-identification `[ROADMAP]` — on landing this *would be* the first concrete slice that flips it) / §7 (redaction is *not* de-id); [CLAUDE.md](../../CLAUDE.md) §9 (de-id planned-not- built, centralize the rules) / §8 (read separators from MSH, never string-slice, parse defensively) / §4 - (`parsing/` is the one pure carve-out lib; this ADR extends that carve-out to `anon/` on acceptance — - dependency direction one-way; *withdrawn 2026-10-27, see the status-banner amendment*) / §12 (don't drift to declarative *logic*). + (`parsing/` is the one pure carve-out lib; this ADR extends that carve-out to `anon/` on acceptance + (*withdrawal proposed 2026-10-27, see the status banner*) — dependency direction one-way) / §12 (don't drift to declarative *logic*). ## Context @@ -124,6 +128,8 @@ verbatim is *"conservative redaction, **not** de-identification."* This ADR must The constraint that shapes everything is a **dependency-split asymmetry**: - The **harness** freely imports the engine (`messagefoundry.parsing`, `.generators`, …) — it already imports engine packages directly, so a new `messagefoundry.anon` is an ordinary import for it. + *(Corrected 2026-10-27: no harness file imports this package; see the correction under §1's + vendored-copy paragraph.)* - The **tee** is **deliberately `messagefoundry`-free** (verified: zero `from messagefoundry` imports). It **vendors** [`tee/hl7_fields.py`](../../tee/hl7_fields.py) and `tee/mllp.py` precisely so the cutover relay can sit on the Epic/Corepoint boundary without dragging in the engine or its attack surface. @@ -144,8 +150,10 @@ The anonymizer ships as **`messagefoundry/anon/`**: a small, **pure-stdlib** pac `parsing/` it is a pure, side-effect-free library, but the §4 carve-out is **verbatim scoped to `parsing/`** and does **not** automatically extend to a new sibling. Accepting this ADR **amends CLAUDE.md §4** to add `anon/` to the console carve-out (listed in *To resolve on acceptance*); until then the console may not import -it. *(Withdrawn 2026-10-27: the flip never landed, and ADR 0201 moves `anon/` out of the engine; see -the status-banner amendment.)* The harness is unaffected — it imports engine packages freely. Files: +it. *(Withdrawal proposed 2026-10-27: the flip never landed, and ADR 0201 moves `anon/` out of the +engine; see the status banner.)* The harness is unaffected — it imports engine packages freely. +*(Corrected 2026-10-27: not this package; see the correction under the vendored-copy paragraph +below.)* Files: - `anon/rules.py` — the **default field-selection map** (data; §2): `FieldRule(path, SurrogateKind)` rows naming *which* HL7 field gets *which kind* of surrogate. No transform expressions — just a field→kind map. @@ -188,16 +196,17 @@ SURROGATES: dict[SurrogateKind, Callable[[str, DatasetContext], str]] = { ... } **The tee vendors a byte-identical copy** at `tee/anon/` (mirroring how `tee/hl7_fields.py`/`tee/mllp.py` were vendored), kept in sync by a CI parity check — the same discipline the tee already lives by. The -harness's [`CaptureSink`](../../harness/reconcile/capture.py) takes the anonymizer as a function its -caller passes in, and no harness file imports the package. This is the **asymmetric resolution**: -both sides run the *same rules*, the harness through the anonymizer its caller hands the sink, the -tee by a parity-pinned vendor copy. - -**CORRECTED 2026-10-27 (BACKLOG #1192; see the 2026-10-27 status-banner amendment):** the second -sentence above read *"The **harness imports `messagefoundry.anon` directly** (it already imports -`parsing`/`generators`)."*, and the third ended *"the harness by import"*. No harness file imports -the package. The owner ruled on 2026-10-12 (BACKLOG #3006 part 3) that such an import gives way to -the record's claim that no file outside the package imports it, and an author who needs the package +harness's [`CaptureSink`](../../harness/reconcile/capture.py) anonymizes only when its caller passes it +an anonymizer function, and no harness file imports the package. The shipped +`python -m harness.reconcile capture` passes none, so it writes the bodies it receives as they +arrive. The tee runs the rules by a parity-pinned vendor copy. + +**CORRECTED 2026-10-27 (BACKLOG #1192; see the 2026-10-27 status-banner bullet):** the paragraph +read, after its first sentence, *"The **harness imports `messagefoundry.anon` directly** (it already +imports `parsing`/`generators`). This is the **asymmetric resolution**: both sides run the *same +rules*, the harness by import, the tee by a parity-pinned vendor copy."* No harness file imports the +package. The owner ruled on 2026-10-12 (BACKLOG #3006 part 3) that such an import gives way to the +record's claim that no file outside the package imports it, and an author who needs the package waits for its move. `tests/test_dependency_boundaries.py` refuses an import of it from the source trees it walks, the harness among them. @@ -398,8 +407,8 @@ re-identification/linkage tooling. On landing, this **would become** the de-id framework's **first bounded slice** — a *test-data* anonymizer, not the full enterprise pipeline. **While this ADR is Proposed nothing is built**, so it does **not** yet flip -[PHI.md](../PHI.md) §9 from `[ROADMAP]` to built — that flip, and the lockstep CLAUDE.md §9 / §4 (carve-out) -and PHI.md §12/§13 edits, happen **only when the code lands**, never on acceptance (CLAUDE.md §9: "don't +[PHI.md](../PHI.md) §9 from `[ROADMAP]` to built — that flip, and the lockstep CLAUDE.md §9 / §4 (carve-out; +*withdrawal proposed 2026-10-27, see the status banner*) and PHI.md §12/§13 edits, happen **only when the code lands**, never on acceptance (CLAUDE.md §9: "don't reference a framework that doesn't exist"). The rule table is **designed to be reusable** as the future AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on landing). @@ -487,7 +496,7 @@ AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on `tee export`; requires `--capture-bodies`; errors when no captures exist; no raw bodies to stdout/CI), harness `CaptureSink._write` hook, and the file-backed corpus source. - [ ] Approve the **doc lockstep, gated on code landing (not acceptance):** add `anon/` to the CLAUDE.md §4 - console carve-out (*withdrawn 2026-10-27, see the status-banner amendment*); flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13. + console carve-out (*withdrawal proposed 2026-10-27, see the status banner*); flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13. --- diff --git a/docs/adr/README.md b/docs/adr/README.md index ec098afb9..0c6961a8a 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -64,7 +64,7 @@ what is withheld and what you can request. | [0027](0027-per-connection-retention.md) | Per-connection retention / pruning windows — a per-connection `messages_days` (inbound) / `dead_letter_days` (outbound) override layered over the global `[retention]` default (`None` = inherit), reusing the #46 per-connection-override plumbing + ADR 0007 `connections.toml`; threads a `{connection → cutoff}` map through the existing purge SQL on all three backends; one `retention_purge` audit row per pass; null-body-keep-metadata + the in-flight guard preserved (#34) | Accepted (built, 0.2.9) | | [0028](0028-base64-binary-carriage-codec.md) | base64 binary-carriage codec (+ HL7 OBX-5 ED embedding) — carry arbitrary bytes over the str/TEXT ingress+store as `mfb64:v1:` unbroken base64; pure stdlib `parsing/binary.py` + `RawMessage.from_bytes`/`.raw_bytes`. Supersedes ADR 0025's latin-1 round-trip (NUL-unsafe across the store) | Accepted (2026-06-20) — Amended 2026-07-13: ERROR-path + happy-path NUL carriage (INGEST-4) — built (pure `parsing/binary.py` codec, the four `RawMessage` additions, OBX-5 ED helpers) | | [0029](0029-email-smtp-destination.md) | Email (SMTP-send) outbound destination — a stdlib `smtplib`/`email` `DestinationConnector` + new `ConnectorType.EMAIL` + a code-first `Email()`/`SMTP()` factory; STARTTLS-by-default with the existing insecure-TLS escape; a new `[egress].allowed_smtp` deny-by-default allowlist arm; `DeliveryError` → staged-queue retry (transform stays pure, SMTP is the side effect, at-least-once like other one-way destinations). IMAP/POP inbound read + XOAUTH2 deferred to Phase 2 (#23; was earmarked 0024 before SMART claimed it) | Accepted (2026-06-27, built) | -| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) -- Amended 2026-10-27: home planned to move to `messagefoundry_toolkit/anon/` ([ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) slice 6, not built) | +| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) -- amended since, at least five times; the ADR's status banner lists each. An amendment proposed 2026-10-27, awaiting the owner's sign-off, moves its home to `messagefoundry_toolkit/anon/` ([ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) slice 6, not built) | | [0031](0031-startup-connection-fault-isolation.md) | Startup connection fault isolation — a single connection that fails to build/bind at startup is isolated (logged + alerted + reported `failed`), not fatal: the engine starts the rest of the graph and serves the API; a failed outbound retries (never drops) and self-heals on reload/restart. Reload stays fail-fast. Amended 2026-10-06 (vault #2371): a refused outbound or `Ftp` poller CA degrades its lane at start; a reload refuses only for a lane it builds or keeps running; a `FhirLookup` CA and an inbound listener CA still refuse | Accepted (2026-06-21) | | [0032](0032-console-desktop-launch.md) | Console desktop launch — a windowed `gui-script` (`messagefoundry-console`) + window/taskbar icon + Desktop/Start-Menu shortcut installer (Phase A); frozen installer (Phase B) retired. **RETIRED (2026-07-13, BACKLOG #103):** the whole PySide6 desktop console is removed — superseded by the browser web console (ADR 0065) + the `apiclient`/`messagefoundry service` extraction (ADR 0088) | RETIRED (2026-07-13) | | [0033](0033-gui-manageable-code-sets.md) | GUI-manageable code sets (translation tables) — a `messagefoundry codeset` CLI (mirrors `connection`: owns validate-before-write + atomic temp+replace + owner-only perms + post-write re-load) + a VS Code webview grid editor that shells it, CSV-first for v1 (TOML sets summarized/shown read-only). Reference sets (ADR 0006) + `db_lookup` (ADR 0010) are out of scope (externally-owned data); a rename/remove can dangle a `code_set(...)` reference, caught by `messagefoundry check`'s dry-run | Accepted (built, 0.2.3, #540) | diff --git a/tests/test_dependency_boundaries.py b/tests/test_dependency_boundaries.py index 699b75257..4335ebda5 100644 --- a/tests/test_dependency_boundaries.py +++ b/tests/test_dependency_boundaries.py @@ -1557,9 +1557,9 @@ def test_the_toolkit_search_sees_a_planted_name(tmp_path: Path, plant: str) -> N # accident, not against an evasion. # # A REAL import is a different matter from a line that only looks like one. ADR 0030 section 1 -# allows the harness to import the package directly, and nothing does yet. An owner ruling says -# which side gives way then, and `_REAL_ANON_IMPORT_ADVICE` cites it in every failure message -# that may be a real import. +# said the harness imports the package directly; nothing does, and its 2026-10-27 correction says +# so. An owner ruling says which side gives way, and `_REAL_ANON_IMPORT_ADVICE` cites it in every +# failure message that may be a real import. # # THE PACKAGE'S OWN FILES. The claim is about files outside `messagefoundry/anon/`, and the package # imports its own modules, so the parse reports nothing there. It counts those imports instead, as @@ -1646,8 +1646,9 @@ def test_the_toolkit_search_sees_a_planted_name(tmp_path: Path, plant: str) -> N #: Shared by every failure that may be a real import, and so put first in each. _REAL_ANON_IMPORT_ADVICE = ( - "If a hit is a REAL import, the import must go. ADR 0030 section 1 allows the harness to " - "import the package. The record's absence claim says no file outside the package imports it. " + "If a hit is a REAL import, the import must go. ADR 0030 section 1 once said the harness " + "imports the package, and its correction says none does. The record's absence claim says no " + "file outside the package imports it. " "The owner ruled that the import gives way to the claim (owner ruling 2026-10-12, BACKLOG " "#3006 part 3). An author who needs the package waits for its move, BACKLOG #1192. Do not " "hide the import from these searches by rewording it or building its name at run time."