Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 50 additions & 9 deletions docs/adr/0030-anonymization-test-harness-tee.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,32 @@
characters of any one key or value, and of the list of unknown keys. **This changes none of the
§2 ratified choices:** the overlay still only maps a field to an existing kind.
[PHI.md](../PHI.md) §9 is the long form for the retarget.
- **AMENDMENT PROPOSED 2026-10-27, AWAITING THE OWNER'S SIGN-OFF (BACKLOG #1192,
[ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md)
section 5 and slice 5): the package's home moves out of the engine.** ADR 0201 was accepted on
2026-09-29 and designs this amendment. Its section 5 also says that changing this ADR's ratified
home needs the owner's sign-off on its own, in its own pull request. So the four items below are
proposed, not ratified, until that sign-off is recorded in this bullet. This builds nothing either
way: the package is still at `messagefoundry/anon/`, and every path in this ADR describes the
tree until ADR 0201 slice 6 moves it. That slice re-points the guards and tests that pin the
path, in the same change.
1. **The home becomes `messagefoundry_toolkit/anon/`**, in the `messagefoundry-toolkit`
distribution rather than the engine wheel. This replaces the §1 *"engine-owned"* home *"sitting
beside `parsing/`"*, one of the forks ratified before the build.
2. **The rule table stays the single authority, and the tee still vendors a copy at
`tee/anon/`.** ADR 0201 section 5 calls it a byte-identical copy, as §1 does. Not every file
is: [`tests/test_anon_parity.py`](../../tests/test_anon_parity.py) records which are.
3. **The parity check re-points to the new path**, in slice 6.
4. **The carve-out extension is withdrawn.** The Related line and §1 say acceptance extends the
CLAUDE.md §4 pure-library carve-out from `parsing/` to `anon/`, and §8 and *To resolve on
acceptance* list that as a doc flip. That flip never landed: CLAUDE.md §4 names `parsing/`
alone. Once `anon/` is not an engine package there is nothing for the carve-out to cover, so
the flip is withdrawn rather than left pending. Each of those four places carries a marker.

What the package does, its rules, its fail-closed behaviour and its limits are unchanged.
**One correction here needs no sign-off**, because it records what the tree and an owner ruling
already say: §1 claimed the harness imports the package, and none does. The correction stands
where that sentence stood, with markers on the two other places that read as permission.
- **Decision in one line:** ship a **pure-stdlib, dependency-free `anon` package** that turns real,
messy HL7 v2 into structurally-faithful **PHI-free** datasets via a **two-layer rule model — a
declarative field-*selection* map (data) over a code registry of pure surrogate *functions* (logic)** —
Expand All @@ -78,8 +104,8 @@
[PHI.md](../PHI.md) §9 (de-identification `[ROADMAP]` — on landing this *would be* the first concrete
slice that flips it) / §7 (redaction is *not* de-id); [CLAUDE.md](../../CLAUDE.md) §9 (de-id planned-not-
built, centralize the rules) / §8 (read separators from MSH, never string-slice, parse defensively) / §4
(`parsing/` is the one pure carve-out lib; this ADR extends that carve-out to `anon/` on acceptance —
dependency direction one-way) / §12 (don't drift to declarative *logic*).
(`parsing/` is the one pure carve-out lib; this ADR extends that carve-out to `anon/` on acceptance
(*withdrawal proposed 2026-10-27, see the status banner*) — dependency direction one-way) / §12 (don't drift to declarative *logic*).

## Context

Expand All @@ -102,6 +128,8 @@ verbatim is *"conservative redaction, **not** de-identification."* This ADR must
The constraint that shapes everything is a **dependency-split asymmetry**:
- The **harness** freely imports the engine (`messagefoundry.parsing`, `.generators`, …) — it already
imports engine packages directly, so a new `messagefoundry.anon` is an ordinary import for it.
*(Corrected 2026-10-27: no harness file imports this package; see the correction under §1's
vendored-copy paragraph.)*
- The **tee** is **deliberately `messagefoundry`-free** (verified: zero `from messagefoundry` imports). It
**vendors** [`tee/hl7_fields.py`](../../tee/hl7_fields.py) and `tee/mllp.py` precisely so the cutover
relay can sit on the Epic/Corepoint boundary without dragging in the engine or its attack surface.
Expand All @@ -122,7 +150,10 @@ The anonymizer ships as **`messagefoundry/anon/`**: a small, **pure-stdlib** pac
`parsing/` it is a pure, side-effect-free library, but the §4 carve-out is **verbatim scoped to `parsing/`**
and does **not** automatically extend to a new sibling. Accepting this ADR **amends CLAUDE.md §4** to add
`anon/` to the console carve-out (listed in *To resolve on acceptance*); until then the console may not import
it. The harness is unaffected — it imports engine packages freely. Files:
it. *(Withdrawal proposed 2026-10-27: the flip never landed, and ADR 0201 moves `anon/` out of the
engine; see the status banner.)* The harness is unaffected — it imports engine packages freely.
*(Corrected 2026-10-27: not this package; see the correction under the vendored-copy paragraph
below.)* Files:

- `anon/rules.py` — the **default field-selection map** (data; §2): `FieldRule(path, SurrogateKind)` rows
naming *which* HL7 field gets *which kind* of surrogate. No transform expressions — just a field→kind map.
Expand Down Expand Up @@ -165,9 +196,19 @@ SURROGATES: dict[SurrogateKind, Callable[[str, DatasetContext], str]] = { ... }

**The tee vendors a byte-identical copy** at `tee/anon/` (mirroring how `tee/hl7_fields.py`/`tee/mllp.py`
were vendored), kept in sync by a CI parity check — the same discipline the tee already lives by. The
**harness imports `messagefoundry.anon` directly** (it already imports `parsing`/`generators`). This is the
**asymmetric resolution**: both sides run the *same rules*, the harness by import, the tee by a
parity-pinned vendor copy.
harness's [`CaptureSink`](../../harness/reconcile/capture.py) anonymizes only when its caller passes it
an anonymizer function, and no harness file imports the package. The shipped
`python -m harness.reconcile capture` passes none, so it writes the bodies it receives as they
arrive. The tee runs the rules by a parity-pinned vendor copy.

**CORRECTED 2026-10-27 (BACKLOG #1192; see the 2026-10-27 status-banner bullet):** the paragraph
read, after its first sentence, *"The **harness imports `messagefoundry.anon` directly** (it already
imports `parsing`/`generators`). This is the **asymmetric resolution**: both sides run the *same
rules*, the harness by import, the tee by a parity-pinned vendor copy."* No harness file imports the
package. The owner ruled on 2026-10-12 (BACKLOG #3006 part 3) that such an import gives way to the
record's claim that no file outside the package imports it, and an author who needs the package
waits for its move. `tests/test_dependency_boundaries.py` refuses an import of it from the source
trees it walks, the harness among them.

### 2. The rule model: field-*selection* is data, surrogate *production* is code (the load-bearing call)

Expand Down Expand Up @@ -366,8 +407,8 @@ re-identification/linkage tooling.

On landing, this **would become** the de-id framework's **first bounded slice** — a *test-data* anonymizer,
not the full enterprise pipeline. **While this ADR is Proposed nothing is built**, so it does **not** yet flip
[PHI.md](../PHI.md) §9 from `[ROADMAP]` to built — that flip, and the lockstep CLAUDE.md §9 / §4 (carve-out)
and PHI.md §12/§13 edits, happen **only when the code lands**, never on acceptance (CLAUDE.md §9: "don't
[PHI.md](../PHI.md) §9 from `[ROADMAP]` to built — that flip, and the lockstep CLAUDE.md §9 / §4 (carve-out;
*withdrawal proposed 2026-10-27, see the status banner*) and PHI.md §12/§13 edits, happen **only when the code lands**, never on acceptance (CLAUDE.md §9: "don't
reference a framework that doesn't exist"). The rule table is **designed to be reusable** as the future
AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on landing).

Expand Down Expand Up @@ -455,7 +496,7 @@ AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on
`tee export`; requires `--capture-bodies`; errors when no captures exist; no raw bodies to stdout/CI),
harness `CaptureSink._write` hook, and the file-backed corpus source.
- [ ] Approve the **doc lockstep, gated on code landing (not acceptance):** add `anon/` to the CLAUDE.md §4
console carve-out; flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13.
console carve-out (*withdrawal proposed 2026-10-27, see the status banner*); flip PHI.md §9 `[ROADMAP]`→built; update CLAUDE.md §9 and PHI.md §12/§13.

---

Expand Down
2 changes: 1 addition & 1 deletion docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ what is withheld and what you can request.
| [0027](0027-per-connection-retention.md) | Per-connection retention / pruning windows — a per-connection `messages_days` (inbound) / `dead_letter_days` (outbound) override layered over the global `[retention]` default (`None` = inherit), reusing the #46 per-connection-override plumbing + ADR 0007 `connections.toml`; threads a `{connection → cutoff}` map through the existing purge SQL on all three backends; one `retention_purge` audit row per pass; null-body-keep-metadata + the in-flight guard preserved (#34) | Accepted (built, 0.2.9) |
| [0028](0028-base64-binary-carriage-codec.md) | base64 binary-carriage codec (+ HL7 OBX-5 ED embedding) — carry arbitrary bytes over the str/TEXT ingress+store as `mfb64:v1:` unbroken base64; pure stdlib `parsing/binary.py` + `RawMessage.from_bytes`/`.raw_bytes`. Supersedes ADR 0025's latin-1 round-trip (NUL-unsafe across the store) | Accepted (2026-06-20) — Amended 2026-07-13: ERROR-path + happy-path NUL carriage (INGEST-4) — built (pure `parsing/binary.py` codec, the four `RawMessage` additions, OBX-5 ED helpers) |
| [0029](0029-email-smtp-destination.md) | Email (SMTP-send) outbound destination — a stdlib `smtplib`/`email` `DestinationConnector` + new `ConnectorType.EMAIL` + a code-first `Email()`/`SMTP()` factory; STARTTLS-by-default with the existing insecure-TLS escape; a new `[egress].allowed_smtp` deny-by-default allowlist arm; `DeliveryError` → staged-queue retry (transform stays pure, SMTP is the side effect, at-least-once like other one-way destinations). IMAP/POP inbound read + XOAUTH2 deferred to Phase 2 (#23; was earmarked 0024 before SMART claimed it) | Accepted (2026-06-27, built) |
| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) |
| [0030](0030-anonymization-test-harness-tee.md) | Anonymizer / de-identification for the test harness + tee (`messagefoundry.anon`) — build de-identified test datasets from real traffic (de-identified is not PHI-free: [PHI.md §9](../PHI.md#9-de-identification) lists what the leak-check refuses and what it lets through); pure-stdlib surrogate pools/encoders (lifted from `generators/_hl7data.py`); tee + harness hooks | Accepted (2026-06-20) -- amended since, at least five times; the ADR's status banner lists each. An amendment proposed 2026-10-27, awaiting the owner's sign-off, moves its home to `messagefoundry_toolkit/anon/` ([ADR 0201](0201-a-messagefoundry-toolkit-distribution-carries-the-authoring-and-development-tooling-out-of-the-engine-wheel.md) slice 6, not built) |
| [0031](0031-startup-connection-fault-isolation.md) | Startup connection fault isolation — a single connection that fails to build/bind at startup is isolated (logged + alerted + reported `failed`), not fatal: the engine starts the rest of the graph and serves the API; a failed outbound retries (never drops) and self-heals on reload/restart. Reload stays fail-fast. Amended 2026-10-06 (vault #2371): a refused outbound or `Ftp` poller CA degrades its lane at start; a reload refuses only for a lane it builds or keeps running; a `FhirLookup` CA and an inbound listener CA still refuse | Accepted (2026-06-21) |
| [0032](0032-console-desktop-launch.md) | Console desktop launch — a windowed `gui-script` (`messagefoundry-console`) + window/taskbar icon + Desktop/Start-Menu shortcut installer (Phase A); frozen installer (Phase B) retired. **RETIRED (2026-07-13, BACKLOG #103):** the whole PySide6 desktop console is removed — superseded by the browser web console (ADR 0065) + the `apiclient`/`messagefoundry service` extraction (ADR 0088) | RETIRED (2026-07-13) |
| [0033](0033-gui-manageable-code-sets.md) | GUI-manageable code sets (translation tables) — a `messagefoundry codeset` CLI (mirrors `connection`: owns validate-before-write + atomic temp+replace + owner-only perms + post-write re-load) + a VS Code webview grid editor that shells it, CSV-first for v1 (TOML sets summarized/shown read-only). Reference sets (ADR 0006) + `db_lookup` (ADR 0010) are out of scope (externally-owned data); a rename/remove can dangle a `code_set(...)` reference, caught by `messagefoundry check`'s dry-run | Accepted (built, 0.2.3, #540) |
Expand Down
11 changes: 6 additions & 5 deletions tests/test_dependency_boundaries.py
Original file line number Diff line number Diff line change
Expand Up @@ -1557,9 +1557,9 @@ def test_the_toolkit_search_sees_a_planted_name(tmp_path: Path, plant: str) -> N
# accident, not against an evasion.
#
# A REAL import is a different matter from a line that only looks like one. ADR 0030 section 1
# allows the harness to import the package directly, and nothing does yet. An owner ruling says
# which side gives way then, and `_REAL_ANON_IMPORT_ADVICE` cites it in every failure message
# that may be a real import.
# said the harness imports the package directly; nothing does, and its 2026-10-27 correction says
# so. An owner ruling says which side gives way, and `_REAL_ANON_IMPORT_ADVICE` cites it in every
# failure message that may be a real import.
#
# THE PACKAGE'S OWN FILES. The claim is about files outside `messagefoundry/anon/`, and the package
# imports its own modules, so the parse reports nothing there. It counts those imports instead, as
Expand Down Expand Up @@ -1646,8 +1646,9 @@ def test_the_toolkit_search_sees_a_planted_name(tmp_path: Path, plant: str) -> N

#: Shared by every failure that may be a real import, and so put first in each.
_REAL_ANON_IMPORT_ADVICE = (
"If a hit is a REAL import, the import must go. ADR 0030 section 1 allows the harness to "
"import the package. The record's absence claim says no file outside the package imports it. "
"If a hit is a REAL import, the import must go. ADR 0030 section 1 once said the harness "
"imports the package, and its correction says none does. The record's absence claim says no "
"file outside the package imports it. "
"The owner ruled that the import gives way to the claim (owner ruling 2026-10-12, BACKLOG "
"#3006 part 3). An author who needs the package waits for its move, BACKLOG #1192. Do not "
"hide the import from these searches by rewording it or building its name at run time."
Expand Down
Loading