fix: preserve mail evidence and correct RFC parsing

.claude/agents/security-reviewer.md

@@ -134,8 +134,12 @@ delimiter, check the search is anchored:
   loop then falls through to older, fully attacker-forged `Received` headers —
   so the failure mode is not "returns nothing", it is "returns the attacker's
   value". Always test the fall-through, not just the single-header case.
-- Use the existing anchored `const._CLAUSE_SPLITTER` rather than a bare `\bby\b`
-  (`\b` still matches inside `host.by.example`, since `.` is a non-word char).
+- Use the shared context-aware scanner in `received.py` through
+  `utils.get_from_clause()`. A bare `\bby\b` still matches inside
+  `host.by.example`; keywords and semicolons inside comments, quoted strings,
+  domain literals and angle addresses are not clause boundaries. Check HELO
+  exclusions cover the remainder of their comment, and measure many IP groups
+  independently to catch repeated full-span scans.
 
 Prove these with a control matrix, not a single PoC: benign hostname, malicious
 hostname alone, forged header alone, and both together. If a benign hostname

AGENTS.md

@@ -114,6 +114,8 @@ optional Outlook dependencies are installed with `uv sync --extra outlook`).
 | --------------- | --------------------------------------------------------------------- |
 | `core.py`       | `MailParser` class + `parse_from_*` factory functions                 |
 | `addresses.py`  | Forensic address scanner and recovery diagnostics                     |
+| `received.py`   | Context-aware lexical scanning of SMTP trace clauses                  |
+| `dates.py`      | Date semantics, obsolete forms and occurrence-aware diagnostics       |
 | `utils.py`      | Stateless parsing helpers (address, received headers, date, encoding) |
 | `const.py`      | Compiled regexes, header sets, clause splitter                        |
 | `exceptions.py` | Custom exception hierarchy                                            |
@@ -163,31 +165,38 @@ header occurrence while retaining first-occurrence semantics for address
 properties. `address_header_defects` is reserved computed metadata; literal
 wire headers with that name remain accessible through `headers`/`message`.
 
-**Received header parsing**: `utils.receiveds_parsing()` tokenizes on RFC 5321 clause keywords
-(`from`, `by`, `via`, `with`, `id`, `for`, `envelope-from`) using `const._CLAUSE_SPLITTER`.
-Output list is ordered first-hop first. Unparseable headers fall back to `{"raw": ...}`.
+**Received header parsing**: `utils.receiveds_parsing()` uses the shared scanner in
+`received.py` for RFC 5321 clause keywords and the timestamp separator. Delimiters
+inside comments, quoted strings, domain literals and angle addresses are not
+structural separators. `get_from_clause()` uses the same scanner. Output is
+ordered first-hop first. Unparseable headers retain their original `raw` value;
+ambiguous syntax also produces `received_header_defects` with occurrence context.
 
 **Sender-IP attribution fails closed**: `get_server_ipaddress()` walks trust-matching `Received`
 headers only while a hop names a *private* IP (an internal relay); a hop naming **no** IP ends the
 search with `None`. Never resume the walk on that case — older `Received` headers are written by
 the sender, so "extraction failed" would become "returns the sender's chosen IP".
 
 Candidate addresses come only from the `from` clause (`utils.get_from_clause()`, which ends at the
-next RFC 5321 keyword). Inside it, `_sender_ip_candidates()` applies one positional rule, because
+next top-level RFC 5321 keyword). Inside it, `_sender_ip_candidates()` applies one positional rule, because
 **text cannot be classified by what it looks like** — a closed `[...]` pair the sender wrote is
 byte-identical to one the MTA wrote, and `EHLO [8.8.8.8]` is a form RFC 5321 §4.1.3 requires:
 
 - the **first token** is the HELO name, whatever its shape, and is never a candidate;
 - an explicit **HELO marker inside a comment group** (`(helo=x)`, `(account a@b HELO x)`) is
-  sender text and is excluded, located with `const._HELO_RE`;
+  located structurally by `received.helo_argument_spans()`; its argument and the remainder of that comment
+  are sender text and are excluded;
 - a candidate must sit **inside a `(`/`[` group** (`utils.group_spans()`), which is what makes a
   clause truncated by a multi-word HELO fail closed — what it leaves behind is bare;
 - IPv4 and IPv6 matches are merged **positionally**, never family-first: choosing IPv4 first let
   one private literal at EHLO suppress the IPv6 scan and hide the real sender.
 
+Use ordered span lookup for IP candidates: rescanning all comment groups for
+each candidate makes attribution quadratic on attacker-chosen trace fields.
+
 Only concession: `from [ip] (helo=x)` (Exim/CommuniGate), accepted when there is no other
-candidate and the marker is in a group. Do not add lookbehind guards to `_HELO_RE` to patch new
-cases — three rounds of that each reopened a hole the previous one closed.
+candidate and the marker is in a group. Keep marker recognition inside the structural scanner; do not reintroduce regex
+lookbehind patches for each new HELO layout.
 
 **Defect detection**: During `parse()`, every MIME part is walked and `_append_defects()` records
 RFC violations. `EPILOGUE_DEFECTS` triggers special epilogue extraction to recover hidden payloads

README.md

@@ -117,13 +117,36 @@ Or contribute with Bitcoin:
 
 Thank you for supporting the evolution of mail-parser!
 
-# mail-parser on Web
+# mail-parser in the ecosystem
 
-Explore mail-parser on these platforms:
+## Packages and distributions
 
-- **[FreeBSD port](https://www.freshports.org/mail/py-mail-parser/)**
-- **[Arch User Repository](https://aur.archlinux.org/packages/mailparser/)**
-- **[REMnux](https://docs.remnux.org/discover-the-tools/analyze+documents/email+messages#mail-parser)**
+Find mail-parser in these package repositories and security toolkits:
+
+- **[FreeBSD](https://www.freshports.org/mail/py-mail-parser/)**: available as the
+  `mail/py-mail-parser` port.
+- **[Arch User Repository (AUR)](https://aur.archlinux.org/packages/mailparser/)**:
+  community-maintained `mailparser` package for Arch Linux.
+- **[Debian](https://packages.debian.org/source/sid/mail-parser)**: `mail-parser`
+  source package in Debian unstable (sid).
+- **[REMnux](https://docs.remnux.org/discover-the-tools/analyze+documents/email+messages#mail-parser)**:
+  included in the REMnux malware analysis toolkit for analyzing email messages.
+
+## Integrations
+
+- **[IBM Security QRadar SOAR — Parse Utilities](https://github.com/ibmresilient/resilient-community-apps/blob/main/fn_parse_utilities/README.md)**:
+  uses mail-parser to extract headers, body parts, and attachments from `.eml`
+  and `.msg` files for incident response workflows.
+
+## Research use
+
+mail-parser is also used in academic research:
+
+- **[A Large-Scale Empirical Study of Modern Phishing Email Content](https://arxiv.org/abs/2609.30683)**
+  — Jaehwan Park et al., arXiv preprint, September 2026.
+  The study analyzes 2.9 million phishing emails and uses mail-parser to extract
+  PDF, image, and calendar invitation attachments from their MIME structure
+  (Section III-A, reference 34).
 
 # Description
 
@@ -324,6 +347,64 @@ The complete original header values remain available through `from_raw`,
 `to_raw`, etc.; literal headers colliding with computed metadata remain in
 `headers` and `message.get_all(...)`.
 
+Display names decode RFC 2047 encoded words only where the header grammar
+permits them. For example, an unquoted `=?utf-8?Q?Alice?=` becomes `Alice`,
+whereas the same text inside a quoted display name remains literal. Decoded
+punctuation cannot introduce another mailbox, and decoding happens only once.
+Recoverable encoded words longer than 75 characters remain readable and
+produce an `overlong-encoded-word` address diagnostic.
+
+### Trace recovery
+
+`Received` clause boundaries and the timestamp separator respect comments,
+quoted strings, domain literals and angle addresses. Ambiguous delimiters keep
+the original field in `received[*]["raw"]` and produce
+`received_header_defects` entries with `reason`, `raw`, `recovered`, `header`
+and zero-based `occurrence`. These diagnostics set `has_defects` and the
+`ReceivedHeaderDefect` category and appear in full and partial JSON output.
+An ambiguous trusted hop stops sender-IP attribution; older headers cannot
+supply a replacement IP. Balanced, unrecognized obsolete trace syntax remains
+raw without being declared invalid solely because it is obsolete.
+
+### Date diagnostics
+
+Impossible calendar dates, clock components and numeric timezone minutes no
+longer normalize silently into different timestamps. They return `None` and
+set `has_defects`. A mismatched weekday retains the numerical calendar date
+and records the inconsistency. `date_header_defects` covers every `Date`,
+`Resent-Date` and parsed `Received` timestamp, with `reason`, `raw`,
+`recovered`, selected ISO `value` (or `None`), `header` and zero-based
+`occurrence`. It appears in full/partial JSON and uses `DateHeaderDefect`.
+
+Valid obsolete comments, short years and alphabetic zones remain supported.
+Leap seconds map to the next representable Python datetime; the raw field
+retains `:60`. The historical UTC representation for `-0000` and unknown
+alphabetic zones does not establish the sender's local timezone. Dates outside
+Python's datetime range carry `date-out-of-range` diagnostics.
+
+### Inline text and charset recovery
+
+A named `text/plain` or `text/html` part with explicit
+`Content-Disposition: inline` appears in both the corresponding text collection
+and `attachments`. The attachment retains its filename, metadata and payload
+bytes. This additional body view is excluded for binary content, explicit
+attachments and named inline parts inside an attached MIME container. HTML is
+returned as text; parsing does not render it or fetch referenced resources.
+
+Body decoding tries the declared charset against the original bytes. When that
+fails, UTF-8 recovery keeps mislabeled readable text available and records a
+`CharsetDecodeDefect`, including the part index, charset and recovery outcome.
+If UTF-8 also fails, the text view uses replacement characters and reports that
+loss; the original bytes remain available in the MIME part and any attachment.
+Recovery sets `has_defects` even when the recovered text is readable.
+
+Raw UTF-8 filenames and MIME metadata remain readable without changing the
+original headers or payload. Invalid metadata bytes produce `MimeHeaderDefect`
+with escaped raw evidence, the header occurrence and MIME part index. A charset
+decoder that produces a non-Unicode scalar also triggers recovery: names retain
+their literal encoded form and text retains a safe byte-derived view, keeping
+JSON and CLI output encodable as UTF-8.
+
 ## Defects and Their Critical Role in Email Security
 
 Email structural defects are not merely technical curiosities—they represent **potential security
@@ -419,6 +500,17 @@ mail = mailparser.parse_from_file_obj(fp)          # Parse from file object
 mail = mailparser.parse_from_string(raw_mail)      # Parse from string
 ```
 
+File paths are read as bytes, preserving undecodable octets and original MIME
+line endings just like `parse_from_bytes()`. This can expose defects that text
+decoding previously hid. Declared charsets now also apply to file input: a
+message mislabeled with a charset that successfully decodes its bytes can
+display differently from the previous unconditional UTF-8 file read. UTF-8
+recovery applies when the declared decode fails, not by guessing which valid
+interpretation the sender intended. For byte-level forensic evidence, inspect the parsed
+`message` and its `raw_items()`; undecodable header octets use Python's
+surrogateescape representation. The string API accepts text already decoded
+by its caller.
+
 ## Accessing Parsed Components
 
 Once parsed, access all email components through intuitive properties:

src/mailparser/addresses.py

@@ -10,8 +10,10 @@
 
 import re
 from dataclasses import dataclass, field
+from email.header import decode_header
 
 _FOLD = re.compile(r"\r?\n[ \t]+")
+_ENCODED_WORD = re.compile(r"=\?([^?\s]+)\?([bBqQ])\?([^?\s]*)\?=")
 # Anchored grammar checks: atom, quote and delimiter character sets do not
 # overlap. In particular, do not search for bare addresses in arbitrary text
 # or hand long malformed word sequences to headerregistry's quadratic parser.
@@ -234,25 +236,130 @@ def _name(value: str) -> str:
     return _escape_surrogates("".join(result).strip(" \t\r\n"))
 
 
+def _decoded_word(
+    value: str, start: int, comment: bool, problems: set[str]
+) -> tuple[str, int] | None:
+    """Recover one lexical word, reporting RFC 2047's length violation.
+
+    Each regex component stops at whitespace or a question mark. Attempts
+    only start at word boundaries, so even a failed match scans its token
+    a constant number of times, rather than rescanning the whole suffix.
+    """
+    match = _ENCODED_WORD.match(value, start)
+    if not match:
+        return None
+    end = match.end()
+    if end < len(value) and value[end] not in (" \t)" if comment else " \t"):
+        return None
+    if end - start > 75:
+        problems.add("overlong-encoded-word")
+    try:
+        pieces = []
+        for data, charset in decode_header(match.group()):
+            if isinstance(data, bytes):
+                encoding = (charset or "ascii").split("*", 1)[0]
+                if encoding == "unknown-8bit":
+                    encoding = "utf-8"
+                data = data.decode(encoding)
+            if _SURROGATE_RE.search(data):
+                problems.add("invalid-encoded-word")
+                return None
+            # Never inject control characters into the structural parser.
+            if any(ord(ch) < 32 and ch != "\t" for ch in data):
+                return None
+            pieces.append(data)
+    except (LookupError, UnicodeError, ValueError):
+        return None
+    return "".join(pieces), end
+
+
+def _display_source(value: str, phrase: bool, problems: set[str]) -> str:
+    """Decode lexical display words into safely quoted parser input.
+
+    Structural mailbox checks use the original input. RFC 2047 words may
+    be decoded in a phrase or comment, never inside quotes or addr-specs.
+    Escape the rendered text before the conventional-name parser sees it,
+    so decoded commas, angles and comments cannot become delimiters.
+    """
+    value = _FOLD.sub(" ", value)
+    output = []
+    comment = 0
+    quoted = escaped = False
+    i = 0
+    while i < len(value):
+        ch = value[i]
+        if escaped:
+            escaped = False
+        elif ch == "\\" and (quoted or comment):
+            escaped = True
+        elif comment and ch == ")":
+            comment -= 1
+        elif not quoted and ch == "(":
+            comment += 1
+        elif not comment and ch == '"':
+            quoted = not quoted
+        elif not quoted and not comment and ch == "<":
+            phrase = False
+        elif (
+            not quoted
+            and (comment or phrase)
+            and ch == "="
+            and (i == 0 or value[i - 1] in (" \t(" if comment else " \t"))
+        ):
+            word = _decoded_word(value, i, bool(comment), problems)
+            if word:
+                pieces = [word[0]]
+                end = word[1]
+                # Adjacent encoded words discard intervening FWS. Join
+                # once, rather than repeatedly copying an expanding name.
+                while end < len(value):
+                    next_start = end
+                    while next_start < len(value) and value[next_start] in " \t":
+                        next_start += 1
+                    if next_start == end:
+                        break
+                    word = _decoded_word(value, next_start, bool(comment), problems)
+                    if not word:
+                        break
+                    pieces.append(word[0])
+                    end = word[1]
+                text = "".join(pieces).replace("\\", "\\\\")
+                if comment:
+                    text = text.replace("(", "\\(").replace(")", "\\)")
+                else:
+                    text = '"' + text.replace('"', '\\"') + '"'
+                output.append(text)
+                i = end
+                continue
+        output.append(ch)
+        i += 1
+    return "".join(output)
+
+
 def _escape_surrogates(value: str) -> str:
     """Keep invalid code points visible without emitting invalid UTF-8 JSON."""
     return value.encode("utf-8", "backslashreplace").decode("utf-8")
 
 
-def parse_address_header(raw: str, strict_parser):
+def parse_address_header(raw: str, strict_parser, *, decode_names=False):
     """Return mailbox tuples and evidence for recovery or ambiguity.
 
     ``strict_parser`` is the runtime-compatible email.utils adapter. Each
     list member is parsed independently so a malformed member cannot change
-    the interpretation of a neighbouring, valid quoted name.
+    the interpretation of a neighbouring, valid quoted name. Set
+    ``decode_names`` to render display words after structural validation.
     """
     results = []
     diagnostics = []
     for item in _scan(raw):
         candidates = []
+        display_problems: set[str] = set()
         # Derive the prefix once: repeating it for every angle-addr would
         # copy quadratically much text on a long ambiguous item.
-        name = _name(item.text[: item.angles[0][0]]) if item.angles else ""
+        prefix = item.text[: item.angles[0][0]] if item.angles else ""
+        name = _name(
+            _display_source(prefix, True, display_problems) if decode_names else prefix
+        )
         for index, (left, right) in enumerate(item.angles):
             address = _mailbox(item.text[left + 1 : right])
             if address:
@@ -288,13 +395,17 @@ def parse_address_header(raw: str, strict_parser):
             # Deep comments are stripped by the iterative scanner before
             # reaching the stdlib's recursive comment parser.
             source = item.raw if item.comment_depth < 16 else item.text
+            if decode_names:
+                source = _display_source(source, bool(item.angles), display_problems)
             parsed = strict_parser([source])
             if [addr for _, addr in parsed if addr] == [a for _, a in selected]:
                 selected = [(name, addr) for name, addr in parsed if addr]
             else:
                 # Parsing can reject legal CFWS too. This records recovery,
                 # not a claim that every rejection proves RFC noncompliance.
                 reason = reason or "structural-recovery"
+        if display_problems:
+            reason = ",".join(filter(None, [reason, *sorted(display_problems)]))
         results.extend(selected)
         if reason:
             diagnostics.append(