fix(tools): patch.sh 2b-ter greps text its patch adds; a gate holds every guard to its patch

.github/workflows/build.yml

@@ -186,6 +186,20 @@ jobs:
           python3 tools/test_net_boundary.py
           python3 tools/check_net_boundary.py
 
+      # patch.sh runs again and again: after every override that merges, and in the
+      # firmware jobs on a warm ~/.platformio cache that already carries them (see
+      # "Apply framework overrides" below). Each step skips itself when a grep finds
+      # the marker its own patch adds. A marker the patch does not add costs nothing
+      # on the first run and ends the second: `patch` meets the applied hunk, asks
+      # "Assume -R?", rejects it, and `set -e` stops the script before the steps after
+      # it and before the cache invalidation. Step 2b-ter (#274) did exactly that
+      # from 2026-10-15. The test breaks each guard four ways and requires exactly one
+      # finding; the gate reads patch.sh against patches/.
+      - name: Framework overrides (every patch.sh guard matches its patch)
+        run: |
+          python3 tools/test_patch_sentinels.py
+          python3 tools/check_patch_sentinels.py
+
       # The configurator's rules (tools/features.toml [[rules]], each measured
       # by compiling the forbidden combination). They are applied twice: by
       # docs/configurador/rules.js, so the page can lock an option, and by

tools/README.md

@@ -1,6 +1,6 @@
 # tools/
 
-169 scripts. This file exists because until 2026-09-08 there was no way
+171 scripts. This file exists because until 2026-09-08 there was no way
 to tell a live bench tool from a leftover, and one of them —
 `compressor.py` — had been superseded for three months while still looking
 usable: it regenerated `WebUI_GZ.h` into the repository root, where nothing
@@ -16,7 +16,7 @@ wrong, the description is the bug.
 
 ---
 
-## Called by CI (44)
+## Called by CI (46)
 
 Invoked from `.github/workflows/build.yml` — or, for the release manifest, from
 `release-ota.yml` when a tag is pushed. Breaking one of these fails a pull request
@@ -25,6 +25,8 @@ or a release.
 | script | what it does | last touched |
 |---|---|---|
 | `arduino_pico_overrides/patch.sh` | patch.sh — aplica overrides SIMUT no framework arduino-pico do PlatformIO. | 2026-08-20 |
+| `check_patch_sentinels.py` | Every step of `arduino_pico_overrides/patch.sh` skips itself on the second run with a `grep -q` for a marker, and the marker must be a line its own patch ADDS to the file the guard greps (a `+` line of that file's hunks: not context, not a removed line, not another file of the patch). Four rules: a `patch` command outside such a guard (`unguarded`; every `patch` command is read, so a new shape is a finding, not a pass), a sentinel `bash` or `grep` would read as syntax or a path from a variable it cannot resolve (`unreadable`), a patch file that is not there (`no-patch`), and a sentinel the patch does not add (`not-added`). Born on 2026-10-16, after step 2b-ter (#274) grepped "drop a non-TLS client" for a patch that adds "tell a non-TLS client from a TLS one by its FIRST BYTE": the first run applied it, the second met the applied patch, `patch` asked "Assume -R?" and rejected the hunk, and `set -e` ended the script there, before the steps after it and the cache invalidation. Exit 0 clean, 1 findings, 2 a file it could not read or a script it read nothing from. | 2026-10-16 |
+| `test_patch_sentinels.py` | The gate bites. The reader of unified diffs (only `+` lines; an added `++ x` is not a header; `\ No newline` is neither side); synthetic trees for the four rules, for the variables `patch.sh` builds its paths from (assigned above the guard, and assigned again) and for the command's exit codes; and the real tree: it passes, the gate reads all of its `patch -p1` commands, and each guard broken four ways (sentinel changed, marker in the patch changed, guard taken away, patch file gone) is exactly one finding, at that step. One more is 2b-ter with the sentinel it had on 2026-10-15: one finding, at its guard, naming its patch. Run on the unfixed `patch.sh` of `2d6650b`, the gate reports that one and no other. The gate only reads (sha256 of `patch.sh` and every patch, before and after). Job `gates`. Mutation-tested on 16/10: 22 of 22 one-line changes to the gate are caught (3 of them by the test crashing on a lookup the removed check was guarding, not by a failed check). | 2026-10-16 |
 | `build_release.sh` | build_release.sh — Generate Arduino IDE-compatible release zips for simut_tft and simut_alpha. | 2026-08-16 |
 | `check_authz.py` | check_authz.py — the authorization matrix, pinned so it cannot silently rot. | 2026-08-19 |
 | `check_license.py` | A licença diz a mesma coisa nas **três** cópias (LICENSE, página `/license` e a string do firmware que a tela Licença do painel desenha, o texto MIT palavra por palavra nas duas telas), nenhum pacote de idioma volta a trazer `@LICENSE`, e os **dois** scripts de release levam o `LICENSE`, o `THIRD_PARTY_NOTICES.md` e a pasta `LICENSES`. Nasceu de dois achados de 21/09: a página web dizia 2025 e o zip do Arduino IDE saía sem `LICENSE`. | 2026-10-02 |

tools/arduino_pico_overrides/README.md

@@ -424,6 +424,15 @@ bash tools/arduino_pico_overrides/restore.sh
 `patch.sh` is idempotent — you can run it as many times as you want. `originals/`
 is preserved after the first patch.
 
+What makes it so: each step greps the framework file for a marker its own patch
+adds (the `// SIMUT override — ...` line at the top of the first hunk) and skips the
+`patch` when it finds one. A marker the patch does not add costs nothing on the first
+run and ends the second: `patch` meets the applied hunk, asks `Assume -R?` (a `y`
+reverses the override), rejects it to a `.rej` file and exits 1, and under `set -e` the
+steps after it and the cache invalidation do not run. `tools/check_patch_sentinels.py`
+holds every guard to its patch in CI (job `gates`); when you add a step, take the
+sentinel from the `+` lines of your patch.
+
 ## HW Validation (2026-05-10) — historical, describes the v1.0.0 slim build
 
 > These numbers were taken when `PBUF_POOL_SIZE` was 12. It is 24 today, so the

tools/arduino_pico_overrides/patch.sh

@@ -96,6 +96,14 @@ fi
 echo "[patch] aplicando patched lwipopts.h"
 cp -v "$OVR/patched_headers/lwipopts.h" "$FW/include/"
 
+# Do 2b em diante, cada passo é guardado por um grep numa MARCA que o PRÓPRIO
+# patch acrescenta ao arquivo: uma linha "SIMUT override — ..." entre as linhas
+# `+` dele. Marca que o patch não acrescenta não quebra a primeira execução, só
+# a segunda: o grep não acha, o `patch` reencontra o patch já aplicado (pergunta
+# "Assume -R?" e rejeita) e, com `set -e`, o script morre ali, sem chegar nos
+# passos seguintes nem na invalidação do cache. Foi o caso do 2b-ter (#274),
+# achado em 16/10/2026. tools/check_patch_sentinels.py (job `gates`) confere cada guarda.
+
 # 2b. Patch cirúrgico no handshake TLS (WiFiClientSecureBearSSL.cpp)
 #
 #   O upstream só tem prazo dentro de _run_until(), que reinicia o próprio
@@ -135,7 +143,7 @@ fi
 WSSECURE="$FW/libraries/WiFi/src/WiFiServerSecureBearSSL.cpp"
 WSSECURE_PATCH="$OVR/patches/wifiserversecure_drop_plaintext.patch"
 save_original "$WSSECURE" "WiFiServerSecureBearSSL.cpp"
-if grep -q "SIMUT override — drop a non-TLS client" "$WSSECURE"; then
+if grep -q "SIMUT override — tell a non-TLS client from a TLS one by its FIRST BYTE" "$WSSECURE"; then
     echo "[patch] recusa não-TLS na porta TLS já aplicada — nada a fazer"
 else
     echo "[patch] aplicando recusa não-TLS na porta TLS (#274)"

tools/check_patch_sentinels.py

@@ -0,0 +1,235 @@
+#!/usr/bin/env python3
+"""check_patch_sentinels.py — every step of tools/arduino_pico_overrides/patch.sh can tell its patch is applied.
+
+patch.sh is run again and again: after every override that merges, after a framework reinstall, and by CI
+on a warm ~/.platformio cache (build.yml: "the script is idempotent, so a warm cache that already carries
+them is fine"). Each step is therefore
+
+    if grep -q "<sentinel>" "<file>"; then
+        echo "... nada a fazer"
+    else
+        patch -p1 -d "$FW" < "<patch>"
+    fi
+
+and the sentinel must be text the PATCH ITSELF adds to <file>: in practice the `// SIMUT override — ...`
+line at the top of its first hunk. Text that is not there costs nothing on the first run, when the file is
+still the factory one and `patch` applies. On the second run grep finds nothing, `patch` meets a patch that
+is already applied and asks "Reversed (or previously applied) patch detected! Assume -R? [n]" (a "y" there
+reverses the override), then rejects the hunk to a .rej file and exits 1, and under `set -e` the script
+dies on that line: the steps after it do not run, nor does the cache invalidation of step 3.
+
+That happened to step 2b-ter (#274, merged 2026-10-15): it grepped "drop a non-TLS client" and its patch
+adds "tell a non-TLS client from a TLS one by its FIRST BYTE". Nothing failed until the first re-run, on
+2026-10-16. The gate reads patch.sh and the patches and holds four rules:
+
+  unguarded   a `patch` command that is not the else-branch of an `if grep -q "<sentinel>" "<file>"; then`
+              and written `patch -p1 -d "$FW" < "<patch>"`. Every `patch` command is read, so a step in
+              another shape is a finding until this gate learns the shape; it is never skipped.
+  unreadable  a sentinel that bash or grep would not read as the characters written ($ ` \\ inside double
+              quotes; [ ] * ^ make `grep -q` a pattern, not a substring), or a path built from a variable
+              this gate cannot resolve (it knows $FW, $OVR and the NAME="..." lines above the guard).
+  no-patch    the patch file the step applies does not exist.
+  not-added   the sentinel is not in a line the patch ADDS (a `+` line: not context, not a removed line)
+              to the file the guard greps (the `+++` path of its hunk, minus the -p1 prefix). The file
+              matters: 2f greps HTTPServer.h, 2m and 2n grep HTTPServer.cpp, in patches that touch both.
+
+What is not `patch` is not read: the backup (guarded by `[ ! -f ... ]`) and the copy of lwipopts.h (a
+plain `cp`, which does the same thing every time) are idempotent by construction.
+
+    python3 tools/check_patch_sentinels.py                    # the gate: `path:LINE: rule: text`, exit 1 on any finding
+    python3 tools/check_patch_sentinels.py --patch-sh FILE    # another script (the test; the old 2b-ter, as a mutation)
+    python3 tools/check_patch_sentinels.py --ovr DIR          # another overrides directory: where $OVR points
+
+Exit 0 clean, 1 findings, 2 a file it could not read or a script in which it found neither a guarded step nor
+any `patch` command (a gate that reads nothing must not pass).
+"""
+import argparse
+import os
+import re
+import sys
+from typing import NamedTuple, Optional
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+ROOT = os.path.dirname(HERE)
+DEFAULT_OVR = os.path.join(HERE, "arduino_pico_overrides")
+FW = "<FW>"      # what $FW stands for here: the root of the framework package, on whatever machine this runs
+
+ASSIGN = re.compile(r'^([A-Za-z_][A-Za-z0-9_]*)="([^"]*)"\s*(?:#.*)?$')
+GUARD = re.compile(r'^\s*if grep -q "([^"]*)" "([^"]*)"; then\s*$')
+PATCH_CMD = re.compile(r"\bpatch\s+-")
+PATCH_OK = re.compile(r'^\s*patch -p1 -d "\$FW" < "([^"]+)"\s*$')
+VAR = re.compile(r"\$(?:([A-Za-z_][A-Za-z0-9_]*)|\{([A-Za-z_][A-Za-z0-9_]*)\})")
+HUNK = re.compile(r"^@@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? @@")
+SYNTAX = re.compile(r"[$`\\\[\]*^]")
+
+RULES = ("unguarded", "unreadable", "no-patch", "not-added")
+HINTS = {
+    "unguarded": 'write the step as `if grep -q "<sentinel>" "<file>"; then ... else ... patch -p1 -d "$FW" '
+                 '< "<patch>"; fi`: a patch with no guard is applied again on the second run',
+    "unreadable": "the sentinel is a plain substring (none of $ ` \\ [ ] * ^) and the paths use only $FW, $OVR and "
+                  "NAME=\"...\" lines above the guard",
+    "no-patch": "the step applies a file that is not in patches/",
+    "not-added": "grep the `// SIMUT override — ...` line the patch ADDS to that file: a sentinel the patch does "
+                 "not add finds nothing on the second run, and `patch` stops the script at the applied hunk",
+}
+
+
+class Finding(NamedTuple):
+    line: int
+    rule: str
+    text: str
+
+
+class Step(NamedTuple):
+    line: int                 # the `if grep` line: where a wrong sentinel is fixed
+    sentinel: str
+    file: Optional[str]       # the file the guard greps, $FW/... resolved; None if it could not be
+    patch: Optional[str]      # the patch file, resolved; None if it could not be
+    patch_line: int
+
+
+def read(path: str) -> str:
+    with open(path, encoding="utf-8") as f:
+        return f.read()
+
+
+def expand(s: str, env: dict) -> Optional[str]:
+    """s with $VAR and ${VAR} from env; None when a variable is unknown or a `$` is left that it cannot read."""
+    try:
+        out = VAR.sub(lambda m: env[m.group(1) or m.group(2)], s)
+    except KeyError:
+        return None
+    return None if "$" in out else out
+
+
+def read_steps(text: str, ovr: str):
+    """The guarded `patch` steps of a patch.sh, and a finding for every `patch` command outside one."""
+    env = {"FW": FW, "OVR": ovr}
+    steps, findings = [], []
+    guard, in_else = None, False        # (line, sentinel, file) from `if grep -q ...; then` to `fi`
+    for n, raw in enumerate(text.splitlines(), 1):
+        line = raw.strip()
+        if not line or line.startswith("#"):
+            continue
+        m = ASSIGN.match(raw)
+        if m:                           # NAME="..." at column 0; the value at the point of use is the one read
+            if m.group(1) not in ("FW", "OVR"):         # those two are what the gate says they are, whatever the script does
+                value = expand(m.group(2), env)
+                if value is not None:
+                    env[m.group(1)] = value
+            continue
+        m = GUARD.match(raw)
+        if m:
+            guard, in_else = (n, m.group(1), expand(m.group(2), env)), False
+            continue
+        if guard and line == "else":
+            in_else = True
+            continue
+        if guard and line == "fi":
+            guard, in_else = None, False
+            continue
+        if PATCH_CMD.search(line):
+            ok = PATCH_OK.match(raw)
+            if guard and in_else and ok:
+                steps.append(Step(guard[0], guard[1], guard[2], expand(ok.group(1), env), n))
+            else:
+                findings.append(Finding(n, "unguarded", line))
+    return steps, findings
+
+
+def added_lines(text: str) -> dict:
+    """{path: [the lines the diff adds]} for a unified diff; path as `patch -p1` reads it (first component dropped).
+
+    Hunks are read by their line counts, not by prefix alone, so an added line that looks like a file header
+    (`++ x`) is still an added line, and a removed one (`-- x`) is not a header."""
+    out, cur = {}, None
+    lines = text.split("\n")
+    i = 0
+    while i < len(lines):
+        if lines[i].startswith("--- ") and i + 1 < len(lines) and lines[i + 1].startswith("+++ "):
+            cur = lines[i + 1][4:].split("\t")[0].split("/", 1)[-1]
+            out.setdefault(cur, [])
+            i += 2
+            continue
+        m = HUNK.match(lines[i])
+        if m and cur is not None:
+            old = int(m.group(1)) if m.group(1) is not None else 1
+            new = int(m.group(2)) if m.group(2) is not None else 1
+            i += 1
+            while (old > 0 or new > 0) and i < len(lines):
+                h = lines[i]
+                if h.startswith("+"):
+                    out[cur].append(h[1:])
+                    new -= 1
+                elif h.startswith("-"):
+                    old -= 1
+                elif not h.startswith("\\"):        # context (a blank one may have lost its space); "\ No newline" is neither
+                    old -= 1
+                    new -= 1
+                i += 1
+            continue
+        i += 1
+    return out
+
+
+def check(sh_path: str, ovr: str):
+    """(findings, steps) for one patch.sh whose $OVR is the directory ovr."""
+    steps, findings = read_steps(read(sh_path), ovr)
+    adds = {}
+    for st in steps:
+        s = st.sentinel
+        if SYNTAX.search(s):
+            findings.append(Finding(st.line, "unreadable", f"the sentinel {s!r} has a character that bash or grep "
+                                    f"reads as syntax, not as itself"))
+            continue
+        if st.file is None or not st.file.startswith(FW + "/"):
+            findings.append(Finding(st.line, "unreadable", f"cannot tell which file of the framework the guard greps"))
+            continue
+        if st.patch is None:
+            findings.append(Finding(st.patch_line, "unreadable", "cannot tell which patch file the step applies"))
+            continue
+        name = os.path.relpath(st.patch, ovr)
+        if not os.path.isfile(st.patch):
+            findings.append(Finding(st.patch_line, "no-patch", f"{name} does not exist"))
+            continue
+        if st.patch not in adds:
+            adds[st.patch] = added_lines(read(st.patch))
+        rel = st.file[len(FW) + 1:]
+        if rel not in adds[st.patch]:
+            touched = ", ".join(sorted(adds[st.patch])) or "no file"
+            findings.append(Finding(st.line, "not-added", f"{name} does not touch {rel} (it touches {touched})"))
+        elif not any(s in line for line in adds[st.patch][rel]):
+            marks = [line.strip()[:90] for line in adds[st.patch][rel] if "SIMUT override" in line][:2]
+            findings.append(Finding(st.line, "not-added", f"{s!r} is in no line {name} adds to {rel}; "
+                                    + (f"it adds {marks}" if marks else "it adds no `SIMUT override` line there")))
+    return sorted(findings, key=lambda f: (f.line, RULES.index(f.rule))), steps
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(description="Every guard of patch.sh against its patch: see the module docstring.")
+    ap.add_argument("--ovr", default=DEFAULT_OVR, help="the overrides directory, where $OVR points (default: this repository's)")
+    ap.add_argument("--patch-sh", default=None, help="the script to read (default: <ovr>/patch.sh)")
+    args = ap.parse_args(argv)
+    ovr = os.path.abspath(args.ovr)
+    sh = os.path.abspath(args.patch_sh or os.path.join(ovr, "patch.sh"))
+    shown = os.path.relpath(sh, ROOT) if sh.startswith(ROOT + os.sep) else sh
+    try:
+        findings, steps = check(sh, ovr)
+    except (OSError, UnicodeDecodeError) as e:
+        print(f"check_patch_sentinels: {e}", file=sys.stderr)
+        return 2
+    if not steps and not findings:
+        print(f"check_patch_sentinels: no `if grep -q ...; then ... patch` step found in {shown}: "
+              f"the gate does not understand the file", file=sys.stderr)
+        return 2
+    for f in findings:
+        print(f"{shown}:{f.line}: {f.rule}: {f.text}")
+    for rule in RULES:
+        if any(f.rule == rule for f in findings):
+            print(f"check_patch_sentinels: {rule}: {HINTS[rule]}", file=sys.stderr)
+    print(f"check_patch_sentinels: {len(findings)} findings in {len(steps)} guarded steps")
+    return 1 if findings else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())