diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index db31de1e..c159c568 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -186,6 +186,20 @@ jobs: python3 tools/test_net_boundary.py python3 tools/check_net_boundary.py + # patch.sh runs again and again: after every override that merges, and in the + # firmware jobs on a warm ~/.platformio cache that already carries them (see + # "Apply framework overrides" below). Each step skips itself when a grep finds + # the marker its own patch adds. A marker the patch does not add costs nothing + # on the first run and ends the second: `patch` meets the applied hunk, asks + # "Assume -R?", rejects it, and `set -e` stops the script before the steps after + # it and before the cache invalidation. Step 2b-ter (#274) did exactly that + # from 2026-10-15. The test breaks each guard four ways and requires exactly one + # finding; the gate reads patch.sh against patches/. + - name: Framework overrides (every patch.sh guard matches its patch) + run: | + python3 tools/test_patch_sentinels.py + python3 tools/check_patch_sentinels.py + # The configurator's rules (tools/features.toml [[rules]], each measured # by compiling the forbidden combination). They are applied twice: by # docs/configurador/rules.js, so the page can lock an option, and by diff --git a/tools/README.md b/tools/README.md index 00304baa..e88ec982 100644 --- a/tools/README.md +++ b/tools/README.md @@ -1,6 +1,6 @@ # tools/ -169 scripts. This file exists because until 2026-09-08 there was no way +171 scripts. This file exists because until 2026-09-08 there was no way to tell a live bench tool from a leftover, and one of them — `compressor.py` — had been superseded for three months while still looking usable: it regenerated `WebUI_GZ.h` into the repository root, where nothing @@ -16,7 +16,7 @@ wrong, the description is the bug. --- -## Called by CI (44) +## Called by CI (46) Invoked from `.github/workflows/build.yml` — or, for the release manifest, from `release-ota.yml` when a tag is pushed. Breaking one of these fails a pull request @@ -25,6 +25,8 @@ or a release. | script | what it does | last touched | |---|---|---| | `arduino_pico_overrides/patch.sh` | patch.sh — aplica overrides SIMUT no framework arduino-pico do PlatformIO. | 2026-08-20 | +| `check_patch_sentinels.py` | Every step of `arduino_pico_overrides/patch.sh` skips itself on the second run with a `grep -q` for a marker, and the marker must be a line its own patch ADDS to the file the guard greps (a `+` line of that file's hunks: not context, not a removed line, not another file of the patch). Four rules: a `patch` command outside such a guard (`unguarded`; every `patch` command is read, so a new shape is a finding, not a pass), a sentinel `bash` or `grep` would read as syntax or a path from a variable it cannot resolve (`unreadable`), a patch file that is not there (`no-patch`), and a sentinel the patch does not add (`not-added`). Born on 2026-10-16, after step 2b-ter (#274) grepped "drop a non-TLS client" for a patch that adds "tell a non-TLS client from a TLS one by its FIRST BYTE": the first run applied it, the second met the applied patch, `patch` asked "Assume -R?" and rejected the hunk, and `set -e` ended the script there, before the steps after it and the cache invalidation. Exit 0 clean, 1 findings, 2 a file it could not read or a script it read nothing from. | 2026-10-16 | +| `test_patch_sentinels.py` | The gate bites. The reader of unified diffs (only `+` lines; an added `++ x` is not a header; `\ No newline` is neither side); synthetic trees for the four rules, for the variables `patch.sh` builds its paths from (assigned above the guard, and assigned again) and for the command's exit codes; and the real tree: it passes, the gate reads all of its `patch -p1` commands, and each guard broken four ways (sentinel changed, marker in the patch changed, guard taken away, patch file gone) is exactly one finding, at that step. One more is 2b-ter with the sentinel it had on 2026-10-15: one finding, at its guard, naming its patch. Run on the unfixed `patch.sh` of `2d6650b`, the gate reports that one and no other. The gate only reads (sha256 of `patch.sh` and every patch, before and after). Job `gates`. Mutation-tested on 16/10: 22 of 22 one-line changes to the gate are caught (3 of them by the test crashing on a lookup the removed check was guarding, not by a failed check). | 2026-10-16 | | `build_release.sh` | build_release.sh — Generate Arduino IDE-compatible release zips for simut_tft and simut_alpha. | 2026-08-16 | | `check_authz.py` | check_authz.py — the authorization matrix, pinned so it cannot silently rot. | 2026-08-19 | | `check_license.py` | A licença diz a mesma coisa nas **três** cópias (LICENSE, página `/license` e a string do firmware que a tela Licença do painel desenha, o texto MIT palavra por palavra nas duas telas), nenhum pacote de idioma volta a trazer `@LICENSE`, e os **dois** scripts de release levam o `LICENSE`, o `THIRD_PARTY_NOTICES.md` e a pasta `LICENSES`. Nasceu de dois achados de 21/09: a página web dizia 2025 e o zip do Arduino IDE saía sem `LICENSE`. | 2026-10-02 | diff --git a/tools/arduino_pico_overrides/README.md b/tools/arduino_pico_overrides/README.md index 20228a64..b252da19 100644 --- a/tools/arduino_pico_overrides/README.md +++ b/tools/arduino_pico_overrides/README.md @@ -424,6 +424,15 @@ bash tools/arduino_pico_overrides/restore.sh `patch.sh` is idempotent — you can run it as many times as you want. `originals/` is preserved after the first patch. +What makes it so: each step greps the framework file for a marker its own patch +adds (the `// SIMUT override — ...` line at the top of the first hunk) and skips the +`patch` when it finds one. A marker the patch does not add costs nothing on the first +run and ends the second: `patch` meets the applied hunk, asks `Assume -R?` (a `y` +reverses the override), rejects it to a `.rej` file and exits 1, and under `set -e` the +steps after it and the cache invalidation do not run. `tools/check_patch_sentinels.py` +holds every guard to its patch in CI (job `gates`); when you add a step, take the +sentinel from the `+` lines of your patch. + ## HW Validation (2026-05-10) — historical, describes the v1.0.0 slim build > These numbers were taken when `PBUF_POOL_SIZE` was 12. It is 24 today, so the diff --git a/tools/arduino_pico_overrides/patch.sh b/tools/arduino_pico_overrides/patch.sh index 27f68cf3..f17970be 100755 --- a/tools/arduino_pico_overrides/patch.sh +++ b/tools/arduino_pico_overrides/patch.sh @@ -96,6 +96,14 @@ fi echo "[patch] aplicando patched lwipopts.h" cp -v "$OVR/patched_headers/lwipopts.h" "$FW/include/" +# Do 2b em diante, cada passo é guardado por um grep numa MARCA que o PRÓPRIO +# patch acrescenta ao arquivo: uma linha "SIMUT override — ..." entre as linhas +# `+` dele. Marca que o patch não acrescenta não quebra a primeira execução, só +# a segunda: o grep não acha, o `patch` reencontra o patch já aplicado (pergunta +# "Assume -R?" e rejeita) e, com `set -e`, o script morre ali, sem chegar nos +# passos seguintes nem na invalidação do cache. Foi o caso do 2b-ter (#274), +# achado em 16/10/2026. tools/check_patch_sentinels.py (job `gates`) confere cada guarda. + # 2b. Patch cirúrgico no handshake TLS (WiFiClientSecureBearSSL.cpp) # # O upstream só tem prazo dentro de _run_until(), que reinicia o próprio @@ -135,7 +143,7 @@ fi WSSECURE="$FW/libraries/WiFi/src/WiFiServerSecureBearSSL.cpp" WSSECURE_PATCH="$OVR/patches/wifiserversecure_drop_plaintext.patch" save_original "$WSSECURE" "WiFiServerSecureBearSSL.cpp" -if grep -q "SIMUT override — drop a non-TLS client" "$WSSECURE"; then +if grep -q "SIMUT override — tell a non-TLS client from a TLS one by its FIRST BYTE" "$WSSECURE"; then echo "[patch] recusa não-TLS na porta TLS já aplicada — nada a fazer" else echo "[patch] aplicando recusa não-TLS na porta TLS (#274)" diff --git a/tools/check_patch_sentinels.py b/tools/check_patch_sentinels.py new file mode 100644 index 00000000..3007c470 --- /dev/null +++ b/tools/check_patch_sentinels.py @@ -0,0 +1,235 @@ +#!/usr/bin/env python3 +"""check_patch_sentinels.py — every step of tools/arduino_pico_overrides/patch.sh can tell its patch is applied. + +patch.sh is run again and again: after every override that merges, after a framework reinstall, and by CI +on a warm ~/.platformio cache (build.yml: "the script is idempotent, so a warm cache that already carries +them is fine"). Each step is therefore + + if grep -q "" ""; then + echo "... nada a fazer" + else + patch -p1 -d "$FW" < "" + fi + +and the sentinel must be text the PATCH ITSELF adds to : in practice the `// SIMUT override — ...` +line at the top of its first hunk. Text that is not there costs nothing on the first run, when the file is +still the factory one and `patch` applies. On the second run grep finds nothing, `patch` meets a patch that +is already applied and asks "Reversed (or previously applied) patch detected! Assume -R? [n]" (a "y" there +reverses the override), then rejects the hunk to a .rej file and exits 1, and under `set -e` the script +dies on that line: the steps after it do not run, nor does the cache invalidation of step 3. + +That happened to step 2b-ter (#274, merged 2026-10-15): it grepped "drop a non-TLS client" and its patch +adds "tell a non-TLS client from a TLS one by its FIRST BYTE". Nothing failed until the first re-run, on +2026-10-16. The gate reads patch.sh and the patches and holds four rules: + + unguarded a `patch` command that is not the else-branch of an `if grep -q "" ""; then` + and written `patch -p1 -d "$FW" < ""`. Every `patch` command is read, so a step in + another shape is a finding until this gate learns the shape; it is never skipped. + unreadable a sentinel that bash or grep would not read as the characters written ($ ` \\ inside double + quotes; [ ] * ^ make `grep -q` a pattern, not a substring), or a path built from a variable + this gate cannot resolve (it knows $FW, $OVR and the NAME="..." lines above the guard). + no-patch the patch file the step applies does not exist. + not-added the sentinel is not in a line the patch ADDS (a `+` line: not context, not a removed line) + to the file the guard greps (the `+++` path of its hunk, minus the -p1 prefix). The file + matters: 2f greps HTTPServer.h, 2m and 2n grep HTTPServer.cpp, in patches that touch both. + +What is not `patch` is not read: the backup (guarded by `[ ! -f ... ]`) and the copy of lwipopts.h (a +plain `cp`, which does the same thing every time) are idempotent by construction. + + python3 tools/check_patch_sentinels.py # the gate: `path:LINE: rule: text`, exit 1 on any finding + python3 tools/check_patch_sentinels.py --patch-sh FILE # another script (the test; the old 2b-ter, as a mutation) + python3 tools/check_patch_sentinels.py --ovr DIR # another overrides directory: where $OVR points + +Exit 0 clean, 1 findings, 2 a file it could not read or a script in which it found neither a guarded step nor +any `patch` command (a gate that reads nothing must not pass). +""" +import argparse +import os +import re +import sys +from typing import NamedTuple, Optional + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +DEFAULT_OVR = os.path.join(HERE, "arduino_pico_overrides") +FW = "" # what $FW stands for here: the root of the framework package, on whatever machine this runs + +ASSIGN = re.compile(r'^([A-Za-z_][A-Za-z0-9_]*)="([^"]*)"\s*(?:#.*)?$') +GUARD = re.compile(r'^\s*if grep -q "([^"]*)" "([^"]*)"; then\s*$') +PATCH_CMD = re.compile(r"\bpatch\s+-") +PATCH_OK = re.compile(r'^\s*patch -p1 -d "\$FW" < "([^"]+)"\s*$') +VAR = re.compile(r"\$(?:([A-Za-z_][A-Za-z0-9_]*)|\{([A-Za-z_][A-Za-z0-9_]*)\})") +HUNK = re.compile(r"^@@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? @@") +SYNTAX = re.compile(r"[$`\\\[\]*^]") + +RULES = ("unguarded", "unreadable", "no-patch", "not-added") +HINTS = { + "unguarded": 'write the step as `if grep -q "" ""; then ... else ... patch -p1 -d "$FW" ' + '< ""; fi`: a patch with no guard is applied again on the second run', + "unreadable": "the sentinel is a plain substring (none of $ ` \\ [ ] * ^) and the paths use only $FW, $OVR and " + "NAME=\"...\" lines above the guard", + "no-patch": "the step applies a file that is not in patches/", + "not-added": "grep the `// SIMUT override — ...` line the patch ADDS to that file: a sentinel the patch does " + "not add finds nothing on the second run, and `patch` stops the script at the applied hunk", +} + + +class Finding(NamedTuple): + line: int + rule: str + text: str + + +class Step(NamedTuple): + line: int # the `if grep` line: where a wrong sentinel is fixed + sentinel: str + file: Optional[str] # the file the guard greps, $FW/... resolved; None if it could not be + patch: Optional[str] # the patch file, resolved; None if it could not be + patch_line: int + + +def read(path: str) -> str: + with open(path, encoding="utf-8") as f: + return f.read() + + +def expand(s: str, env: dict) -> Optional[str]: + """s with $VAR and ${VAR} from env; None when a variable is unknown or a `$` is left that it cannot read.""" + try: + out = VAR.sub(lambda m: env[m.group(1) or m.group(2)], s) + except KeyError: + return None + return None if "$" in out else out + + +def read_steps(text: str, ovr: str): + """The guarded `patch` steps of a patch.sh, and a finding for every `patch` command outside one.""" + env = {"FW": FW, "OVR": ovr} + steps, findings = [], [] + guard, in_else = None, False # (line, sentinel, file) from `if grep -q ...; then` to `fi` + for n, raw in enumerate(text.splitlines(), 1): + line = raw.strip() + if not line or line.startswith("#"): + continue + m = ASSIGN.match(raw) + if m: # NAME="..." at column 0; the value at the point of use is the one read + if m.group(1) not in ("FW", "OVR"): # those two are what the gate says they are, whatever the script does + value = expand(m.group(2), env) + if value is not None: + env[m.group(1)] = value + continue + m = GUARD.match(raw) + if m: + guard, in_else = (n, m.group(1), expand(m.group(2), env)), False + continue + if guard and line == "else": + in_else = True + continue + if guard and line == "fi": + guard, in_else = None, False + continue + if PATCH_CMD.search(line): + ok = PATCH_OK.match(raw) + if guard and in_else and ok: + steps.append(Step(guard[0], guard[1], guard[2], expand(ok.group(1), env), n)) + else: + findings.append(Finding(n, "unguarded", line)) + return steps, findings + + +def added_lines(text: str) -> dict: + """{path: [the lines the diff adds]} for a unified diff; path as `patch -p1` reads it (first component dropped). + + Hunks are read by their line counts, not by prefix alone, so an added line that looks like a file header + (`++ x`) is still an added line, and a removed one (`-- x`) is not a header.""" + out, cur = {}, None + lines = text.split("\n") + i = 0 + while i < len(lines): + if lines[i].startswith("--- ") and i + 1 < len(lines) and lines[i + 1].startswith("+++ "): + cur = lines[i + 1][4:].split("\t")[0].split("/", 1)[-1] + out.setdefault(cur, []) + i += 2 + continue + m = HUNK.match(lines[i]) + if m and cur is not None: + old = int(m.group(1)) if m.group(1) is not None else 1 + new = int(m.group(2)) if m.group(2) is not None else 1 + i += 1 + while (old > 0 or new > 0) and i < len(lines): + h = lines[i] + if h.startswith("+"): + out[cur].append(h[1:]) + new -= 1 + elif h.startswith("-"): + old -= 1 + elif not h.startswith("\\"): # context (a blank one may have lost its space); "\ No newline" is neither + old -= 1 + new -= 1 + i += 1 + continue + i += 1 + return out + + +def check(sh_path: str, ovr: str): + """(findings, steps) for one patch.sh whose $OVR is the directory ovr.""" + steps, findings = read_steps(read(sh_path), ovr) + adds = {} + for st in steps: + s = st.sentinel + if SYNTAX.search(s): + findings.append(Finding(st.line, "unreadable", f"the sentinel {s!r} has a character that bash or grep " + f"reads as syntax, not as itself")) + continue + if st.file is None or not st.file.startswith(FW + "/"): + findings.append(Finding(st.line, "unreadable", f"cannot tell which file of the framework the guard greps")) + continue + if st.patch is None: + findings.append(Finding(st.patch_line, "unreadable", "cannot tell which patch file the step applies")) + continue + name = os.path.relpath(st.patch, ovr) + if not os.path.isfile(st.patch): + findings.append(Finding(st.patch_line, "no-patch", f"{name} does not exist")) + continue + if st.patch not in adds: + adds[st.patch] = added_lines(read(st.patch)) + rel = st.file[len(FW) + 1:] + if rel not in adds[st.patch]: + touched = ", ".join(sorted(adds[st.patch])) or "no file" + findings.append(Finding(st.line, "not-added", f"{name} does not touch {rel} (it touches {touched})")) + elif not any(s in line for line in adds[st.patch][rel]): + marks = [line.strip()[:90] for line in adds[st.patch][rel] if "SIMUT override" in line][:2] + findings.append(Finding(st.line, "not-added", f"{s!r} is in no line {name} adds to {rel}; " + + (f"it adds {marks}" if marks else "it adds no `SIMUT override` line there"))) + return sorted(findings, key=lambda f: (f.line, RULES.index(f.rule))), steps + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description="Every guard of patch.sh against its patch: see the module docstring.") + ap.add_argument("--ovr", default=DEFAULT_OVR, help="the overrides directory, where $OVR points (default: this repository's)") + ap.add_argument("--patch-sh", default=None, help="the script to read (default: /patch.sh)") + args = ap.parse_args(argv) + ovr = os.path.abspath(args.ovr) + sh = os.path.abspath(args.patch_sh or os.path.join(ovr, "patch.sh")) + shown = os.path.relpath(sh, ROOT) if sh.startswith(ROOT + os.sep) else sh + try: + findings, steps = check(sh, ovr) + except (OSError, UnicodeDecodeError) as e: + print(f"check_patch_sentinels: {e}", file=sys.stderr) + return 2 + if not steps and not findings: + print(f"check_patch_sentinels: no `if grep -q ...; then ... patch` step found in {shown}: " + f"the gate does not understand the file", file=sys.stderr) + return 2 + for f in findings: + print(f"{shown}:{f.line}: {f.rule}: {f.text}") + for rule in RULES: + if any(f.rule == rule for f in findings): + print(f"check_patch_sentinels: {rule}: {HINTS[rule]}", file=sys.stderr) + print(f"check_patch_sentinels: {len(findings)} findings in {len(steps)} guarded steps") + return 1 if findings else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/test_patch_sentinels.py b/tools/test_patch_sentinels.py new file mode 100644 index 00000000..c928a59f --- /dev/null +++ b/tools/test_patch_sentinels.py @@ -0,0 +1,341 @@ +#!/usr/bin/env python3 +"""Tests tools/check_patch_sentinels.py, the gate that keeps every step of patch.sh able to tell its patch is applied. + +Run: python3 tools/test_patch_sentinels.py (CI runs it as a gate, next to the gate itself) + +A gate that finds nothing is the failure this test exists for, so each rule is proved the way a rule is +proved: the violation put into a clean case gives exactly ONE finding, of that rule, on its line, and the +same text where the rule is not broken gives none. + + * The reader of unified diffs: only `+` lines count (not context, not removed lines, not the `+++` header, + and an added line that LOOKS like a header, `++ x`, is still an added line); the path is the `+++` path + minus the -p1 prefix; two files in one patch stay apart; "\\ No newline" is neither side. + * Synthetic trees (a temp directory with a patch.sh and a patches/): the four rules, the variables patch.sh + builds its paths from (assigned above the guard, and assigned AGAIN), and the command's exit codes and + output. + * The real tree: it passes, and the gate reads every `patch -p1` of the script. Then each of its guards is + broken in turn, four ways (its sentinel changed; the marker in its patch changed; the guard taken away; + the patch file gone), and each gives exactly one finding, at that step. One more is 2b-ter as it was on + 2026-10-15, the sentinel that started this: "drop a non-TLS client" against a patch that adds "tell a + non-TLS client from a TLS one by its FIRST BYTE". It passed the first run; on the second, `patch` met a + patch already applied and `set -e` stopped patch.sh there (reproduced on an isolated core, 2026-10-16). + * The gate only reads: the sha256 of patch.sh and of every patch is the same after every run. +""" +import contextlib +import hashlib +import os +import re +import shutil +import subprocess +import sys +import tempfile + +HERE = os.path.dirname(os.path.abspath(__file__)) +GATE = os.path.join(HERE, "check_patch_sentinels.py") +OVR = os.path.join(HERE, "arduino_pico_overrides") +sys.path.insert(0, HERE) +import check_patch_sentinels as cps # noqa: E402 + +FAILS = [] +RAN = [] + + +def check(name, cond, detail=""): + RAN.append(name) + tag = "ok " if cond else "FAIL" + print(f"[{tag}] {name}" + (f" — {detail}" if detail and not cond else "")) + if not cond: + FAILS.append(name) + + +def brief(found): + return "; ".join(f"{f.line}:{f.rule}:{f.text[:70]}" for f in found) or "no finding" + + +def exactly(name, found, rule, line, mention=None): + """`found` is ONE finding, of this rule, on this line (and its text says `mention`, if given).""" + check(name, len(found) == 1 and found[0].rule == rule and found[0].line == line + and (mention is None or mention in found[0].text), brief(found)) + + +# ── synthetic trees ──────────────────────────────────────────────────────────────────────────── + +HEAD = '#!/usr/bin/env bash\nset -e\nFW=""\nOVR="$ROOT/tools/arduino_pico_overrides"\n' # 4 lines +MARK = "SIMUT override — alpha" +ADD = "// " + MARK + ", the marker" + + +def step(guard=MARK, file='"$A"', patch='"$OVR/patches/a.patch"'): + return (f'if grep -q "{guard}" {file}; then\n echo "[patch] a já aplicada — nada a fazer"\nelse\n' + f' echo "[patch] aplicando a"\n patch -p1 -d "$FW" < {patch}\nfi\n') + + +def script(*body, assigns='A="$FW/lib/A.cpp"\n'): + return HEAD + assigns + "".join(body) + + +def diff(path, *added, before="int a;", after="int b;", removed=()): + lines = [" " + before] + ["-" + r for r in removed] + ["+" + a for a in added] + [" " + after] + return (f"--- a/{path}\n+++ b/{path}\n@@ -1,{2 + len(removed)} +1,{2 + len(added)} @@\n" + "\n".join(lines) + "\n") + + +def line_of(text, needle, nth=1): + return [i for i, line in enumerate(text.splitlines(), 1) if needle in line][nth - 1] + + +@contextlib.contextmanager +def tree(sh, patches): + with tempfile.TemporaryDirectory() as d: + os.makedirs(os.path.join(d, "patches")) + files = {"patch.sh": sh, **{f"patches/{k}": v for k, v in patches.items()}} + for rel, body in files.items(): + with open(os.path.join(d, rel), "w", encoding="utf-8") as f: + f.write(body) + yield d + + +def run(sh, patches): + with tree(sh, patches) as d: + return cps.check(os.path.join(d, "patch.sh"), d) + + +def cli(sh, patches, *extra): + with tree(sh, patches) as d: + r = subprocess.run([sys.executable, GATE, "--ovr", d, *extra], capture_output=True, text=True) + return r.returncode, r.stdout, r.stderr, d + + +def diff_cases(): + d = ("--- a/lib/A.cpp\n+++ b/lib/A.cpp\n@@ -1,3 +1,4 @@\n int a;\n-int old;\n+int new;\n+// SIMUT override — added\n int b;\n" + "--- a/lib/B.h\t2026-10-16\n+++ b/lib/B.h\t2026-10-16\n@@ -1,2 +1,3 @@\n int c;\n+++ looks like a header\n int d;\n" + "@@ -10 +11,2 @@\n--- looks like a header too\n+// kept\n+// ok\n") + got = cps.added_lines(d) + check("added_lines: only the + lines, per file, -p1 prefix and header timestamp dropped", + got == {"lib/A.cpp": ["int new;", "// SIMUT override — added"], + "lib/B.h": ["++ looks like a header", "// kept", "// ok"]}, str(got)) + got = cps.added_lines(diff("lib/A.cpp", "int x;", before="// " + MARK, removed=["// " + MARK + " v0"])) + check("added_lines: a marker in a context line or a removed line is not an added line", + got == {"lib/A.cpp": ["int x;"]}, str(got)) + nn = "--- a/f\n+++ b/f\n@@ -1,2 +1,2 @@\n a\n-b\n\\ No newline at end of file\n+b\n\\ No newline at end of file\n" + check("added_lines: '\\ No newline at end of file' inside a hunk counts for neither side", + cps.added_lines(nn) == {"f": ["b"]}, str(cps.added_lines(nn))) + blank = "--- a/f\n+++ b/f\n@@ -1,3 +1,4 @@\n a\n\n+x\n c\n" # the blank context line lost its space + check("added_lines: a context line that lost its space is still context", + cps.added_lines(blank) == {"f": ["x"]}, str(cps.added_lines(blank))) + + +def synthetic_cases(): + ok = {"a.patch": diff("lib/A.cpp", ADD)} + f, st = run(script(step()), ok) + check("a guard on the text its patch adds is no finding (and is read as a step)", not f and len(st) == 1, brief(f)) + + t = script(step(guard="SIMUT override — drop a non-TLS client")) + new_line = "// SIMUT override — tell a non-TLS client from a TLS one by its FIRST BYTE," + f, _ = run(t, {"a.patch": diff("lib/A.cpp", new_line)}) + exactly("2b-ter as it was: a sentinel the patch does not add is one finding, at the guard, naming what it adds", + f, "not-added", line_of(t, "if grep"), mention="tell a non-TLS client") + + t = script(step()) + f, _ = run(t, {"a.patch": diff("lib/A.cpp", "int x;", before="// " + MARK)}) + exactly("a sentinel only in a context line is not added", f, "not-added", line_of(t, "if grep")) + f, _ = run(t, {"a.patch": diff("lib/A.cpp", "int x;", removed=["// " + MARK])}) + exactly("a sentinel only in a removed line is not added", f, "not-added", line_of(t, "if grep")) + f, _ = run(t, {"a.patch": diff("lib/Z.cpp", ADD)}) + exactly("a patch that does not touch the grepped file is not added", f, "not-added", line_of(t, "if grep"), + mention="does not touch lib/A.cpp") + + t = script(step(), step(file='"$B"'), assigns='A="$FW/lib/A.cpp"\nB="$FW/lib/B.cpp"\n') + f, _ = run(t, {"a.patch": diff("lib/A.cpp", "int x;") + diff("lib/B.cpp", ADD)}) + exactly("the marker added to the OTHER file of the patch is not added; the guard on that file is clean", + f, "not-added", line_of(t, "if grep", 1)) + f, _ = run(t, {"a.patch": diff("lib/A.cpp", ADD) + diff("lib/B.cpp", ADD)}) + check("the same patch with the marker added to both files is clean", not f, brief(f)) + + t = script(step()) + f, _ = run(t, {}) + exactly("a patch file that is not there is no-patch, at the patch command", f, "no-patch", + line_of(t, "patch -p1 -d"), mention="patches/a.patch") + + unguarded = { + "a bare patch -p1": 'patch -p1 -d "$FW" < "$OVR/patches/a.patch"\n', + "a patch fed by a pipe": 'cat "$OVR/patches/a.patch" | patch -p1 -d "$FW"\n', + "a patch with other flags": 'if grep -q "' + MARK + '" "$A"; then\n :\nelse\n patch -p0 -d "$FW" < "$OVR/patches/a.patch"\nfi\n', + "a patch behind a test that is not a grep": 'if [ -f "$A" ]; then\n :\nelse\n patch -p1 -d "$FW" < "$OVR/patches/a.patch"\nfi\n', + "a patch in the THEN branch of a guard": 'if grep -q "' + MARK + '" "$A"; then\n patch -p1 -d "$FW" < "$OVR/patches/a.patch"\nfi\n', + } + for name, extra in unguarded.items(): + t = script(step(), extra) + f, _ = run(t, ok) # the guarded step before it is clean: one finding, not two + exactly(f"unguarded: {name} is one finding, at its patch command", f, "unguarded", line_of(t, "patch -p", 2)) + + for ch in ("$HOME", "`x`", "\\\\", "[a]", "a*", "^a"): + t = script(step(guard=MARK + " " + ch)) + f, _ = run(t, {"a.patch": diff("lib/A.cpp", "// " + MARK + " " + ch)}) + exactly(f"unreadable: a sentinel with {ch!r} is one finding although the patch adds that very text", + f, "unreadable", line_of(t, "if grep")) + for what, t in (("a variable that was never assigned", script(step(file='"$NOPE"'))), + ("a file outside $FW", script(step(file='"/etc/hosts"'))), + ("a command substitution", script(step(file='"$(pwd)/A.cpp"'))), + ("a command substitution after $FW", script(step(file='"$FW/lib/$(echo A).cpp"')))): + f, _ = run(t, ok) + exactly(f"unreadable: a guard file from {what} is one finding", f, "unreadable", line_of(t, "if grep")) + t = script(step(patch='"$NOPE/a.patch"')) + f, _ = run(t, ok) + exactly("unreadable: a patch path from a variable never assigned is one finding, at the patch command", + f, "unreadable", line_of(t, "patch -p1 -d")) + + f, st = run(script(step(), assigns='D="$FW/lib"\nA="$D/A.cpp"\n'), ok) + check("variables: a path built from another variable is resolved", not f and len(st) == 1, brief(f)) + f, st = run(script(step(), assigns='D="$FW/lib"\nA="${D}/A.cpp"\n'), ok) + check("variables: ${NAME} is resolved too", not f and len(st) == 1, brief(f)) + twice = script(step(), 'A="$FW/lib/B.cpp"\n', step(patch='"$OVR/patches/b.patch"')) + f, st = run(twice, {"a.patch": diff("lib/A.cpp", ADD), "b.patch": diff("lib/B.cpp", ADD)}) + check("variables: a name assigned AGAIN is read as of the guard that uses it", not f and len(st) == 2, brief(f)) + f, _ = run(twice, {"a.patch": diff("lib/A.cpp", ADD), "b.patch": diff("lib/A.cpp", ADD)}) + exactly("variables: the second guard greps B, so a patch that adds to A is one finding, at that guard", + f, "not-added", line_of(twice, "if grep", 2)) + + t = script(step(), step(guard="SIMUT override — beta", file='"$B"', patch='"$OVR/patches/b.patch"'), + assigns='A="$FW/lib/A.cpp"\nB="$FW/lib/B.cpp"\n') + f, st = run(t, {"a.patch": diff("lib/A.cpp", ADD), "b.patch": diff("lib/B.cpp", "// SIMUT override — beta")}) + check("two clean steps are two steps and no finding", not f and len(st) == 2, brief(f)) + + +def cli_cases(): + ok = {"a.patch": diff("lib/A.cpp", ADD)} + rc, out, err, _ = cli(script(step()), ok) + check("command: a clean tree exits 0 and says how many steps it read", + rc == 0 and out.strip().endswith("0 findings in 1 guarded steps") and not err, f"{rc} {out!r} {err!r}") + t = script(step(guard="SIMUT override — drop a non-TLS client")) + rc, out, err, d = cli(t, ok) + line = line_of(t, "if grep") + check("command: a finding exits 1 as `path:LINE: rule: text`, with the rule's hint on stderr", + rc == 1 and out.startswith(f"{d}/patch.sh:{line}: not-added: ") and "1 findings in 1 guarded steps" in out + and "check_patch_sentinels: not-added: grep the" in err, f"{rc} {out!r} {err!r}") + rc, out, err, _ = cli(script(), ok) + check("command: a script with no guarded step exits 2, not 0", rc == 2 and "does not understand" in err, f"{rc} {out!r} {err!r}") + t = script('patch -p1 -d "$FW" < "$OVR/patches/a.patch"\n') + rc, out, err, _ = cli(t, ok) + check("command: a script whose only patch is unguarded exits 1 with that finding (not 2: it did read something)", + rc == 1 and f":{line_of(t, 'patch -p1')}: unguarded: " in out, f"{rc} {out!r} {err!r}") + with tempfile.TemporaryDirectory() as d: + r = subprocess.run([sys.executable, GATE, "--ovr", d], capture_output=True, text=True) + check("command: no patch.sh exits 2 and says which file", r.returncode == 2 and "patch.sh" in r.stderr, f"{r.returncode} {r.stderr!r}") + with tree(script(step()), ok) as d: + other = os.path.join(d, "other.sh") + with open(other, "w", encoding="utf-8") as f: + f.write(script(step(guard="SIMUT override — drop a non-TLS client"))) + r = subprocess.run([sys.executable, GATE, "--ovr", d, "--patch-sh", other], capture_output=True, text=True) + check("command: --patch-sh reads another script against the same patches", r.returncode == 1 and "other.sh:6: not-added" in r.stdout, + f"{r.returncode} {r.stdout!r}") + + +def real_cases(): + sh = os.path.join(OVR, "patch.sh") + with open(sh, encoding="utf-8") as f: + text = f.read() + found, steps = cps.check(sh, OVR) + check("the real patch.sh: no finding", not found, brief(found)) + n = len(re.findall(r"^\s*patch -p1\b", text, re.M)) + check(f"the gate reads every `patch -p1` of the real script ({len(steps)} steps of {n})", n > 0 and len(steps) == n) + + def broken(edit): + """The real script with `edit(lines)` applied: (findings, steps, new text).""" + lines = text.split("\n") + edit(lines) + new = "\n".join(lines) + with tempfile.TemporaryDirectory() as d: + p = os.path.join(d, "patch.sh") + with open(p, "w", encoding="utf-8") as f: + f.write(new) + got, st = cps.check(p, OVR) + return got, st, new + + def one(got, rule, line): + return len(got) == 1 and got[0].rule == rule and got[0].line == line + + bad = [] + for s in steps: + got, _, _ = broken(lambda ls, s=s: ls.__setitem__(s.line - 1, ls[s.line - 1].replace(s.sentinel, "SIMUT override — nothing like this"))) + if not one(got, "not-added", s.line): + bad.append((s.line, brief(got))) + check(f"each of the {len(steps)} sentinels, changed in patch.sh, is ONE not-added finding, at its own guard", not bad, str(bad)) + + bad = [] + for s in steps: + with tempfile.TemporaryDirectory() as d: + shutil.copytree(os.path.join(OVR, "patches"), os.path.join(d, "patches")) + p = os.path.join(d, os.path.relpath(s.patch, OVR)) + with open(p, encoding="utf-8") as f: + body = f.read().split("\n") + hit = [i for i, line in enumerate(body) if line.startswith("+") and s.sentinel in line] + for i in hit: + body[i] = body[i].replace(s.sentinel, "SIMUT overrid3") + with open(p, "w", encoding="utf-8") as f: + f.write("\n".join(body)) + got, _ = cps.check(sh, d) + if not hit or not one(got, "not-added", s.line): + bad.append((s.line, len(hit), brief(got))) + check(f"each of the {len(steps)} markers, changed in its patch, is ONE not-added finding, at that step's guard", not bad, str(bad)) + + bad = [] + for s in steps: + got, _, _ = broken(lambda ls, s=s: ls.__setitem__(s.line - 1, "if true; then")) + if not one(got, "unguarded", s.patch_line): + bad.append((s.line, brief(got))) + check(f"each of the {len(steps)} steps, with its guard taken away, is ONE unguarded finding, at its patch command", not bad, str(bad)) + + bad = [] + for s in steps: + base = os.path.basename(s.patch) + with tempfile.TemporaryDirectory() as d: + p = os.path.join(d, "patch.sh") + with open(p, "w", encoding="utf-8") as f: + f.write(text.replace(base, "no_such_file.patch")) + got, _ = cps.check(p, OVR) + if not one(got, "no-patch", s.patch_line): + bad.append((s.patch_line, brief(got))) + check(f"each of the {len(steps)} patch files, gone, is ONE no-patch finding, at its patch command", not bad, str(bad)) + + ter = [s for s in steps if s.patch.endswith("wifiserversecure_drop_plaintext.patch")] + check("2b-ter is among the steps read", len(ter) == 1) + if ter: + s = ter[0] + got, _, _ = broken(lambda ls: ls.__setitem__(s.line - 1, ls[s.line - 1].replace(s.sentinel, "SIMUT override — drop a non-TLS client"))) + exactly('2b-ter with the sentinel of 2026-10-15 ("drop a non-TLS client") is one finding, at its guard, naming its patch', + got, "not-added", s.line, mention="wifiserversecure_drop_plaintext.patch") + + got, _, new = broken(lambda ls: ls.insert(len(ls) - 1, 'patch -p1 -d "$FW" < "$OVR/patches/new.patch"')) + exactly("a step added to the real script with no guard is one unguarded finding, at its line", + got, "unguarded", line_of(new, "new.patch")) + + +def read_only_cases(): + def digest(): + h = hashlib.sha256() + for path in [os.path.join(OVR, "patch.sh")] + sorted( + os.path.join(OVR, "patches", n) for n in os.listdir(os.path.join(OVR, "patches"))): + h.update(path.encode()) + with open(path, "rb") as f: + h.update(f.read()) + return h.hexdigest() + before = digest() + r = subprocess.run([sys.executable, GATE], capture_output=True, text=True) + cps.main([]) + check("the real tree through the command: exit 0, and the sha256 of patch.sh and every patch is the same after", + r.returncode == 0 and re.search(r"0 findings in \d+ guarded steps", r.stdout) and digest() == before, + f"{r.returncode} {r.stdout!r} {r.stderr!r}") + + +# ── run ──────────────────────────────────────────────────────────────────────────────────────── + +diff_cases() +synthetic_cases() +cli_cases() +real_cases() +read_only_cases() + +print(f"\n{len(RAN) - len(FAILS)} of {len(RAN)} checks passed") +if FAILS: + print("FAILED:\n " + "\n ".join(FAILS)) + sys.exit(1)