feat: close out the scanner improvement backlog

CHANGELOG.md

@@ -15,6 +15,11 @@ All notable changes to this project will be documented in this file.
 - `scan --git-history --rev-range <range>` restricts a history scan to a revision range; the pre-push hook uses it to scan exactly the pushed commits
 - Detectors for GitHub fine-grained PATs (`github_pat_`), GitHub refresh tokens (`ghr_`), AWS temporary keys (`ASIA`/`ABIA`/`ACCA`), AWS secret access keys (context-gated), modern Slack bot tokens, Slack app tokens (`xapp-`), Slack webhook URLs, and modern OpenAI project keys (`sk-proj-`/`sk-svcacct-`/`sk-admin-`)
 - Non-verbose scans print the location, type and redacted match of every finding instead of only a count
+- GitHub classic tokens are validated against their built-in CRC32 checksum, so base62 lookalike strings are rejected offline
+- UTF-16 files and staged blobs with a byte-order mark are decoded and scanned instead of being skipped as binary
+- Base64 runs of 24 or more characters are decoded and their text is scanned once more, so an encoded credential cannot pass unseen
+- `--trusted-detectors` and `--no-repo-config` split the two behaviors of `--no-config-discovery`, which remains as their combination
+- `--max-file-size <MB>` skips larger files and reports them as unscannable
 
 ### Changed
 

README.md

@@ -53,45 +53,54 @@ Control the output:
 ```sh
 key-watch scan . --verbose               # print the full JSON report
 key-watch scan . --output report.json    # write the report to a file
-key-watch scan . --format sarif          # write SARIF instead of JSON
+key-watch scan . --format sarif --output report.sarif   # write SARIF to a file
 ```
 
 By default, KeyWatch prints one line per finding with the file, line number, and a redacted preview.
 Reports never contain the full matched text unless you pass `--show-secrets`.
 
 ### Scan options
 
-| Option | Purpose |
-| ------ | ------- |
-| `--exclude <patterns>` | Skip paths that match these comma-separated glob patterns |
-| `--exit-mode <mode>` | `strict` fails on any finding (default), `critical` fails only on HIGH or CRITICAL findings, `always` never fails |
-| `--fail-on-unscannable` | Fail when a file or directory could not be read |
-| `--baseline <path>` | Use a specific baseline file |
-| `--no-baseline-discovery` | Do not look for a baseline file automatically |
-| `--update-baseline` | Record the current findings in the baseline instead of reporting them |
-| `--prune-baseline` | With `--update-baseline`, also remove baseline entries that no longer match anything |
-| `--config <path>` | Use a specific `.keywatch.toml` configuration file |
-| `--no-config-discovery` | Ignore configuration and detector files found in the scanned repository |
-| `--show-secrets` | Include the full matched text in reports |
+| Option                    | Purpose                                                                                                           |
+| ------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| `--exclude <patterns>`    | Skip paths that match these comma-separated glob patterns                                                         |
+| `--exit-mode <mode>`      | `strict` fails on any finding (default), `critical` fails only on HIGH or CRITICAL findings, `always` never fails |
+| `--fail-on-unscannable`   | Fail when a file or directory could not be read; applies in `strict` exit mode and not with `--update-baseline`   |
+| `--baseline <path>`       | Use a specific baseline file                                                                                      |
+| `--no-baseline-discovery` | Do not look for a baseline file automatically                                                                     |
+| `--update-baseline`       | Record the current findings in the baseline instead of reporting them                                             |
+| `--prune-baseline`        | With `--update-baseline`, also remove baseline entries that no longer match anything                              |
+| `--config <path>`         | Use a specific `.keywatch.toml` configuration file                                                                |
+| `--trusted-detectors`     | Ignore a `detectors.toml` supplied by the scanned repository; use only built-in or operator rules                 |
+| `--no-repo-config`        | Do not look for `.keywatch.toml` in the scanned tree; an explicit `--config` still loads                          |
+| `--no-config-discovery`   | Shorthand for `--trusted-detectors` plus `--no-repo-config`; the installed hooks pass it                          |
+| `--show-secrets`          | Include the full matched text in reports                                                                          |
+| `--max-file-size <MB>`    | Skip files larger than this size and report them as unscannable                                                   |
 
 Notes:
 
 - Lock files such as `Cargo.lock`, `package-lock.json`, `pnpm-lock.yaml`, and `yarn.lock` are always skipped.
   They contain checksums, not credentials.
 - `--staged` reads the content you staged with `git add`, not the files on disk.
   A secret that is staged but already removed from the working copy is still found.
+  A secret whose lines were staged in separate commits can span change hunks the diff never shows together; run `key-watch scan .` on the tree to catch that case.
 - `--git-history` scans every branch and tag.
   Use `--rev-range` to scan only a range of commits.
 - A scan path that does not exist, is a symbolic link, or cannot be read is an error.
   The scan never reports a clean result for input it could not read.
+- Files that start with a UTF-16 byte-order mark are decoded and scanned.
+  Other files that contain NUL bytes are treated as binary and reported as unscannable.
+- Base64 runs of 24 or more characters are decoded, and the decoded text is scanned as well.
+  An encoded credential is reported at the line that contains it.
+- GitHub tokens are checked against their built-in checksum, so lookalike strings do not appear in results.
 
 ### Exit codes
 
-| Code | Meaning |
-| ---- | ------- |
-| 0 | No secrets found, or `--exit-mode always` |
-| 1 | Secrets found, or an unreadable file with `--fail-on-unscannable` |
-| 2 | Invalid input, configuration error, or runtime error |
+| Code | Meaning                                                           |
+| ---- | ----------------------------------------------------------------- |
+| 0    | No secrets found, or `--exit-mode always`                         |
+| 1    | Secrets found, or an unreadable file with `--fail-on-unscannable` |
+| 2    | Invalid input, configuration error, or runtime error              |
 
 ## Git hooks
 
@@ -101,7 +110,7 @@ KeyWatch installs two git hooks:
   A secret in staged content blocks the commit.
   Findings in lines you did not change never block a commit.
 - The **pre-push** hook scans the commits you are about to push.
-  A secret in those commits blocks the push.
+  It runs in `critical` exit mode, so HIGH and CRITICAL findings block the push; MEDIUM and LOW findings are reported but do not block.
   Uncommitted files never block a push.
 
 Install and remove hooks inside a repository:
@@ -122,11 +131,11 @@ key-watch hook uninstall pre-commit --global
 
 ### Hook options
 
-| Option | Applies to | Purpose |
-| ------ | ---------- | ------- |
-| `--exclude <patterns>` | pre-commit | Skip staged paths that match these patterns |
-| `--allowed-repos <urls>` | pre-push | Allow pushes only to these repositories |
-| `--blocked-repos <urls>` | pre-push | Block pushes to these repositories |
+| Option                   | Applies to | Purpose                                     |
+| ------------------------ | ---------- | ------------------------------------------- |
+| `--exclude <patterns>`   | pre-commit | Skip staged paths that match these patterns |
+| `--allowed-repos <urls>` | pre-push   | Allow pushes only to these repositories     |
+| `--blocked-repos <urls>` | pre-push   | Block pushes to these repositories          |
 
 ### How hooks behave
 
@@ -212,15 +221,15 @@ The Action installs a released KeyWatch binary, verifies its checksum, and write
 It supports Linux x64 and macOS runners.
 Pin an exact release tag or commit SHA when you need a fixed version.
 
-| Input | Default | Purpose |
-| ----- | ------- | ------- |
-| `version` | Action release version | Exact KeyWatch release to install |
-| `paths` | `.` | Space-separated paths or globs to scan |
-| `args` | empty | Extra scanner arguments; Action-managed options cannot be overridden |
-| `exit-mode` | `strict` | `strict`, `critical`, or `always` |
-| `output` | temporary file | Path for the JSON report |
-| `config` | empty | Path to a trusted `.keywatch.toml` |
-| `verbose` | `false` | Deprecated; enabling it is rejected to keep secrets out of logs |
+| Input       | Default                | Purpose                                                              |
+| ----------- | ---------------------- | -------------------------------------------------------------------- |
+| `version`   | Action release version | Exact KeyWatch release to install                                    |
+| `paths`     | `.`                    | Space-separated paths or globs to scan                               |
+| `args`      | empty                  | Extra scanner arguments; Action-managed options cannot be overridden |
+| `exit-mode` | `strict`               | `strict`, `critical`, or `always`                                    |
+| `output`    | temporary file         | Path for the JSON report                                             |
+| `config`    | empty                  | Path to a trusted `.keywatch.toml`                                   |
+| `verbose`   | `false`                | Deprecated; enabling it is rejected to keep secrets out of logs      |
 
 The Action exposes `findings-count` and `exit-code` as step outputs.
 
@@ -251,6 +260,49 @@ rm -f ~/.local/bin/key-watch
 
 In both cases, remove the `key-watch init` line from your shell configuration file if you added one.
 
+## Architecture
+
+KeyWatch is a single Rust binary.
+`main.rs` starts the program and maps every validation, configuration, or runtime failure to exit code 2.
+Scans exit with code 0 or 1.
+Separate modules own detector loading, scanning, baselines, reports, and hooks.
+
+### Modules and adapters
+
+![KeyWatch CLI module and adapter architecture](docs/architecture/cli-modules.svg)
+
+Green boxes are internal modules.
+Blue boxes are entry and output boundaries.
+Yellow boxes are external adapters such as git and the installed hook scripts, which call `key-watch scan` themselves.
+
+### Scan pipeline
+
+![KeyWatch scan pipeline](docs/architecture/scan-pipeline.svg)
+
+Path scans collect files and scan them in parallel.
+Stdin and git-based scans stream their input in overlapping chunks.
+`--update-baseline` writes the baseline instead of producing a report.
+
+### Detector and configuration trust
+
+![KeyWatch detector and configuration trust boundaries](docs/architecture/detector-config-trust.svg)
+
+Detector rules and repository configuration are separate systems.
+External detector sources take precedence, and the compiled-in rules are the fallback.
+Trusted scans ignore files supplied by the scanned repository but still honor explicit configuration and operator-supplied detector sources.
+
+### Core data types
+
+- **Detector** — one named rule: pattern, finding type, severity, optional keywords, entropy threshold, allowlist, and validator.
+- **Finding** — one detected secret: file path, line number, finding type, severity, matched content, and the detector that produced it.
+- **Severity** — `Critical`, `High`, `Medium`, `Low`.
+- **KeywatchConfig** — parsed `.keywatch.toml`: custom rules, per-detector overrides, and exclude patterns.
+- **Baseline** — versioned fingerprint entries that filter out known findings.
+- **ScanMetadata** — files scanned, total lines, and skipped files, reported alongside findings.
+
+The diagram sources are in `docs/architecture/*.d2`.
+After editing them, run `scripts/render-diagrams.sh render` with D2 v0.7.1, or `scripts/render-diagrams.sh check` to detect stale images.
+
 ## Development
 
 ```sh
@@ -260,9 +312,6 @@ cargo fmt
 cargo clippy
 ```
 
-Architecture diagrams live in `docs/architecture/`.
-Edit the `.d2` sources and run `scripts/render-diagrams.sh render` to update the rendered images.
-
 ## License
 
 KeyWatch is licensed under the GPL-3.0-only license.

detectors.toml

@@ -67,6 +67,10 @@ pattern = "\\b(?:ghp|gho|ghu|ghs|ghr)_[0-9a-zA-Z]{36,255}\\b"
 finding_type = "GitHub Token"
 severity = "HIGH"
 keywords = ["ghp_", "gho_", "ghu_", "ghs_", "ghr_"]
+# Classic 40-character tokens carry a CRC32 checksum in their last six
+# characters; validating it offline rejects lookalike strings. Longer
+# bodies pass unchecked so a future format is not silently dropped.
+validate = "github-token-checksum"
 
 [[detectors]]
 name = "GitHubFineGrainedPATDetector"

docs/architecture/detector-config-trust.d2

@@ -39,11 +39,11 @@ classes: {
   }
 }
 
-mode: "SCAN MODE\nnormal or trusted¹\n¹ --no-config-discovery" {class: warning}
+mode: "SCAN FLAGS\n¹ dropped by --trusted-detectors\n² skipped by --no-repo-config\n--no-config-discovery = both" {class: warning}
 
 detectors: "DETECTOR SOURCE\nKEYWATCH_CONFIG_PATH → repository¹\n→ user → executable → embedded" {class: embedded}
 
-policy: "POLICY SOURCE\nexplicit --config, or repository¹\n.keywatch.toml → keywatch.toml → .kw.toml" {class: repository}
+policy: "POLICY SOURCE\nexplicit --config, or repository²\n.keywatch.toml → keywatch.toml → .kw.toml" {class: repository}
 
 merge: "VALIDATE + MERGE\ncustom rules · overrides · excludes" {class: trusted}
 result: "FINAL RULES\ndetectors + exclusion policy" {class: trusted}