Reorganise the Martha Hamilton catalog to Ashley's folder order

.github/workflows/martha-catalog-check.yml

@@ -0,0 +1,85 @@
+name: Martha catalog checks
+
+# verify.py is the only re-runnable check on the 167-page Martha catalog. Without CI it
+# only runs when a human remembers, and an edit can silently break the byte-identity and
+# provenance claims the catalog rests on. Ashley's ordered folder is not available here,
+# so the byte-for-byte comparison against her originals is skipped; everything else runs.
+
+on:
+  push:
+    paths:
+      - "review/martha/**"
+      - "review/martha.html"
+      - "review/martha-audit.html"
+      - "review/martha-schema.md"
+      - ".github/workflows/martha-catalog-check.yml"
+  pull_request:
+    paths:
+      - "review/martha/**"
+      - "review/martha.html"
+      - "review/martha-audit.html"
+      - "review/martha-schema.md"
+      - ".github/workflows/martha-catalog-check.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Verify catalog, provenance and reports
+        run: python review/martha/verify.py
+
+      # verify.py asserts things about the rotation table; check_matrix.py asserts that
+      # verify.py actually enforces them. It runs the whole matrix in a temp copy of
+      # review/, so it cannot dirty the checkout the next step diffs.
+      - name: Verify the guards actually fire
+        run: python review/martha/check_matrix.py
+
+      # check_matrix.py audits prose and its own control proves that audit can still see. Neither
+      # can notice the control being deleted, so this check is made from outside the module: read
+      # the retired list out of the source text, inject one decoded phrase into a copy, and require
+      # the copy to refuse to run. An emptied list, a broken decode, a neutered control and a dead
+      # audit all fail here; the module's own control cannot see that last one.
+      - name: Verify the prose ratchet from outside the module
+        run: |
+          python - <<'PY'
+          import pathlib, re, shutil, subprocess, sys, tempfile
+          src = pathlib.Path("review/martha/check_matrix.py").read_text(encoding="utf-8")
+          m = re.search(r"RETIRED_PROSE = \[(.*?)\n\]", src, re.S)
+          if not m:
+              sys.exit("could not find the RETIRED_PROSE list: the ratchet is gone")
+          entries = re.findall(r'"([^"]+)"', m.group(1))
+          if not entries:
+              sys.exit("RETIRED_PROSE is empty: the ratchet guards nothing")
+          d = pathlib.Path(tempfile.mkdtemp())
+          try:
+              probe = d / "check_matrix.py"
+              probe.write_text(src + "\n# " + entries[0].replace("_", " ") + "\n", encoding="utf-8")
+              try:
+                  p = subprocess.run([sys.executable, str(probe)], capture_output=True,
+                                     text=True, timeout=300)
+              except subprocess.TimeoutExpired:
+                  sys.exit("injected copy never returned: the audit did not short-circuit")
+          finally:
+              shutil.rmtree(d, ignore_errors=True)
+          out = p.stdout or ""
+          if p.returncode == 0 or "retired prose" not in out:
+              sys.exit("the prose ratchet did not fire on an injected retired phrase (rc=%s)"
+                       % p.returncode)
+          print("prose ratchet fires from outside: %d entries" % len(entries))
+          PY
+
+      - name: Rebuild reports
+        run: python review/martha/build_reports.py
+
+      - name: Fail if the committed reports are stale
+        run: git diff --exit-code -- review/martha.html review/martha-audit.html

oral-histories/services.html

@@ -62,7 +62,7 @@ <h2>Oral History Project Services</h2>
 
         <section>
             <h2>Pricing</h2>
-            <p>My professional consulting rate is $200 per hour.</p>
+            <p>My professional consulting rate is $250 per hour.</p>
             <p>A typical oral history project requires approximately 15 hours of work per interview subject, including research, preparation, interviewing, transcription management, editing, and final deliverables.</p>
             <p>Estimated project cost per oral history interview:</p>
             <p><strong>15 hours × $250/hour = $3,750 per interview subject</strong></p>

review/martha-review.js

@@ -288,10 +288,14 @@ const MarthaReview = (() => {
         const pages = (doc.pages || []).map((p) => ({
           imageFile: p.imageFile,
           rotation: p.rotation || 0,
+          pageNumber: p.pageNumber,
           label: `${doc.docId} — ${doc.title} · p. ${p.pageNumber}`,
         }))
         if (!pages.length) return
-        openLightbox(pages, 0)
+        // Optional 1-based page number so a per-page thumbnail opens on that page.
+        const want = parseInt(el.getAttribute("data-open-page"), 10)
+        const at = Number.isNaN(want) ? 0 : pages.findIndex((p) => p.pageNumber === want)
+        openLightbox(pages, at < 0 ? 0 : at)
       })
     })
   }

review/martha-schema.md

@@ -4,7 +4,9 @@ This schema governs how each image in the source folder is described and how the
 images are grouped into abstracted documents. All reader sub-agents must return
 strict JSON in this shape.
 
-Source folder: `C:\Users\cubap\LesTrouves\Martha M. Hamilton Documents`
+Source folder (original scans): `review/martha/images/` in this repository.
+Ordered collection (defines the catalog): `U:\Downloads\Martha M. Hamilton Documents (ordered)\Martha M. Hamilton Documents` —
+Ashley Hamilton's own grouping of the scans into numbered folders.
 
 ## Rules for reading
 
@@ -38,32 +40,109 @@ Pick the best fit:
   "documents": [
     {
       "id": "DOC-001",
+      "folder": "Martha 1",
+      "order": 1,
+      "orderBasis": "internal-pagination",
       "title": "Short title",
       "documentType": "arbitration-exhibit",
       "subject": "Who/what it concerns",
       "date": "YYYY or approx date or unknown",
       "pageCount": 2,
       "confidence": "high",
+      "orderConfidence": "high",
       "notes": "Brief summary / why grouped this way",
+      "previousDocIds": ["DOC-012", "DOC-056"],
       "pages": [
         {
           "imageFile": "00037A57-A636-46C7-8FEC-7954417D9EBE_1_105_c.jpeg",
           "pageNumber": 1,
           "pageKind": "content",
-          "transcript": "Full transcription of visible text"
+          "transcript": "Full transcription of visible text",
+          "transcriptionConfidence": "high",
+          "orderBasis": "internal-pagination",
+          "previousDocId": "DOC-012",
+          "sourceFilename": "Martha 1/486E2B93-....jpeg",
+          "supersedes": "B559737D-....jpeg",
+          "notFiledReason": "only for pages in the Not filed group"
         }
       ]
     }
-  ]
+  ],
+  "supersededPages": []
 }
 ```
 
 `pageKind` values: `content`, `cover`, `divider`, `table-photo`, `photograph`, `blank`, `partial`.
 
-`confidence` values: `high`, `medium`, `low`.
+Two confidence fields, deliberately separate:
 
-Group consecutive images that appear to be the same document into one document
-objects, preserving order. Separate them into different documents when the
-content clearly changes to a different item (different header, different
-document, obvious new exhibit). Include empty `table-photo` separators inside the
-document `pages` array where they appear between the real pages.
+| field | on | rates | values |
+| --- | --- | --- | --- |
+| `confidence` | document | how reliably the text could be transcribed | `high`, `medium`, `low` |
+| `orderConfidence` | document | how firmly the page sequence is evidenced | `high`, `mixed`, `medium` |
+
+`confidence` is the original schema's meaning and is inherited from the document's least
+legible page. `orderConfidence` is derived from the pages' `orderBasis` (see below).
+
+## Catalog structure and order
+
+The catalog mirrors Ashley Hamilton's ordered collection rather than the order the
+scans happened to be read in. **One folder is one document**, and the image files
+inside it are that document's pages, in sequence.
+
+- `documents` is emitted in catalog order and `build_reports.py` must not re-sort it.
+- `docId` runs `DOC-001`…`DOC-029` following the folder number.
+- `folder` is Ashley's folder name; `order` is its 1-based position in the catalog.
+- Pages Ashley left loose in the folder root form the `Unfiled` document; payload
+  pages with no counterpart in her set form the `Not filed` document. Both come last.
+- `previousDocIds` / `previousDocId` point back at the pre-reorganisation catalogue,
+  which grouped pages by what appeared to be one item rather than by folder. Because
+  Ashley's folders are thematic, one old document can be split across several
+  folders and one folder can gather pages from several old documents.
+- `sourceFilename` records Ashley's own filename where it differs from the stored
+  image name. It is the provenance link for pages she renamed or re-scanned; identity
+  was established by content hash, never by filename.
+- `supersedes` / `supersededPages` record pages removed from the catalogue because
+  Ashley supplied a better scan of the same page. The old files stay on disk.
+
+### `orderBasis`
+
+How much the page sequence inside a document can be trusted:
+
+| value | meaning |
+| --- | --- |
+| `explicit` | Ashley's own filename states the page number. |
+| `internal-pagination` | Printed page numbers or unbroken paragraph numbering on the pages fix the sequence. |
+| `content-sequence` | Pages of one physical document, rejoined by continuity of the text. |
+| `editorial-alphabetical` | Our arrangement, by comparator surname. |
+| `editorial-chronological` | Our arrangement, by dates written on the pages. |
+| `not-asserted` | The grouping is Ashley's; the sequence within it is ours and makes no claim about her intent. |
+
+Ashley encoded folder membership, not page order, so only `explicit` and
+`internal-pagination` may be presented as her ordering.
+
+`orderBasis` is recorded **per page** as well as per document. A document's own value is
+the label its folder was given; where the evidence only covers part of the folder, the
+weaker pages carry their own value and the document is rated `orderConfidence: "mixed"`.
+Downgrades are declared in `order_table.py` as `page_basis={pageNumber: basis}` so the
+curated table stays the single source of truth. A document is `high` when every page is
+`explicit` or `internal-pagination`, `mixed` when only some are, and `medium` when none
+are. Do not let every page simply copy the folder value — a uniform copy carries no
+information and will contradict the notes.
+
+Grouping follows Ashley's folders, not a judgement about where one item ends: every
+folder is exactly one document, and every image in it is a page of that document.
+Do not split a folder across documents or merge two folders, even when the content
+looks like separate items — that is what her folder names are for. Keep empty
+`table-photo` separators inside the document `pages` array where they appear between
+the real pages.
+
+### Scripts
+
+| file | role |
+| --- | --- |
+| `order_table.py` | the curated page order for each folder, with its `orderBasis` |
+| `titles.py` | title, type, subject, date and notes per document; not-filed reasons |
+| `build_reports.py` | renders `martha.html` and `martha-audit.html` from `master-data.json` |
+| `build_master.py` | one-shot migration that produced the current `master-data.json` (provenance only) |
+| `verify.py` | re-runnable checks on the catalogue and reports |

review/martha/.gitignore

@@ -1,2 +1,3 @@
 __pycache__/
 *.pyc
+master-data.json.bak