Fix recorder re-entry and atomic trace buffer delivery

docs/experiments/trace-integrity-results.json

@@ -0,0 +1,125 @@
+{
+  "measurement_date": "2026-10-13",
+  "traincheck_base": "9eb4f320d3db34d372145261704b805b88e39bda",
+  "benchmark_source": "ae6601978217bc0f06b9a78479cf00f0bc7242c9",
+  "controlled_reentry": {
+    "buggy": {
+      "application_calls": 100,
+      "events": 400,
+      "metadata_helper_events": 200,
+      "application_results_match": true,
+      "recursion_errors": 0,
+      "scope": "Controlled injected conversion helper, not a natural workload frequency"
+    },
+    "fixed": {
+      "application_calls": 100,
+      "events": 200,
+      "metadata_helper_events": 0,
+      "application_results_match": true,
+      "recursion_errors": 0,
+      "scope": "Controlled injected conversion helper, not a natural workload frequency"
+    }
+  },
+  "controlled_buffer_interleavings": {
+    "scope": "Controlled forced interleavings, not a natural workload frequency or performance benchmark",
+    "trials_per_mode": 50,
+    "records_preloaded_per_trial": 100,
+    "results": {
+      "buggy": {
+        "duplicate": {
+          "records": 10000,
+          "duplicates": 5000,
+          "missing": 0
+        },
+        "loss": {
+          "records": 5000,
+          "duplicates": 0,
+          "missing": 50
+        }
+      },
+      "fixed": {
+        "duplicate": {
+          "records": 5000,
+          "duplicates": 0,
+          "missing": 0
+        },
+        "loss": {
+          "records": 5050,
+          "duplicates": 0,
+          "missing": 0
+        }
+      }
+    }
+  },
+  "natural_workload_matrix": {
+    "runs_completed": 45,
+    "instrumented_runs": 36,
+    "conditions": [
+      "baseline",
+      "buggy",
+      "guard",
+      "buffer",
+      "fixed"
+    ],
+    "seeds": [
+      42,
+      43,
+      44
+    ],
+    "training_steps": 5,
+    "batch_size": 2,
+    "native_threads": 1,
+    "runtime": {
+      "python": "3.10.16",
+      "torch": "2.2.2+cu121",
+      "gpu": "RTX A2000 12 GB"
+    },
+    "scope": "Adapted small model workloads with ordinary local tensors; not full published benchmark reproductions. All instrumented variants share the same dynamic-type-name compatibility patch.",
+    "instrumentation": "Default API coverage, use_full_instr=False, sampling interval one, no model/proxy tracking; training bodies only.",
+    "workloads": [
+      {
+        "name": "mnist",
+        "parameters": 1199882,
+        "instrumented_runs": 12,
+        "records_per_run": 654,
+        "raw_bytes_per_run": 472863,
+        "instrumented_variants_and_seeds_equal": true
+      },
+      {
+        "name": "transformer",
+        "parameters": 23522,
+        "instrumented_runs": 12,
+        "records_per_run": 2006,
+        "raw_bytes_per_run": 1564970,
+        "instrumented_variants_and_seeds_equal": true
+      },
+      {
+        "name": "dcgan",
+        "parameters": 190672,
+        "instrumented_runs": 12,
+        "records_per_run": 4596,
+        "raw_bytes_per_run": 3401128,
+        "instrumented_variants_and_seeds_equal": true
+      }
+    ],
+    "total_instrumented_records": 87072,
+    "total_instrumented_raw_bytes": 65267532,
+    "all_recorded_losses_and_final_model_state_hashes_match_baseline": true,
+    "observed_duplicate_records": 0,
+    "observed_recursion_error_events": 0,
+    "natural_trace_size_reduction_observed": false
+  },
+  "evidence_artifact_sha256": {
+    "matrix.csv": "f196211cc478ec532338de06247895cfa5bcd379770fd66583c5b80189127cfe",
+    "matrix-summary.json": "a6a18173e3a0550500160b0e0b3169553ee714bb0ce175048cc5998f19c2ba06",
+    "controlled-races.json": "6b0ab42d2d8f1ff37557627bd8b261610f7d84adf65f88e2025739dfd875c4aa",
+    "finite-reentry-buggy.json": "25f5dfeab0f68c080682f21f0fd6e0314e7c4254a2a9e538b9081aad3a382475",
+    "finite-reentry-fixed.json": "18470fb8d6c929296feecf3485242775dd63331e89c0669244d7ed763bdf6595"
+  },
+  "limitations": [
+    "Injected helpers and forced interleavings establish possible failure severity, not natural occurrence rates.",
+    "All 45 ordinary-tensor runs completed; none of the 36 instrumented runs showed trace inflation or a trace-size reduction from these fixes.",
+    "No general performance improvement or invariant-checking overhead claim is made.",
+    "Historical incomplete and completed TorchTitan traces are not comparable percentage-size-reduction measurements."
+  ]
+}

docs/experiments/trace-integrity.md

@@ -0,0 +1,68 @@
+# Recorder re-entry and buffer integrity
+
+Recorder-owned argument conversion, tensor metadata inspection, and JSON encoding can call instrumented APIs again. The wrapper previously did not honor the conversion helper's local suppression flag, so recorder work could generate more records or recurse. Independently, the producer and periodic flusher accessed the same buffer without synchronization: overlapping flushes could duplicate a batch, and a producer append could be cleared before delivery.
+
+The recorder now uses an exception-safe, thread-local suppression depth around its own work. It releases suppression before executing application code, preserving nested application calls and generator execution. A class context manager preserves frozen framework exceptions. The buffer snapshots mutable values synchronously, then locks append, queue handoff, and clear as one ordered operation. Buffers still belong to individual producer threads for trace routing.
+
+## Controlled impact
+
+These tests deliberately inject a conversion helper or force a thread interleaving; they do not estimate how often an ordinary training job triggers either defect.
+
+| Controlled trigger | Buggy | Fixed |
+|---|---:|---:|
+| 100 application calls with a wrapped serialization helper | 400 records, including 200 recorder-owned records | 200 records, zero recorder-owned records |
+| Two overlapping flushes; 50 trials × 100 records | 10,000 emitted, including 5,000 duplicates | 5,000 emitted, zero duplicates |
+| Append during paused flush; 50 trials, 5,050 expected records | 50 records missing | Zero records missing |
+
+The finite re-entry case preserves application return values and raises no `RecursionError`: self-tracing can inflate a trace without an obvious recursion failure. The buffer race can cause loss as well as duplication; valid pre/post pairing alone cannot detect a completely missing pair.
+
+## Ordinary workload control
+
+All 45 runs completed: MNIST CNN, a small Transformer, and reduced-width DCGAN × five conditions × seeds 42/43/44. Each run used five training steps, batch size two, one GPU, and one native CPU thread. Conditions were no instrumentation, the buggy snapshot, guard only, buffer lock only, and both fixes.
+
+None of the 36 instrumented runs triggered either defect. Each instrumented condition had the same record count and raw byte size at every seed:
+
+| Workload | Records per run | Raw bytes per run | Buggy → fixed size change |
+|---|---:|---:|---:|
+| MNIST CNN | 654 | 472,863 | 0% |
+| Small Transformer | 2,006 | 1,564,970 | 0% |
+| DCGAN | 4,596 | 3,401,128 | 0% |
+
+All recorded losses and canonical final model-state hashes matched their corresponding uninstrumented baselines. These are adapted micro-workloads with ordinary local tensors on PyTorch 2.2.2+cu121, not full published benchmark reproductions or DTensor coverage. The observations support these correctness repairs; they do not establish a general performance improvement or invariant-checking overhead result.
+
+## Validation and provenance
+
+Run the eleven portable regression tests from the checkout:
+
+```sh
+OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python -m unittest tests.test_trace_integrity -v
+```
+
+The suite covers recorder re-entry, JSON emission re-entry, real tensor snapshots, nested application calls, generators, original and frozen exceptions, thread-local suppression, overlapping flushes, and concurrent producer ordering.
+
+[Sanitized results](trace-integrity-results.json) retain controlled counts, workload settings, base/source revisions, and evidence-file hashes. Measurements used TrainCheck base `9eb4f320d3db34d372145261704b805b88e39bda` and adapted benchmark source `ae6601978217bc0f06b9a78479cf00f0bc7242c9`; all instrumented conditions shared the same dynamic-type-name compatibility patch. No private paths, raw training traces, or checkpoint data are included.
+
+## Contract hardening (2026-10-25)
+
+Runtime checkpoint `aba4ee8e164f4d24eedae97ea641cd9a6f69c343` makes the
+thread-local suppression state, context-manager exception/traceback interface,
+and buffer queue/record types explicit. Suppression starts at zero independently
+in each thread; process/thread IDs retain their existing lazy initialization.
+`__exit__` returns `Literal[False]`, preserving application exceptions. Buffer
+serialization and queue failures propagate; a failed enqueue retains its batch
+for a later successful flush without dropping or duplicating records.
+
+Validation used Python 3.10.16, PyTorch 2.2.2, and one OpenMP/MKL/OpenBLAS thread:
+
+```sh
+python -m unittest discover -s tests -p 'test_trac*.py'
+pre-commit run --files traincheck/instrumentor/tracer.py traincheck/instrumentor/dumper.py tests/test_tracing_contracts.py
+mypy --follow-imports=silent --ignore-missing-imports --check-untyped-defs --disallow-untyped-defs --disallow-any-generics tests/test_tracing_contracts.py
+```
+
+All 15 tests passed (11 existing regressions and 4 state/write-failure contract
+tests). All repository-pinned hooks passed; the stronger check of the new test
+module used pinned mypy 1.9. Legacy serializer and proxy-wrapper annotations are
+outside this narrowly scoped change; this does not claim repository-wide strict
+typing. No benchmark timings were rerun for this typing/contract checkpoint;
+the experiment numbers above remain evidence for their original revisions.