[Essoz]

The previous identity-cache and time-window experiments used different baselines, so multiplying their speedups did not establish the combined end-to-end result. This documentation-only PR adds a direct matched comparison against the final main code and a separate tracing hotspot investigation.

Complete offline inference

On the same saved 573,494-event trace from a 10-step, 6.16M-parameter TorchTitan Llama debug run:

Metric	Corrected pre-identity-cache baseline	Final main
Complete offline wall time	1,439.538 s	76.686 s
Sampled peak process-tree PSS	2.126628 GiB	2.165641 GiB
Invariants / failed hypotheses	836 / 104	836 / 104

18.77× faster, 94.67% less wall time, +39.95 MiB peak PSS. The entire invariant file, failed-hypothesis file, and both 711-state context snapshots are byte-identical. No IDs, arrays, duplicates, or ordering were discarded.

The frozen baseline already includes the earlier context/call indexes; both variants receive the same #145 pruning correction. Candidate: 205a687. The denominator covers the entire offline process, including startup, preparation, inference, output writing, and exit. This is one baseline-first pair with one worker/native thread on a shared CPU host; it is not a distributed, training, or selective-checking speedup.

Tracing investigation

The original bounded CPU audit enabled API tracing only. It found object-to-dictionary conversion, repeated warning contexts, and type classification dominating separate profiles. Final JSON encoding alone was about 0.4%; this did not measure tensor hashing or full model tracing. The detailed report preserves disjoint self-time accounting and shows that Transformer has 2.57x the intercepted calls and 2.85x the added tracing time, with a much smaller plain-time denominator.

API versus model tracing, actually activated

A new v0.2.0 CPU experiment registers all eight MNIST model parameters using subclass tracking, installs normal static observers, and enables default tensor hashes. Three fresh-process repeats in rotated order measure three training steps:

Mode	Median training wall time
Plain	29.668 ms
API only	338.689 ms
Model/VAR only	414.599 ms
API + model/VAR	727.127 ms

The combined added time is 697.459 ms. Differences of configuration medians give 309.021 ms API (44.31%), 384.932 ms VAR (55.19%), and 3.506 ms interaction/noise (0.50%). These are ablations, not an exclusive physical-stage timer. The small interaction is below what these three repeats can resolve; VAR-only ranges 372.319–474.165 ms. Adding model tracking atop API tracing costs 388.438 ms in the median comparison. Setup/imports, model-state validation, and final writer drain are excluded.

Every VAR-enabled run emits 64 VAR records covering all eight parameters, with 112 data/gradient hash fields and changed pre/post-update hashes. API-enabled runs emit 520 complete API records. All 12 ordinary cases and the separate profile preserve complete losses and initial/final model state. A separate combined profile has variable snapshots at 38.05% cumulative time, including tensor hashing at 24.28%; these nested percentages use profiled time, not ordinary added time. The 128x9216 FC weight accounts for 98.31% of logical hashed input bytes, not necessarily time.

Tracker style is explicit: this measures subclass, not the CLI-default proxy mode, whose writer is synchronous. VAR-only takes the production no-dump API-wrapper path. The Transformer VAR run fails when assigning its attention-mask tensor: subclass_setattr_hook passes a Tensor to proxy_parameter, which assumes named_parameters. The failure is retained; no Transformer VAR/combined performance claim is made.

Why CUDA hashing cannot simply be enabled

The unchanged v0.2.0 hash module was tested on labmach's RTX A2000, Torch2.2.2+cu121/Numba0.61.2. Daniel disabled the CUDA dumper path in fec69179 with message temp fixes; no rationale is documented.

• Of 17 shape/dtype cases, 15 succeed with equal values and unchanged inputs. Scalar inputs work on CPU but fail on CUDA; an empty-row shape fails differently on both.
• In controlled nondefault-stream tests, CUDA returns stale hashes in 6/6 cases, while the CPU backend and CUDA-after-producer-synchronization controls each pass6/6. This is not safe blanket backend parity.
• Seven alternating warmed measurements on CUDA-resident float32 tensors give 49.5→125.2us for16elements (CUDA slower), 106.3→119.9us for64x64, and11.969→0.304ms for1024x1024 (39.4x faster). These include conversion/transfer/synchronization, exclude initialization/tracing/training, and do not include a future stream repair. Equal element counts can prefer opposite backends depending on shape.
• For the actual dominant MNIST weight shape (128, 9216), a separate matched default-stream hash microbenchmark measures 14.576 → 1.511 ms (9.64x) with identical results. This is a CUDA-resident input comparison, not a combined training speedup; current scalar/stream blockers remain.
• Both existing backends return only the first-row accumulator and miss changes confined to other rows. This shared correctness limitation is separate from backend switching.

No production code, backend default, instrumentation coverage, or hash algorithm changed. The actionable follow-ups are the Transformer assignment bug, CUDA stream/scalar contracts, shape-aware backend choice, and a separate decision about whole-tensor hash semantics. Exact sources, scripts, hashes, raw outcomes, timings, and limits are in the reports.

Validation and evidence

Two independent audits verify source/input/harness hashes and full raw-output parity. All 330 downloaded inference evidence files and the tracing archive's raw records were rechecked. JSON math, report links, and diff checks pass; documentation-only hooks select no runtime code. The deeper breakdown recomputes exactly from retained profiles/raw traces, checks caller-edge accounting and call completeness, and its retained calculator passes strict mypy. A separate review verifies the partition and hashing coverage. Exact sources, commands, configurations, denominator definitions, memory costs, and reproducible archive locations are recorded in the two reports and compact JSON files. Large traces and machine-specific experiment files remain outside Git.

The new model/hash studies additionally verify all77 production Python files against v0.2.0, exact MNIST loss/state parity, raw traces and executed hash fields, 17 hash shape/dtype outcomes, and 18 controlled stream cases. Diagnostic harnesses pass targeted strict mypy; hash harness formatting checks pass. Documentation/data links, arithmetic, and git diff checks pass; runtime hooks select no changed production code.

Targets main directly. The release-preparation PR #152 is already merged; production remains unchanged.