Skip to content

Fix recorder re-entry and atomic trace buffer delivery - #142

Merged
Essoz merged 4 commits into
codex/parallel-infer-integrationfrom
codex/trace-integrity
Oct 26, 2026
Merged

Essoz merged 4 commits into
codex/parallel-infer-integrationfrom
codex/trace-integrity

Conversation

@Essoz

@Essoz Essoz commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Recorder-owned conversion and encoding could re-enter instrumented APIs, producing extra records or recursion. Concurrent buffer flushes could emit the same batch twice or clear a newly appended record before delivery.

Use an exception-safe thread-local suppression depth around recorder work, releasing it for application calls and generator execution. Keep mutable snapshots synchronous and lock buffer append, queue handoff, and clear. The class context manager also preserves frozen framework exceptions.

Controlled measurements using the actual wrapper/buffer implementations:

Trigger Before After
100 application calls with an injected serialization helper 400 records 200 records
Overlapping flushes, 50 trials × 100 records 5,000 duplicate records 0 duplicates
Append during paused flush, 50 trials 50 missing records 0 missing

These are forced triggers, not occurrence-rate estimates. All 45 ordinary-tensor workload runs completed; the 36 instrumented MNIST/Transformer/DCGAN runs showed no trace-size change, and recorded losses/final model-state hashes matched their baselines. No general performance improvement or invariant-checking overhead claim is made.

Validation: 11 trace-integrity regression tests and repository-pinned Ruff, Mypy, Black, and isort hooks. The tests preserve nested application calls, generator calls, exception identity, real tensor snapshots, thread-local suppression, and buffer delivery order. docs/experiments/trace-integrity.md and its compact JSON retain scope, counts, runtime settings, and evidence hashes.

Engineering hardening

Typed per-thread suppression state, context-manager exception/traceback handling and Literal[False] exit results, plus queue/buffer/record contracts (aba4ee8). Four additional tests cover independent threads, exception restoration, enqueue-failure retention, and serialization failure. All 15 tracing tests and pinned hooks pass; stronger Mypy 1.9 checks cover the new contract tests. Legacy serializer/proxy bodies remain outside a full strict-typing migration.

Separately composed updated production sources pass 140 affected Linux regression tests, whole-source Mypy (77 files), Ruff, and isort. Whole-source Black still reports three unchanged legacy files, down from six in the original base; changed-file hooks pass. Strict checks are scoped and do not establish repository-wide strict typing. Historical performance numbers above retain their original measured revisions; this hardening pass does not re-estimate them.

Integration with current main

Based on #150 (codex/parallel-infer-integration), with signed merge checkpoint 094182d5112c. The shared base is #150, which reconciles Daniel's parallel-inference tip 122e769 with main ed54c7d. History is preserved; no force push or GitHub PR merge was used. The overlapping import conflict retains this PR's complete typed suppression implementation and the upstream shutdown guard. All nine PR heads merge cleanly with their declared bases and current main, and the full stack composes without conflicts.

The resolved composition passes 140 production + 28 harness tests locally (Python 3.10.16 / Torch 2.2.2), pinned Mypy over 77 production files, and stricter Mypy over 9 harness/test files. Conflict-file commit hooks pass. Full-stack Ruff/Mypy/isort pass; Black retains the same three pre-existing formatting failures (config.py, proxy_wrapper/hash.py, proxy_wrapper/subclass.py). Remote CI is ignored because the self-hosted runner is unavailable. These checks validate integration; existing performance measurements retain their original revisions and are not remeasured here.

Land the parent in main first, then retarget this PR to main before merging it. Repeat for its children. Use merge commits to preserve stack ancestry, or refresh descendants after a squash/rebase merge. Exact heads and local logs are retained in outputs/conflict-resolution-20261026/.

@Essoz
Essoz changed the base branch from daniel/parallel-infer to codex/parallel-infer-integration October 26, 2026 08:48
@Essoz
Essoz merged commit cf93345 into codex/parallel-infer-integration Oct 26, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant