Skip to content

Add fresh-run validation and reproduce object-ID false alarms - #148

Open
Essoz wants to merge 7 commits into
codex/inference-identity-cachefrom
codex/fresh-run-invariant-validation
Open

Essoz wants to merge 7 commits into
codex/inference-identity-cachefrom
codex/fresh-run-invariant-validation

Conversation

@Essoz

@Essoz Essoz commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Frozen invariants need to be checked in fresh training processes with measured predicate execution: zero reported violations alone can hide dormant or skipped checks. This adds a reproducible, bounded CPU harness that learns from a 10-step reference, then checks unchanged rules against a fresh same-seed control and a 20-step different-seed holdout.

The validation found a real portability failure. Two Transformer APIContain rules save _TRAINCHECK_src_mask_ID literally in their child-argument selectors. The required child calls are present in the fresh runs with otherwise equal arguments, but the new object IDs cause false alarms—even in the same-seed control with identical losses and model parameters. This PR preserves and reports that failure; it does not change production inference or checking semantics.

Workload Learned / exercised rules Reference violations Fresh same-seed violations Longer changed-seed violations
MNIST, 1,199,882 parameters 204 / 204 0 0 0
Small Transformer, 36,197 parameters 564 / 564 0 2 2
  • No dormant rules, skipped requested proxy checks, or checker errors in these runs. Missing-value skips remain explicit: MNIST holdout has 3,880 IO opportunities and 3,760 actual comparisons (120 skipped).
  • Complete loss trajectories and initial/final canonical parameter hashes match for reference/control and instrumented/plain holdout. Observed and direct checker outputs match exactly.
  • One-field trace-metadata controls exercise actual detection: MNIST adds three violations; Transformer adds two, for four total including its two existing false alarms. These are post-hoc sensitivity controls, not reproduced training bugs.
  • Thirteen focused observer/mutation tests pass locally and on the matching Linux runtime; 29 integration tests pass on the composed production stack. Independent audits verify raw artifacts, source/input hashes, counters, mutations, and the object-ID diagnosis.

The harness uses signed, explicitly composed sources based on #144 with the tracing, TorchScript, precondition-state, and type-name fixes (#142, #143, #145, #146). Each phase has a fresh process, one native thread, and sampled timeout/memory/trace budgets. Production code is unchanged in this PR.

Results cover these two small CPU workloads and one reference/holdout seed pair each. Their new rule sets have no PID/TID precondition clauses, so they do not validate the original TorchTitan corpus's three literal-thread-ID rules. Frozen IDs and predicates are never normalized to make checks pass.

Reproduction, limitations, failed-attempt history, and per-rule decision ledgers are in docs/measurements/fresh-run-invariant-validation.md and docs/measurements/data/fresh-run-invariant-validation*.

Engineering hardening

Signed code checkpoint f3804c0 types all eight reusable harness modules: incomplete function signatures decrease 36 → 0. Required JSON fields and family-specific observer identities are validated; completed/observed reports undergo conditional schema validation. Three controlled source-drift cases change from 3 false completions with null identities → 0. A further regression preserves an observer failure even when the checker catches it and then raises a different error.

Coverage reports now declare schema version 2. Disabled, unsupported, and failed observation remain explicit; unknown coverage or checker outcomes produce null aggregate totals, with observed-subset counts retained. These are deliberate report-contract changes. The sole mapping cast follows runtime key validation; there are no new Any annotations or type ignores.

All 28 targeted tests pass locally and on the updated Linux production composition. Pinned Mypy 1.9 passes all eight modules plus the new typed test module with strengthened checks and actual Torch type resolution; configured hooks and diff checks pass. Independent review found no remaining blocker.

The updated production composition also passes 140 production regressions. Eight frozen MNIST/Transformer reference/control/holdout/fault replays preserve every original semantic field across 3,072 per-rule reports, including raw trace identities, decision arrays, checker payloads, and summary counts. The two Transformer healthy false alarms remain visible. Source/input hashes remain unchanged; the expected production import path is verified. This validates integration in an isolated snapshot while retaining this PR's branch history; it does not rerun training or establish new performance/generalization results.

Whole-production Mypy, Ruff, and isort pass. Whole-source Black retains three unchanged legacy failures; changed-file hooks pass. Strict checks remain scoped. See docs/measurements/fresh-validation-contract-hardening.md and its linked before/after and integration evidence for exact source revisions, commands, schema compatibility, and limitations.

Integration with current main

Based on #144 (codex/inference-identity-cache), with signed merge checkpoint 62b829ed8a4c. The shared base is #150, which reconciles Daniel's parallel-inference tip 122e769 with main ed54c7d. History is preserved; no force push or GitHub PR merge was used. This branch now includes the actual current #144 ancestry, including its contract hardening. All nine PR heads merge cleanly with their declared bases and current main, and the full stack composes without conflicts.

The resolved composition passes 140 production + 28 harness tests locally (Python 3.10.16 / Torch 2.2.2), pinned Mypy over 77 production files, and stricter Mypy over 9 harness/test files. Conflict-file commit hooks pass. Full-stack Ruff/Mypy/isort pass; Black retains the same three pre-existing formatting failures (config.py, proxy_wrapper/hash.py, proxy_wrapper/subclass.py). Remote CI is ignored because the self-hosted runner is unavailable. These checks validate integration; existing performance measurements retain their original revisions and are not remeasured here.

Land the parent in main first, then retarget this PR to main before merging it. Repeat for its children. Use merge commits to preserve stack ancestry, or refresh descendants after a squash/rebase merge. Exact heads and local logs are retained in outputs/conflict-resolution-20261026/.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant