Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Frozen invariants need to be checked in fresh training processes with measured predicate execution: zero reported violations alone can hide dormant or skipped checks. This adds a reproducible, bounded CPU harness that learns from a 10-step reference, then checks unchanged rules against a fresh same-seed control and a 20-step different-seed holdout.
The validation found a real portability failure. Two Transformer APIContain rules save
_TRAINCHECK_src_mask_IDliterally in their child-argument selectors. The required child calls are present in the fresh runs with otherwise equal arguments, but the new object IDs cause false alarms—even in the same-seed control with identical losses and model parameters. This PR preserves and reports that failure; it does not change production inference or checking semantics.The harness uses signed, explicitly composed sources based on #144 with the tracing, TorchScript, precondition-state, and type-name fixes (#142, #143, #145, #146). Each phase has a fresh process, one native thread, and sampled timeout/memory/trace budgets. Production code is unchanged in this PR.
Results cover these two small CPU workloads and one reference/holdout seed pair each. Their new rule sets have no PID/TID precondition clauses, so they do not validate the original TorchTitan corpus's three literal-thread-ID rules. Frozen IDs and predicates are never normalized to make checks pass.
Reproduction, limitations, failed-attempt history, and per-rule decision ledgers are in
docs/measurements/fresh-run-invariant-validation.mdanddocs/measurements/data/fresh-run-invariant-validation*.Engineering hardening
Signed code checkpoint
f3804c0types all eight reusable harness modules: incomplete function signatures decrease 36 → 0. Required JSON fields and family-specific observer identities are validated; completed/observed reports undergo conditional schema validation. Three controlled source-drift cases change from 3 false completions with null identities → 0. A further regression preserves an observer failure even when the checker catches it and then raises a different error.Coverage reports now declare schema version 2. Disabled, unsupported, and failed observation remain explicit; unknown coverage or checker outcomes produce
nullaggregate totals, with observed-subset counts retained. These are deliberate report-contract changes. The sole mapping cast follows runtime key validation; there are no newAnyannotations or type ignores.All 28 targeted tests pass locally and on the updated Linux production composition. Pinned Mypy 1.9 passes all eight modules plus the new typed test module with strengthened checks and actual Torch type resolution; configured hooks and diff checks pass. Independent review found no remaining blocker.
The updated production composition also passes 140 production regressions. Eight frozen MNIST/Transformer reference/control/holdout/fault replays preserve every original semantic field across 3,072 per-rule reports, including raw trace identities, decision arrays, checker payloads, and summary counts. The two Transformer healthy false alarms remain visible. Source/input hashes remain unchanged; the expected production import path is verified. This validates integration in an isolated snapshot while retaining this PR's branch history; it does not rerun training or establish new performance/generalization results.
Whole-production Mypy, Ruff, and isort pass. Whole-source Black retains three unchanged legacy failures; changed-file hooks pass. Strict checks remain scoped. See
docs/measurements/fresh-validation-contract-hardening.mdand its linked before/after and integration evidence for exact source revisions, commands, schema compatibility, and limitations.Integration with current main
Based on #144 (
codex/inference-identity-cache), with signed merge checkpoint62b829ed8a4c. The shared base is #150, which reconciles Daniel's parallel-inference tip122e769with mained54c7d. History is preserved; no force push or GitHub PR merge was used. This branch now includes the actual current #144 ancestry, including its contract hardening. All nine PR heads merge cleanly with their declared bases and current main, and the full stack composes without conflicts.The resolved composition passes 140 production + 28 harness tests locally (Python 3.10.16 / Torch 2.2.2), pinned Mypy over 77 production files, and stricter Mypy over 9 harness/test files. Conflict-file commit hooks pass. Full-stack Ruff/Mypy/isort pass; Black retains the same three pre-existing formatting failures (
config.py,proxy_wrapper/hash.py,proxy_wrapper/subclass.py). Remote CI is ignored because the self-hosted runner is unavailable. These checks validate integration; existing performance measurements retain their original revisions and are not remeasured here.Land the parent in main first, then retarget this PR to main before merging it. Repeat for its children. Use merge commits to preserve stack ancestry, or refresh descendants after a squash/rebase merge. Exact heads and local logs are retained in
outputs/conflict-resolution-20261026/.