Skip to content

test(vela): keep root head checks and document the fixed L20 replay profile - #205

Draft
psv666 wants to merge 2 commits into
codex/l20-b-vela-executor-basefrom
codex/l20-b-vela-replay-supplement
Draft

psv666 wants to merge 2 commits into
codex/l20-b-vela-executor-basefrom
codex/l20-b-vela-replay-supplement

Conversation

@psv666

@psv666 psv666 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Required human contributor answers

The contributor confirmed the stage scope and these answers on 2026-11-19; the detailed confirmation is retained in the contributor's task record.

  1. Outcome: Bring assigned model work to reproducible, reviewable acceptance so supported models can be used reliably.
  2. Highest-leverage variable: Reliable model coverage and numerical consistency.
  3. How this moves it: Reuse existing implementations, resolve contract and evidence gaps, and preserve meaningful regression checks and reproducible setup.
  4. User and need: Model, application and agent developers who need stable interfaces, reliable outputs and reproducible deployment.

Is a PR needed? Yes. The contributor authorized necessary stage draft PRs and independent supplements to existing authors' work, preserving their commits.

Bug-fix evidence: This changes test placement and documentation. At #177 head, the three head-loader test bodies live under src/, contrary to CONTRIBUTING.md, and the parity-example command omits VELA_NATIVE_MANIFEST_SHA256, which its loader requires. No runtime numerical defect is claimed.

Summary

Keep Vela's head-loader regressions under root tests and document the independently reproduced L20 native-library profile with anchored release replay commands.

root head-loader checks
  + external manifest SHA in the parity example
  + fixed L20 CPU/head/response/GPU replay evidence

This is a stacked supplement to #175–#177. Its B-owned base is fixed to MrDongsls's #177 head 55ed516d302ad5bf1f5824f5ba53208236080a3b. Current head 5679f595e9ecd324326907ba913aa8f209a36a69: three files, +62/-43. Original author history is preserved. Keep draft; integration into main requires the existing stack to land and a fresh retarget/check.

Evidence

  • Before: Three private head-loader tests were implemented under src, and the parity-example command omitted its required manifest anchor. This real L20/toolkit profile lacked an independent replay.
  • After: All three checks are discovered from root tests and pass; examples supply the anchor and optimized test profile; all original 0.8B real-package checks passed on L20 with their unchanged assertions and gates.

Personally executed:

Check Result
Original #177 ordinary crate tests 63 passed, 0 failed, 5 opt-in tests ignored
Candidate discovery and ordinary crate tests All three moved tests present; same63/0/5
fmt, crate clippy, full workspace clippy/tests/release build PASS
Strict MkDocs, including latest recipe update PASS
Original token/structured-state package replay PASS
Original FP32 head replay, 5 cases PASS; max logit3.719e-5, answer7.252e-7, gates4e-5/1e-6
Original CPU response replay, 4 cases PASS; max1.490e-6, gate1.5e-6
Original GPU executor replay, 4 cases PASS; max0.3297, gate0.35

GPU stdout:

example-request: worst logit drift 0.1487
example-request-set-threshold: worst logit drift 0.0152
example-request-cjk: worst logit drift 0.3297
example-request-score: worst logit drift 0.0158
test the_gpu_backbone_replays_the_reference_readouts ... ok
test result: ok. 1 passed; 0 failed

Purpose

Resolve the root-test placement and runnable-example gaps while adding current target-hardware evidence. Related: #137, #169, #175, #176, #177.

Test Plan

  • Model/runtime/tokenizer: vllm-sr/Vela-2.0-0.8B@326b01d81f61fff7fd4ae570e54b87a49119e490. Full source shard LFS SHA256 c70237ecd579c60289c177fa9806e71c24840c2fc557b5c76af1395dc3081fa6. Original committed fixtures unchanged.
  • Export: original mixed tensors retained; uniform-BF16 backbone plus FP32 heads. External manifest anchor 1d789daa559b74a65c814b212ea5789d95d418d11c2f3ec67fa41267de9ecddb. Pre/post-export package gates pass.
  • Hardware: one NVIDIA L20 sm_89,46068MiB, driver580.178.04, GPU1 UUID GPU-8634ad01-28fd-6f5b-a574-bf1832a8dbc8; nvcc12.8.61, Rust1.100.0. Independent checkout/venv/target/export and GPU1 functional lock.
  • CUDA library built from unchanged vela: add the 0.8B native response finishing and backbone execution on CUDA #177 source, SHA256 47e918cbbaebff25d62f841c98b405e8325e80e17cb3b46d979b71de3d301db5. Release executor test binary SHA256 b9ef97773f086be82555f7ec5841d1a61082877f868e8d23b223e40f14d9bf0f.
  • CPU exporter dependencies: Python3.12.13, torch2.11.0+cpu, transformers5.19.0, tokenizers0.23.2, numpy2.5.3, safetensors0.8.0. Rust executes CUDA independently of PyTorch.
  • Real-package replay uses cargo test --release --locked as documented. All numerical override envs unset. No source, case, structural assertion or gate was changed.
  • An initial debug head replay was interrupted while CPU-bound before any case/assertion; it is retained as incomplete. Ordinary debug checks passed separately. The release replays completed; the GPU test's137.32s includes loading/verification and is not serving latency.

Test Result

This exact L20 native-library logit profile passes the original0.35 gate. CPU head/response and processing gates also pass. The code tested is the unchanged executor/backend at55ed516 plus the root-test supplement; the subsequent commit adds recipe text only.

Native HTTP, GPU end-to-end answers, positive long-window GPU parity, 4B/9B and other hardware remain separate. The original0.8B long-span head fixture has an empty expected answer; passing it does not establish a positive long-span GPU decision. Historical RTX profile descriptions remain attributed to their sources.

Demo / evidence

The raw excerpt above is this actual run. Complete stdout, interrupted debug attempt, model metadata, environment, hashes and protocol are retained in vela08-evidence.tar.gz, SHA256 9c681b1067b38abcbf1c3fa9a63a5235eea4fff9a53edd55e48f989c7665aeb7, on both contributor-local and L20 task storage. No generated run dump or archive is added to product Git.

No latency/throughput comparison or video claim. The runtime/kernel implementation is unchanged. #205's staging-base head returned no pull-request workflow run; the personally run checks above are the evidence.

Merge Danger

Door: two-way.

Blast Radius: tests.

Private test wiring and a focused recipe update only. Retarget after the existing stack lands and rerun relevant integration checks.

Self-review

Agent precheck reviewed the complete three-file supplement, registration/discovery, preserved assertions, actual profile/gates, artifact hygiene and strict docs. Human contributor checklist remains for review; no Ready/merge request.

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked that the change follows the project's architecture and stays focused on the stated purpose.
  • I have run the checks appropriate to this change and reported commands, results, and anything that I could not verify above.
  • I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

@psv666 psv666 changed the title test(vela): keep head-loader checks in root tests and anchor replay example test(vela): keep root head checks and document the fixed L20 replay profile Nov 19, 2026

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DRAFT review — PR #205 "test(vela): keep root head checks and document the fixed L20 replay profile"

Verdict: COMMENTED. The PR is in draft state (draft=true, author psv666, first-time author in this repository). Per the review policy, an explicitly draft PR receives a COMMENTED review regardless of findings; this review does not approve, request changes, or alter the draft state, and does not fill the contributor's self-review checkboxes.

Template and draft state: The repository PR template (.github/PULL_REQUEST_TEMPLATE.md: Purpose / Test Plan / Test Result / Demo-evidence / Self-review with four checkboxes) is fully instantiated in the body. All four self-review checkboxes are unchecked, which is consistent with the declared draft state ("no Ready/merge request"). The template's "System1-Omni Version / Commit" line under Test Plan is not present verbatim; base/head SHAs are stated in Summary and Test Plan instead (minor, see notes).

Pinned metadata (observed via GitHub API and local git)

  • Base: codex/l20-b-vela-executor-base @ 55ed516d302a — this is the head of open, unmerged PR #177 (verified via API). The PR is a stacked supplement; its own body says so and pins the base to MrDongsls's #177 head. Mergeable=MERGEABLE / CLEAN into the recorded base.
  • Head: codex/l20-b-vela-replay-supplement @ 5679f595e9ec (head repo InferMatrixLab — PRs live in the InferMatrixLab parent; ThinkFlowLab API resolves the same numbers). Two commits, both authored psv666 <2693925048@qq.com>; base author history untouched (verified).
  • Files: 3, +62/−43 — recipe/vela/native.md (+20/−3), src/models/vela/native/src/heads.rs (+2/−40), tests/vela/native_head_tensors.rs (new, +40).
  • Related approved work: this PR does not touch the #195 (4B/9B) test surface; it sits on the #177 (0.8B native) line, which is part of the same approved-family stack (#195's head a8ecddd and #205's head are both ancestors of #213's base, confirming the chain 177 → 205 → 210 → 213).

Body gate (post-#138 required answers)

All four mandatory human contributor answers from #138 are present and specific (Outcome / Highest-leverage variable / How this moves it / User and need), plus "Is a PR needed? Yes." and a bug-fix evidence paragraph (test placement contrary to CONTRIBUTING.md's "keep all tests under root tests/", and a parity-example command missing VELA_NATIVE_MANIFEST_SHA256; explicitly no runtime numerical defect claimed). Gate status: present and complete.

Diff review — what was removed vs moved

The root head checks are KEPT, not deleted. Verified line-by-line:

  • src/models/vela/native/src/heads.rs (−40/+2): the entire inline #[cfg(test)] mod tests block (three tests: tensor_rejects_a_shape_that_only_matches_when_flattened, tensor_rejects_non_finite_values, scalar_tensor_accepts_the_checkpoint_rank_zero_shape, plus the f32_tensor helper) is removed and replaced by #[path = "../../../../../tests/vela/native_head_tensors.rs"] mod tests; (heads.rs:332).
  • tests/vela/native_head_tensors.rs (+40): byte-for-byte the same test bodies modulo de-indentation (compared programmatically: identical names, assertions, values; only the removed 4-space module indent and a new one-line doc comment differ).
  • Path arithmetic checked: from src/models/vela/native/src/heads.rs, ../../../../../ reaches the repo root, so the include resolves to the new file. Root Cargo.toml is a virtual workspace (no [package]), so root tests/vela/*.rs is not a cargo auto-discovery directory — no duplicate-target risk. The #[path] wiring follows the crate's pre-existing convention (contract.rs:152 and render.rs:804 already include root tests/vela/*.rs this way at the base 55ed516). The three tests still run as crate unit tests via cargo test -p omni-vela-native.

Removed vs moved accounting: −43 = 40 moved test-block lines + 3 recipe command lines that were rewritten (same commands, now with --release --locked); +62 = 40 moved file + 20 recipe additions + 2 include-wiring lines. Nothing was deleted outright.

Where the L20 replay profile is documented: in recipe/vela/native.md, "Recorded hardware evidence" section, new paragraph at native.md:148–159 (head). It records: NVIDIA L20 sm_89 46,068 MiB, driver 580.178.04, nvcc 12.8.61, Rust 1.100.0; unchanged executor source at 55ed516; the pinned 0.8B package with uniform-BF16 export / FP32 heads; the release replay commands; maxima (GPU drift 0.3297 under the original 0.35 ceiling; CPU head 3.719e-5 / answer 7.252e-7; CPU response 1.490e-6) — all under unchanged gates; and explicit scope limits (native-library validation only; GPU end-to-end, long-window GPU and native HTTP remain separate). The paragraph links to the PR itself at InferMatrixLab/system1-omni/pull/205 — this is the correct repository (PRs live in the InferMatrixLab parent; verified both APIs return the same PR).

Also fixed and verified: examples/parity.rs at the base panics without VELA_NATIVE_MANIFEST_SHA256 (parity.rs:16-17), so the base recipe's parity command was genuinely broken; the PR adds the anchor (native.md:126–128) and pins replay commands to --release --locked (native.md:87,92).

Gate preservation (extends, does not weaken)

No tolerance, gate, fixture, or assertion changed; no skipped or ignored tests added; the only behavioral doc edits strengthen command pinning (--release --locked) and repair a broken command. The documented GPU ceiling stays the original 0.35. This PR maintains the approved-family test surface.

Findings

  • P2 — evidence gap: no CI run and no locally runnable Rust check for the moved test wiring. Trigger: Actions API returns total_count=0 workflow runs for head 5679f595e9ec in both ThinkFlowLab and InferMatrixLab; gh pr checks 205 reports "no checks reported on the 'codex/l20-b-vela-replay-supplement' branch". Consequence: the compile and pass of the #[path]-included test module — the entire point of this PR — has no remote verification, and cargo is unavailable in this review environment, so it was verified by inspection only (verbatim bodies + convention-matching wiring). Risk is low but nonzero (e.g., a path typo would break the crate's test build). The author discloses this ("the staging-base head returned no pull-request workflow run"). Required before ready: one CI run or a maintainer-run cargo test -p omni-vela-native at this head.
  • P3 — template's "System1-Omni Version / Commit" line absent from Test Plan; the tested base/head SHAs are recorded in Summary/Test Plan prose instead (trigger: template mismatch; consequence: automated template checks, if any, would flag it).
  • Notes (informational):
    • The "Recorded hardware evidence" paragraph cites the PR itself as the evidence anchor; raw run artifacts live off-repo (vela08-evidence.tar.gz, SHA256 9c681b10…, author-retained). Reviewer cannot inspect them; hashes are recorded for later correlation.
    • The interrupted initial debug replay is disclosed as incomplete rather than hidden — good artifact hygiene.
    • All GPU/CPU replay numbers are author-reported with detailed disclosure (GPU UUID, library/binary SHA256s, environment pins, explicit "not serving latency" caveats). Acceptable as disclosed for a draft; they are not independently verifiable by this review (no GPU).

Provenance (separated)

  • Source inspected: PR body and metadata via GitHub API; full diff 55ed516d302a..5679f595e9ec locally; heads.rs at base and head; new test file; native.md at base and head; examples/parity.rs at base; root Cargo.toml; CONTRIBUTING.md root-tests rule; #138 issue (required answers); #169/#151/#174/#177/#195/#210 states.
  • Commands run (this review, macOS, no cargo/GPU; Python 3.12.8 / pytest 8.4.1 / mkdocs 1.6.1 in miniconda): worktree at 5679f595e9ec (removed after): PYTHONPATH=src python -m pytest tests/vela -q → 141 passed; PYTHONPATH=. python -m pytest tests/benchmarks -q → 133 passed; mkdocs build --strict → exit 0, 0 warnings.
  • Observed CI: none exists for this head (0 runs in both repos, verified per SHA via Actions API on both ThinkFlowLab and InferMatrixLab).
  • Author-reported (not verified here): cargo suite counts (63 passed / 5 opt-in ignored), clippy/fmt/release build, all L20 CPU/GPU replays and drift maxima, artifact hashes.

Bottom line

A faithful, convention-following relocation that keeps all three head-loader checks and documents a disclosed L20 replay profile with unchanged gates. COMMENTED. The single readiness item before it can leave draft: obtain any CI/cargo evidence at this head for the moved Rust tests (P2 above), then retarget onto the landed #177 stack as the body itself requires.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants