Repository navigation
Conversation
hsliuustc0106
left a comment
There was a problem hiding this comment.
DRAFT review — PR #205 "test(vela): keep root head checks and document the fixed L20 replay profile"
Verdict: COMMENTED. The PR is in draft state (draft=true, author psv666, first-time author in this repository). Per the review policy, an explicitly draft PR receives a COMMENTED review regardless of findings; this review does not approve, request changes, or alter the draft state, and does not fill the contributor's self-review checkboxes.
Template and draft state: The repository PR template (.github/PULL_REQUEST_TEMPLATE.md: Purpose / Test Plan / Test Result / Demo-evidence / Self-review with four checkboxes) is fully instantiated in the body. All four self-review checkboxes are unchecked, which is consistent with the declared draft state ("no Ready/merge request"). The template's "System1-Omni Version / Commit" line under Test Plan is not present verbatim; base/head SHAs are stated in Summary and Test Plan instead (minor, see notes).
Pinned metadata (observed via GitHub API and local git)
- Base:
codex/l20-b-vela-executor-base@55ed516d302a— this is the head of open, unmerged PR #177 (verified via API). The PR is a stacked supplement; its own body says so and pins the base to MrDongsls's #177 head. Mergeable=MERGEABLE / CLEAN into the recorded base. - Head:
codex/l20-b-vela-replay-supplement@5679f595e9ec(head repo InferMatrixLab — PRs live in the InferMatrixLab parent; ThinkFlowLab API resolves the same numbers). Two commits, both authoredpsv666 <2693925048@qq.com>; base author history untouched (verified). - Files: 3, +62/−43 —
recipe/vela/native.md(+20/−3),src/models/vela/native/src/heads.rs(+2/−40),tests/vela/native_head_tensors.rs(new, +40). - Related approved work: this PR does not touch the #195 (4B/9B) test surface; it sits on the #177 (0.8B native) line, which is part of the same approved-family stack (#195's head
a8ecdddand #205's head are both ancestors of #213's base, confirming the chain 177 → 205 → 210 → 213).
Body gate (post-#138 required answers)
All four mandatory human contributor answers from #138 are present and specific (Outcome / Highest-leverage variable / How this moves it / User and need), plus "Is a PR needed? Yes." and a bug-fix evidence paragraph (test placement contrary to CONTRIBUTING.md's "keep all tests under root tests/", and a parity-example command missing VELA_NATIVE_MANIFEST_SHA256; explicitly no runtime numerical defect claimed). Gate status: present and complete.
Diff review — what was removed vs moved
The root head checks are KEPT, not deleted. Verified line-by-line:
src/models/vela/native/src/heads.rs(−40/+2): the entire inline#[cfg(test)] mod testsblock (three tests:tensor_rejects_a_shape_that_only_matches_when_flattened,tensor_rejects_non_finite_values,scalar_tensor_accepts_the_checkpoint_rank_zero_shape, plus thef32_tensorhelper) is removed and replaced by#[path = "../../../../../tests/vela/native_head_tensors.rs"] mod tests;(heads.rs:332).tests/vela/native_head_tensors.rs(+40): byte-for-byte the same test bodies modulo de-indentation (compared programmatically: identical names, assertions, values; only the removed 4-space module indent and a new one-line doc comment differ).- Path arithmetic checked: from
src/models/vela/native/src/heads.rs,../../../../../reaches the repo root, so the include resolves to the new file. RootCargo.tomlis a virtual workspace (no[package]), so roottests/vela/*.rsis not a cargo auto-discovery directory — no duplicate-target risk. The#[path]wiring follows the crate's pre-existing convention (contract.rs:152andrender.rs:804already include roottests/vela/*.rsthis way at the base55ed516). The three tests still run as crate unit tests viacargo test -p omni-vela-native.
Removed vs moved accounting: −43 = 40 moved test-block lines + 3 recipe command lines that were rewritten (same commands, now with --release --locked); +62 = 40 moved file + 20 recipe additions + 2 include-wiring lines. Nothing was deleted outright.
Where the L20 replay profile is documented: in recipe/vela/native.md, "Recorded hardware evidence" section, new paragraph at native.md:148–159 (head). It records: NVIDIA L20 sm_89 46,068 MiB, driver 580.178.04, nvcc 12.8.61, Rust 1.100.0; unchanged executor source at 55ed516; the pinned 0.8B package with uniform-BF16 export / FP32 heads; the release replay commands; maxima (GPU drift 0.3297 under the original 0.35 ceiling; CPU head 3.719e-5 / answer 7.252e-7; CPU response 1.490e-6) — all under unchanged gates; and explicit scope limits (native-library validation only; GPU end-to-end, long-window GPU and native HTTP remain separate). The paragraph links to the PR itself at InferMatrixLab/system1-omni/pull/205 — this is the correct repository (PRs live in the InferMatrixLab parent; verified both APIs return the same PR).
Also fixed and verified: examples/parity.rs at the base panics without VELA_NATIVE_MANIFEST_SHA256 (parity.rs:16-17), so the base recipe's parity command was genuinely broken; the PR adds the anchor (native.md:126–128) and pins replay commands to --release --locked (native.md:87,92).
Gate preservation (extends, does not weaken)
No tolerance, gate, fixture, or assertion changed; no skipped or ignored tests added; the only behavioral doc edits strengthen command pinning (--release --locked) and repair a broken command. The documented GPU ceiling stays the original 0.35. This PR maintains the approved-family test surface.
Findings
- P2 — evidence gap: no CI run and no locally runnable Rust check for the moved test wiring. Trigger: Actions API returns
total_count=0workflow runs for head5679f595e9ecin both ThinkFlowLab and InferMatrixLab;gh pr checks 205reports "no checks reported on the 'codex/l20-b-vela-replay-supplement' branch". Consequence: the compile and pass of the#[path]-included test module — the entire point of this PR — has no remote verification, and cargo is unavailable in this review environment, so it was verified by inspection only (verbatim bodies + convention-matching wiring). Risk is low but nonzero (e.g., a path typo would break the crate's test build). The author discloses this ("the staging-base head returned no pull-request workflow run"). Required before ready: one CI run or a maintainer-runcargo test -p omni-vela-nativeat this head. - P3 — template's "System1-Omni Version / Commit" line absent from Test Plan; the tested base/head SHAs are recorded in Summary/Test Plan prose instead (trigger: template mismatch; consequence: automated template checks, if any, would flag it).
- Notes (informational):
- The "Recorded hardware evidence" paragraph cites the PR itself as the evidence anchor; raw run artifacts live off-repo (
vela08-evidence.tar.gz, SHA2569c681b10…, author-retained). Reviewer cannot inspect them; hashes are recorded for later correlation. - The interrupted initial debug replay is disclosed as incomplete rather than hidden — good artifact hygiene.
- All GPU/CPU replay numbers are author-reported with detailed disclosure (GPU UUID, library/binary SHA256s, environment pins, explicit "not serving latency" caveats). Acceptable as disclosed for a draft; they are not independently verifiable by this review (no GPU).
- The "Recorded hardware evidence" paragraph cites the PR itself as the evidence anchor; raw run artifacts live off-repo (
Provenance (separated)
- Source inspected: PR body and metadata via GitHub API; full diff
55ed516d302a..5679f595e9eclocally; heads.rs at base and head; new test file; native.md at base and head;examples/parity.rsat base; rootCargo.toml; CONTRIBUTING.md root-tests rule; #138 issue (required answers); #169/#151/#174/#177/#195/#210 states. - Commands run (this review, macOS, no cargo/GPU; Python 3.12.8 / pytest 8.4.1 / mkdocs 1.6.1 in miniconda): worktree at
5679f595e9ec(removed after):PYTHONPATH=src python -m pytest tests/vela -q→ 141 passed;PYTHONPATH=. python -m pytest tests/benchmarks -q→ 133 passed;mkdocs build --strict→ exit 0, 0 warnings. - Observed CI: none exists for this head (0 runs in both repos, verified per SHA via Actions API on both ThinkFlowLab and InferMatrixLab).
- Author-reported (not verified here): cargo suite counts (63 passed / 5 opt-in ignored), clippy/fmt/release build, all L20 CPU/GPU replays and drift maxima, artifact hashes.
Bottom line
A faithful, convention-following relocation that keeps all three head-loader checks and documents a disclosed L20 replay profile with unchanged gates. COMMENTED. The single readiness item before it can leave draft: obtain any CI/cargo evidence at this head for the moved Rust tests (P2 above), then retarget onto the landed #177 stack as the body itself requires.
Required human contributor answers
The contributor confirmed the stage scope and these answers on 2026-11-19; the detailed confirmation is retained in the contributor's task record.
Is a PR needed? Yes. The contributor authorized necessary stage draft PRs and independent supplements to existing authors' work, preserving their commits.
Bug-fix evidence: This changes test placement and documentation. At #177 head, the three head-loader test bodies live under
src/, contrary to CONTRIBUTING.md, and the parity-example command omitsVELA_NATIVE_MANIFEST_SHA256, which its loader requires. No runtime numerical defect is claimed.Summary
Keep Vela's head-loader regressions under root tests and document the independently reproduced L20 native-library profile with anchored release replay commands.
This is a stacked supplement to #175–#177. Its B-owned base is fixed to MrDongsls's #177 head
55ed516d302ad5bf1f5824f5ba53208236080a3b. Current head5679f595e9ecd324326907ba913aa8f209a36a69: three files, +62/-43. Original author history is preserved. Keep draft; integration into main requires the existing stack to land and a fresh retarget/check.Evidence
Personally executed:
GPU stdout:
Purpose
Resolve the root-test placement and runnable-example gaps while adding current target-hardware evidence. Related: #137, #169, #175, #176, #177.
Test Plan
vllm-sr/Vela-2.0-0.8B@326b01d81f61fff7fd4ae570e54b87a49119e490. Full source shard LFS SHA256c70237ecd579c60289c177fa9806e71c24840c2fc557b5c76af1395dc3081fa6. Original committed fixtures unchanged.1d789daa559b74a65c814b212ea5789d95d418d11c2f3ec67fa41267de9ecddb. Pre/post-export package gates pass.GPU-8634ad01-28fd-6f5b-a574-bf1832a8dbc8; nvcc12.8.61, Rust1.100.0. Independent checkout/venv/target/export and GPU1 functional lock.47e918cbbaebff25d62f841c98b405e8325e80e17cb3b46d979b71de3d301db5. Release executor test binary SHA256b9ef97773f086be82555f7ec5841d1a61082877f868e8d23b223e40f14d9bf0f.cargo test --release --lockedas documented. All numerical override envs unset. No source, case, structural assertion or gate was changed.Test Result
This exact L20 native-library logit profile passes the original0.35 gate. CPU head/response and processing gates also pass. The code tested is the unchanged executor/backend at55ed516 plus the root-test supplement; the subsequent commit adds recipe text only.
Native HTTP, GPU end-to-end answers, positive long-window GPU parity, 4B/9B and other hardware remain separate. The original0.8B long-span head fixture has an empty expected answer; passing it does not establish a positive long-span GPU decision. Historical RTX profile descriptions remain attributed to their sources.
Demo / evidence
The raw excerpt above is this actual run. Complete stdout, interrupted debug attempt, model metadata, environment, hashes and protocol are retained in
vela08-evidence.tar.gz, SHA2569c681b1067b38abcbf1c3fa9a63a5235eea4fff9a53edd55e48f989c7665aeb7, on both contributor-local and L20 task storage. No generated run dump or archive is added to product Git.No latency/throughput comparison or video claim. The runtime/kernel implementation is unchanged. #205's staging-base head returned no pull-request workflow run; the personally run checks above are the evidence.
Merge Danger
Door: two-way.
Blast Radius: tests.
Private test wiring and a focused recipe update only. Retarget after the existing stack lands and rerun relevant integration checks.
Self-review
Agent precheck reviewed the complete three-file supplement, registration/discovery, preserved assertions, actual profile/gates, artifact hygiene and strict docs. Human contributor checklist remains for review; no Ready/merge request.