Skip to content

test(vela): require attributed member replays and qualify 9B on L20 - #213

Draft
psv666 wants to merge 2 commits into
codex/vela195-9b-acceptance-basefrom
codex/vela195-9b-acceptance-20261120
Draft

psv666 wants to merge 2 commits into
codex/vela195-9b-acceptance-basefrom
codex/vela195-9b-acceptance-20261120

Conversation

@psv666

@psv666 psv666 commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Require valid row attribution in the 4B/9B GPU replay before any row readout, and document the independently measured 9B reference memory and current-paired library profile.

fresh 4B/9B calls → every row is an in-range integer
  → unique row + kind + head match → every call consumed
legacy 0.8B all-unattributed calls → original kind order

This is a focused stacked supplement to #210, based on its delivered head 77b31fc2863c41713002c083140f15629a930395. It retains MrDongsls's #195 a8ecddd, accepted #177 df02f95, #208 / #205 and main f1e674f2. Original author branches are unchanged. The 4B captures in #210 already carry valid rows; this guard preserves their qualification. Retarget after the underlying stack lands and revalidate the resulting source/profile.

Required human contributor answers

The contributor explicitly confirmed this model-acceptance scope, necessary draft supplements and the fresh-member attribution guard; the detailed confirmation is retained in the contributor's stage record.

  1. Outcome: reproducible, reviewable acceptance so supported models can be used reliably.
  2. Highest-leverage variable: reliable model coverage and numerical consistency.
  3. How this moves it: reuse existing implementations, resolve contract and evidence gaps, and keep meaningful regression checks and reproducible validation.
  4. User and need: model, application and agent developers needing stable interfaces, reliable outputs and reproducible deployment.

Is a PR needed? Yes. Necessary focused draft supplements preserving other authors' commits were explicitly authorized.

Evidence

  • Before: the weight-free regression accepted a fresh member recording with every row omitted under the inherited legacy fallback semantics; actual exit 101 and failing source/log are retained. No live model numerical defect is claimed.
  • After: all 7 ordinary executor checks pass (device target ignored); missing/mixed/null/negative/fractional/string/out-of-range rows reject, valid owners pass and legacy 0.8B retains its order. Existing row/kind/head matching, unique consumption, finite/shape/count assertions and numerical gates remain.
  • Real 9B current-paired heads 5/5 pass original logit 1.6e-4 / answer 1.2e-2 gates, including exact 373 positive-window spans and Unicode offsets; maxima 3.624e-5 / 1.311e-6 (stdout).
  • Original native CPU response 4/4 passes 1.5e-2, max 7.708e-7 (stdout). A separate probe runs native FP32 candidate and span heads from frozen reference hidden inputs, then native finishing: four complete typed JSON envelopes pass, max 1.2516975402832031e-6, with usage independently summed from validated plans (1041/133/597/228).
  • Native GPU executor: one complete four-case run, no diagnostic retry:
Original case Worst logit drift (stdout) Original gate
example-request 0.2521 0.35
example-request-set-threshold 0.0166 0.35
example-request-cjk 0.1753 0.35
example-request-score 0.0177 0.35

Purpose

A new member capture that loses every row must fail closed rather than inherit the historical 0.8B call-order fallback. The memory documentation now distinguishes FP32 parameter residency and autocast caches/workspace from checkpoint or uniform-BF16 export size. No production forward/scoring, CUDA/ABI or root manifest change is introduced by this supplement.

Test Plan

Head: 422e74f8b3030d5cf766cf3ce4ee281d44691d58, tree 957b4bb97a52c42423fe2829891d4fbf12edd09d. Numerical Rust code is the validated 3a8e054089196a16f264cf29d31aac878a45e98f; subsequent source changes are documentation only.

From repository root (use the archived current-paired replay view for 9B opt-ins):

cargo test --release --locked -p omni-vela-native --test vela_native_executor
cargo test --release --locked -p omni-vela-native --test vela_native_processing -- --ignored --nocapture
cargo test --release --locked -p omni-vela-native --test vela_native_heads -- --ignored --nocapture
cargo test --release --locked -p omni-vela-native --test vela_native_response -- --ignored --nocapture
cargo test --release --locked -p omni-vela-native --test vela_native_executor -- --ignored --nocapture
cargo fmt --all --check
cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings
cargo test --workspace --locked --features omni-laya/serve
cargo build --workspace --release --locked --features omni-laya/serve
PYTHONPATH=src python -m pytest tests/vela -q
mkdocs build --strict

Test Result

Personally executed on L20: ordinary Vela 77 passed / 5 ignored; workspace 504 passed / 126 ignored; Vela Python 168 passed; all Rust CI commands and strict docs pass. Original 9 row-cap regressions pass in debug and release. Real 9B processing 2/2 covers all five plans plus structured-state tokens; the recorder independently verifies fresh plans equal those expectations. The actual tokenizer admits final 16383/16384-token rows and rejects the next row with whole-request 413/too_long. No workspace-wide ignored run or unrelated model GPU campaign.

Demo / evidence

Package vllm-sr/Vela-2.0-9B@068425bb145c05884a98b870c219047da22d9aea. All 46 official-revision files, Git blobs and 15 LFS pointers/content hashes verified; all 426 faithful/rounded-BF16 backbone tensors, 37 FP32 heads, tokenizer/calibration/config and exact bundle file set checked. Manifest anchor 2b57dd1bd9ee2b10d60fa8ff0567d13a45cb411fd67e739c5589c21cd05688c2.

Reference: Python 3.12.13, torch 2.11.0+cu128, transformers 5.19.0, tokenizers 0.23.2, numpy 2.5.3, safetensors 0.8.0. All FP32 parameters, BF16-autocast backbone and FP32 heads; default batch 32 / token budget 24576, highest float32 matmul precision / CUDA matmul TF32 off, Transformers torch GDN fallback. Nine real reference TestClient versus production Python-worker cases match status/body/key order; all-kind warmup observes candidate/router/broad heads. Each of the five captures traces exactly one _predict, including batches and same-run typed oracles. JSON+tensors frozen before any native numerical comparison; frozen manifest SHA256 e1bb8eb7e8a8e97e14d837e084f5a1efd09fbacc38d0dfdb83d4b28fa86a0f40 remains unchanged afterwards.

Reference PyTorch peak allocated 44305.62 MiB / peak reserved 45026 MiB; CUDA free after the long-window capture 79.81 MiB. This fixed run completes with very little headroom; allocator figures exclude some driver/library allocations and do not establish general capacity or full 16K inference support.

Native: one L20 sm89, driver 580.178.04, Rust 1.100.0 / CUDA 12.8.61, uniform BF16 backbone / FP32 heads / individual fixed-M64 readouts, Graph/opt-ins unset. Library 63f3f5199420a22dec2976899939e6a749ef998202bad26fcf8928545b95235b reused read-only after actual C/CUDA/loader source, symbols, ABI 9, sm89 and default-profile match. Actual process maps, /proc/exe hashes and GPU UUID samples bind execution to this private library/GPU1.

Raw commands/exits, original failures, frozen pairs, historical deltas, typed responses, source/binary/package/library/env hashes and reproduction helpers are retained off-repository in vela195-9b-evidence.tar.gz, SHA256 0d6f006a8aab9d8ae150f0417416f1141ab4e584044dec33923e6780fefcf6d8 (67,364,164 bytes; 410 members). The contributor retains the artifact. No generated run dump or new binary fixture is added to product Git; all original #195 fixtures remain unchanged.

Historical exact replay remains unrun because original paired 9B tensors/hashes were unavailable. Fresh and historical long-window captures both have 373 spans but some positions differ; the archived delta is a reference-run difference, not a native failure. Current native comparisons use the frozen fresh pairs throughout.

CPU typed responses use reference-backbone hidden inputs. Complete native GPU typed answers, native HTTP/readiness/frontend/lifecycle, native positive-window GPU, full-length reference inputs and other hardware remain unrun. No performance claim; test wall time includes repeated full-bundle verification. Video/performance A/B: N/A for this harness/docs supplement.

Merge Danger

Door: two-way.

Blast Radius: validation.

Damaged member metadata is rejected earlier; valid finite-input production arithmetic is unchanged. This narrow qualification does not establish family-wide readiness.

Self-review

The focused complete diff, callers, ownership, original gate preservation, observed red/green failure and artifact hygiene were checked. No remaining blocking finding in this supplement. The inherited main-to-stack diff still exceeds 3,000 authored-code lines and requires full contributor/maintainer review. Keep draft; this is personally executed evidence, not maintainer approval or candidate CI (main-only workflow filters may skip this stacked base).

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked architecture and scope.
  • I have run appropriate checks and reported unverified checks.
  • I have checked that claims match implementation and evidence.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DRAFT review — PR #213 "test(vela): require attributed member replays and qualify 9B on L20"

Verdict: COMMENTED. The PR is in draft state (draft=true, author psv666, first-time author). Per policy, an explicitly draft PR receives a COMMENTED review regardless of findings; this review does not approve, request changes, or alter the draft state, and does not fill the contributor's self-review checkboxes.

Template and draft state: All required sections present (Summary, Required human contributor answers, Evidence, Purpose, Test Plan, Test Result, Demo/evidence, Merge Danger, Self-review). All four self-review checkboxes are unchecked, consistent with draft. Note: checkbox texts for items 2 and 4 are abbreviated paraphrases of the template's exact wording ("I have checked architecture and scope." / "I have checked that claims match implementation and evidence.") — same substance, but the canonical template sentences are not used verbatim (P3 note below).

Pinned metadata (observed via GitHub API and local git)

  • Base: codex/vela195-9b-acceptance-base @ 77b31fc2863c — verified to be the head of draft PR #210 ("fix(vela): preserve row ownership in decoder executor replay"). Stacked directly on top of the approved-family work: ancestry of the base verified for #195's head a8ecddd (the approved 4B/9B PR), #177's accepted head df02f95, #205's head 5679f595, #208's head 1d460564, and main f1e674f2 — exactly as the body claims. Mergeable=MERGEABLE / CLEAN into the recorded base.
  • Head: codex/vela195-9b-acceptance-20261120 @ 422e74f8b303; head repo InferMatrixLab (PRs live in the InferMatrixLab parent). Two commits, both authored psv666 <2693925048@qq.com>; underlying author branches unchanged (bases verified).
  • Files: 3, +117/−4 — tests/vela/native_executor.rs (+49), recipe/vela/native.md (+67/−3), src/models/vela/README.md (+1/−1). No production source, CUDA/ABI, or manifest changes — verified by diffstat.

Body gate (post-#138 required answers)

All four mandatory answers present and specific; "Is a PR needed? Yes." present. Gate status: present and complete (checkbox wording deviation noted below).

What "attributed member replays" enforces

A fail-closed call-attribution guard in the GPU replay test, inserted before any row readout (tests/vela/native_executor.rs:119):

  • New helper validate_call_rows (native_executor.rs:232–243), called as validate_call_rows(&calls, planned.rows.len(), m != "Vela-2.0-0.8B"). For any member other than the legacy 0.8B, every recorded executor call must carry an integer "row" with 0 ≤ row < planned_row_count; missing, mixed (some rows present, some absent), null, negative, fractional, string, and out-of-range values all fail the assertion before any row is read out or matched.
  • The legacy 0.8B fallback is deliberately preserved: if no call carries any row key, the historical kind-order matching still passes; but a mixed legacy recording (some attributed, some not) is rejected — the any(...is_some) trigger makes the all-calls rule apply. The rationale is documented in code comment and in recipe/vela/native.md:108–112 ("Only the original 0.8B recordings may omit all row metadata…; this compatibility does not apply to a fresh 4B/9B capture").
  • A dedicated weight-free unit test fresh_member_replays_require_every_call_to_identify_a_valid_row (native_executor.rs:345–377) exercises the helper directly across 4B/9B members with each invalid shape, valid-ownership pass, and the legacy all-unattributed pass. This closes the disclosed red state ("the weight-free regression accepted a fresh member recording with every row omitted under the inherited legacy fallback") — that before/after demonstration is author-reported (exit-101 log retained off-repo in vela195-9b-evidence.tar.gz, SHA256 0d6f006a…).

This strengthens the replay gate; existing row/kind/head uniqueness, finite/shape/count assertions, and all numerical ceilings are untouched (verified in diff).

The 9B-on-L20 qualification: test gate or recorded run?

A recorded run, not a test gate. The only code change is the attribution guard above. The 9B qualification lives in documentation: a new ### Independent L20 9B library profile section in recipe/vela/native.md:210+ recording: the pinned 9B package vllm-sr/Vela-2.0-9B@068425bb…; FP32-resident parameters (31.8 GB) with BF16-autocast backbone and FP32 heads; nine reference-vs-worker exact cases; five head captures hash-frozen before native comparison (373-span positive window); FP32-head maxima 3.624e-5 / 1.311e-6 against the original 1.6e-4 / 1.2e-2 ceilings; response finishing 7.708e-7 at 1.5e-2; the CPU hidden-input probe 1.252e-6; and a one-run four-case GPU logit replay (worst 0.2521) against the original 0.35 gates, no retries or overrides. All GPU numbers are author-reported and disclosed: one L20 sm_89, driver 580.178.04, private library SHA256 63f3f519… with process-map/GPU-UUID binding claims, evidence archive SHA256 + byte size + member count, and an explicit unverified list (historical exact replay unrun because original paired tensors were unrecoverable; full-length inputs, native positive-window GPU, complete native GPU typed answers, HTTP lifecycle, other hardware). Memory figures are properly caveated ("exclude some driver/library allocations and are not a general capacity guarantee"). Acceptable as disclosed for a draft; none of it is independently verifiable by this review (no GPU), and no CI validates the numbers.

The README row for 9B (src/models/vela/README.md:37) is corrected from an understated "~18 GB + FP32 heads" to "~31.8 GB FP32 parameters plus caches/workspace; measured L20 peak reserved 45,026 MiB" with a working relative link to the recipe anchor — a conservative correction that aligns docs with the recorded measurement.

Gate preservation vs approved #195 (extends, does not weaken)

Verified ancestry: #195's approved head is in this PR's base, and the diff only makes the fresh-member replay stricter (attribution required) while preserving legacy behavior explicitly and documented. No tolerance loosened, no gate constant changed, no skipped or ignored tests added (the --ignored GPU opt-ins are pre-existing convention). The strict-4B/9B evidence wording replaces an over-broad "remains unverified until rerun" sentence with an accurate split (historical unverified; current-paired 4B in #210, 9B below) — clarification, not weakening. Extends the approved surface.

Findings

  • P2 — no CI run for the head in either repository; Rust-side effect unverified remotely and not runnable in this review. Trigger: Actions API total_count=0 for 422e74f8b303 in both ThinkFlowLab and InferMatrixLab; gh pr checks 213 reports no checks. Consequence: compile and behavior of the new guard + unit test at this head rest on author-reported runs only (body: ordinary 77/5, workspace 504/126, clippy/fmt/release all pass; the "168 Vela Python passed" claim was independently reproduced here). The body discloses that main-only workflow filters may skip this stacked base. Required before ready: one CI run (or maintainer-run cargo suite) at the eventual head.
  • P3 — self-review checkboxes 2 and 4 deviate from the canonical template sentences. Trigger: body uses abbreviated paraphrases. Consequence: template-checking tooling and the #138 gate expect the canonical four questions' checklist verbatim; restore the template text before marking ready.
  • P3 — stacked on a draft (#210). Trigger: base 77b31fc2863c is #210's head, and #210 is draft=true (verified). Consequence: this PR cannot land until #210 and the rest of the stack land; a retarget and revalidation of source/profile will be required, as the body itself states. Not a defect of this diff — a sequencing constraint reviewers should not lose.
  • Notes (informational):
    • recipe/vela/native.md:27 cites draft PR #210 as the current-paired 4B evidence anchor; draft-PR evidence can change or vanish — fine for now, re-anchor at merge time.
    • The unit test's std::panic::catch_unwind style is acceptable; assertion-based helpers would read cleaner. No action required.
    • The "Before" red demonstration (all-unattributed fresh-member capture accepted) is a valuable negative control; the retained log is off-repo, so its content is trusted from the body.

Provenance (separated)

  • Source inspected: PR body/metadata via API; full diff 77b31fc2863c..422e74f8b303; native_executor.rs at base and head; native.md and README.md at base and head; #210/#195/#177/#205/#208 states and heads; #138 issue.
  • Commands run (this review, macOS, no cargo/GPU; Python 3.12.8 / pytest 8.4.1 / mkdocs 1.6.1): worktree at 422e74f8b303 (removed after): PYTHONPATH=src python -m pytest tests/vela -q → 168 passed (independently reproduces the body's "Vela Python 168 passed"); PYTHONPATH=. python -m pytest tests/benchmarks -q → 133 passed; mkdocs build --strict → exit 0, 0 warnings.
  • Observed CI: none exists for this head (0 runs in both repos, verified per-SHA via Actions API on both ThinkFlowLab and InferMatrixLab).
  • Author-reported (not verified here): all cargo suite counts and clippy/fmt/build results; every 9B/0.8B L20 CPU and GPU replay number and gate comparison; the red/green attribution demonstration; memory and allocator figures; all artifact/library/bundle hashes.

Bottom line

A genuinely strengthening supplement: fail-closed row attribution for fresh 4B/9B replays with the legacy 0.8B fallback explicitly scoped, plus a disclosed (but unverifiable here) recorded 9B L20 profile with unchanged original gates and a corrected memory table. COMMENTED. Before ready: obtain CI or maintainer-run cargo evidence at the head (P2), restore canonical checklist wording (P3), and re-stack after #210 and the underlying work land (P3).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants