Repository navigation
Conversation
Align patch, LayerNorm, rotary, head-64 attention and vision GEMM boundaries with the pinned BF16 reference. Keep legacy/JEMM behavior and optional ABI compatibility. The original 280-answer whole-model precision gate still fails; this is the bounded vision stage for InferMatrixLab#114.
4 tasks
hsliuustc0106
left a comment
Contributor
There was a problem hiding this comment.
Review draft — #212 "omnijev: add grouped reference LM features and all-type execution"
Verdict: COMMENTED. The PR is OPEN and explicitly DRAFT; first review, not approval or completed validation. Contributor checklist left unchecked, correctly.
Provenance and pins
- Base:
main@f1e674f2d324f5bb4e1ba2162551d0b633f20be0(matches GitHub and body). - Head:
0e69d34cb3e30c51d1bbd76477bb63204b3bed40(matchesorigin/pr/212); treeb153b1f848dcfd048934d0841c6d03a56ca67f22— verified. - Diff vs base: 31 files, +2307/−52 = 2359 changed lines — matches the body's "main-based stack is 31 files / 2359 changed lines". Incremental vs #211: 14 files, +951/−44 = 995 — matches the body (body says "798 authored-code" within that; plausible from numstat, not separately re-derived).
- Commit history honesty check: the body discloses that the corpus numbers were measured at validation commit
e28256254…whose tree was retained as published commit46c1ae5c— verified:git rev-parse 46c1ae5c^{tree}=b7c0e36c87a4ecb33baa50eb1602018b0e963d24, exactly the tree the body pins for the validation commit. The final commit0e69d34ctouches only markdown docs plus an 18-line test-only adjustment intests/omnijev/worker.rs— verified (git diff --stat 46c1ae5c..0e69d34cb), so "unchanged inference source" holds.
Position in the series (cross-reference)
Tip of the verified OmniJev stack 209 ⊂ 211 ⊂ 212 (first-parent chain 6410e763 → 72af54fb → 46c1ae5c → 0e69d34c from base f1e674f2). #215 is not a descendant — see the #209/#215 bodies. Review order applied: #209 → #211 → #212 → #215.
Template gates
- Four contributor answers restated with links to #211/#209; necessity answered; draft decision explicit ("keep this PR draft and #114 open").
- Checklist: unchecked, correct.
- Evidence pins verified as above; artifact hashes (export manifest
f5ffba7e…, libraryb1f08a21…) are author-reported and consistent with #211's pins (library unchanged across the stage, as claimed).
Findings
No P0/P1/P2 findings.
- Grouping semantics:
processing::reference_lm_chunks(src/models/omnijev/native/src/processing.rs:109-172at head) validates each row's token/target layout exhaustively (zq/u/target positions,OPTION_OPEN/OPTION_CLOSEsentinels, token < 248,079,first == targets.first(), overflow-checked) and chunks the complete request's rows into the official default 64-row groups (MSO_BRANCH_CHUNK=64documented in the doc comment). Target budget: the whole request is bounded toMAX_TARGETS = 16,384including Score rows in this profile — enforced in CPU preparation (Processor::load_one_shot_reference+ prepare hook), so over-budget requests fail before model work or HTTP dispatch; the body correctly calls this a documented restriction relative to the merged profile's Score-exempt budget, surfaced as 422 (verified in the worker tests' status expectations and theAppError-style status mapping inmain.rs). - All-type dispatch:
Executor::run_referenceexecutes every row as an independent full-row reference forward, collects ownedBf16HiddenRowsper chunk, then performs exactly one grouped option-token projection and one grouped first-token projection per chunk viaModel::reference_lm_logprobs; finishing dispatches per question kind — Score →heads.ordinal, Choice/Noul →heads.option_probabilitieswithheads::lm_featurescomputed per question group (at..at + rows.len()slicing preserves question boundaries across chunk edges; log-softmax normalization stays per question after all chunks execute). - Feature isolation between request types: selected hidden is owned CPU BF16 (
Bf16HiddenRows, finite-checked, no device pointers, survives subsequent forwards — tested through scratch growth inreference_selected_rows_preserve_full_row_and_recoverand the whole-request recovery test); Score keeps its LM rows in the normative gather but consumes only zq/u; Choice/Noul features are computed from each question's own rows' picked/first values only. - Qwen seam respects ownership:
reference_lm_logprobs(src/models/qwen3_5/native/src/model.rs:2040-2149at head) executes the caller-declared ordered group as one full-vocabulary GEMM over the tied embedding (duplicates retained, never split at 64), with overflow-checked byte extents, buffer growth that synchronizes before replacing allocations, "no padding enters the denominator" (BF16 logits stride = actual vocab), post-hoc finite/≤0 checks, and synchronize on success and error. The CPU FP64token_logprobreduction is explicitly declared approximate to reference FP32, not bit exact — the compensated-sum lesson is respected (measured approximation declared as such: log-prob max 9.5e-7, features 1.5e-5, heads 2.4e-7 measured; test bounds 1e-5/2e-5/1e-5 sit just above the measurements). - Worker profile:
OMNIJEV_PROFILE(merged-fixeddefault |unmerged-reference| bail) selects at startup (main.rs:163-171); default worker behavior unchanged; all-type warmup runs before readiness with the original complete request as first post-health request (tested). - Honest corpus accounting: 179/280 within gate (125/140 PNG, 54/140 JPEG), 3 protected JPEG flips, max returned drift 0.1878 — internally consistent (125+54=179) and consistent with #209's 120/140 PNG-within baseline (improvement attributable to the Score/LM stage, as expected). The gate fails; draft retained; #114 open.
- No CUDA/ABI changes: no
.cu/ops.h/Cargo/lockfile/root-manifest changes in the delta (verified). - Informational:
MAX_REFERENCE_LM_ROWS(qwen crate) andMAX_TARGETS(omnijev crate) are two constants with the same value 16,384 and different roles (projection allocation bound vs preparation budget) — consistent today; a near-budget request would attempt a very large (multi-GB) logits allocation before failing on allocation rather than 422; acceptable given real requests are two orders of magnitude smaller, worth remembering if budgets grow. The 64-row chunk constant mirrors the pinned official default; the author's negative control (old fixed/64-target grouping value-exact on actual hidden) is explicitly not generalized into a row-independence claim — correctly scoped.
Evidence provenance
- Source inspected: full incremental diff
72af54fbc..0e69d34cb(14 files) plus the tree-retention check46c1ae5c..0e69d34cb;processing.rs,executor.rs, qwenmodel.rs,main.rsin depth; registration docs (README,docs/supported-models.md,docs/architecture.md, recipe READMEs) read in full. - Commands run (personally): pins/tree verification; incremental and cumulative diff stats; validation-commit tree retention check; binary scan (none); status-mapping spot check; docs-vs-code cross-check of budget, chunking, and profile names.
- Observed CI (both repo hostnames, exact head_sha
0e69d34cb…):CIsuccess (44689245471),Docssuccess (44689245470). Fork repo: 0 runs. - Author-reported: all L20 measurements (official capture shapes 112×2560 / 31×2560, same-hidden logits/features/heads deltas, 280-answer table, negative control, 62 corpus hashes, worker HTTP behaviors including 400/422/413 and recovery, warmup/readiness ordering).
Unverified checks / evidence gaps
- No GPU and no cargo locally: the six named L20 checks, the complete-request HTTP worker behaviors, and the 280-answer replay are author-reported. The full numerical gate fails (179/280) and the PR says so; nothing here closes #114 or qualifies serving readiness.
- The internal first-difference layer trace and branch-topology/JPEG repair are explicitly unrun (body). These remain the binding gaps for #114 acceptance, correctly deferred.
4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The explicit unmerged profile previously rejected complete requests and LM targets, leaving Choice/Noul unavailable. Add owned BF16 selected hidden, caller-declared LM projection groups, complete Choice/Noul/Score execution, and an explicit worker profile. The original 280-answer numerical gate still fails; keep this PR draft and #114 open.
Stacked on unchanged #209 (
6410e7634fd5e077000810ae07dc2f251b4f044d) and #211 (72af54fbca76fe34fc525cc4e5d49a7bbe89bd36), preserving their commits and authors. Main remainsf1e674f2d324f5bb4e1ba2162551d0b633f20be0. Head0e69d34cb3e30c51d1bbd76477bb63204b3bed40, treeb153b1f848dcfd048934d0841c6d03a56ca67f22. The incremental diff against #211 is 14 files / 995 changed lines, including 798 authored-code lines. The main-based stack is 31 files / 2359 changed lines.OmniJev owns request chunks, token/candidate order and features. Qwen executes the declared group without inferring model fields or silently splitting M. Score's unused LM rows remain in the normative gather. Existing selected-readout/fixed/window/reuse/FP32-GDN guards remain.
OMNIJEV_PROFILE=unmerged-referencerequires the separate artifact and selects eager reference execution; the default remainsmerged-fixed. The reference profile's 16,384-target preparation budget includes Score rows and returns 422 above that bound, a documented restriction relative to the merged profile's Score-exempt budget. This incremental stage changes no CUDA implementation, ABI, ops.h or root manifest.Required human contributor answers
The contributor's explicit shared answers and necessary-draft decision in #211 and its #209 source remain applicable to this bounded #114 successor:
PR needed: Yes; necessary implementation, L20 validation, self-review and a reviewable draft for each bounded stage were authorized. No review/Ready/merge/issue transition is requested.
Evidence
reference_whole_request_outputs_and_recoverytest fails on omnijev: add eager unmerged reference Score readout #211 withfull requests/reuse/LM targets are unsupported.After: All-type cardinality, repeated FP32 heads, invalid-target recovery, finishing and reuse rejection pass on the complete original request. Owned hidden remains valid across subsequent forwards and scratch growth.
Complete original 28 requests / 280 answers, unchanged raw/returned 0.01 and protected-decision gates:
All 140 PNG decisions agree; fifteen probabilities/confidences still exceed the gate. Protected JPEG failures remain
board.jpg/q8,arcade.jpg/long_start,web.jpg/one_token. Maximum returned-field drift is 0.1878. Every raw native answer rounds back to production exactly. No samples or failures were removed.The largest PNG residual is original full-context
video.png/q6. Its native zq/u equal official unmerged plain full-row values exactly, but differ from normative branch hidden (max 0.0625). Official LM/head on the same native hidden differs by only 5.96e-8 at q6's head. This bounds a successor to the full-row versus cached-branch backbone boundary; the first differing internal operator is untraced. The original branch stays the oracle. JPEG and branch-topology repairs are outside this slice.Frozen reference: source
14dbec4f71e194852c8d7b88ab36ef639493f400, checkpointffe5f436eaf22e20e2f041f8e74e121fd057a6cb, base851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a; original BF16 base / separate FP32 rank32 scale2 adapters, heads and calibration. Python 3.10.12 / Torch 2.14.0+cu130 / Transformers 5.17.0 / PEFT 0.19.1, no FLA, unchanged 64-package lock. One L20 GPU0; Rust 1.100 / native CUDA 12.8. Numerical validation only; no performance claim.Test plan and result
cargo fmt --all --check; locked workspace Clippy/all-targets/omni-laya/serve/-D warnings; ordinary workspace tests; locked release workspace build; discovery; exporter AST; strict MkDocs: pass. Ordinary workspace: 431 passed / 0 failed / 127 ignored. Ignored tests are not device evidence.reference_actual_lm_groups_and_features(also repeated after the test-only Clippy correction),reference_whole_request_outputs_and_recovery,reference_selected_rows_preserve_full_row_and_recover,unmerged_reference_original_q2_gate, existingtext_readout_equals_forward_fixed, existingprefix_reuse_matches_plain_passes_bit_for_bit: pass.f5ffba7e53e3289b0019a90bcb487483f7bc33e122c2d57ceaafafe2cf5e33bf; unchanged library SHA256b1f08a2177ffe5aa5701c710eec3a9dd71c0efe43b1b6b12bc6182f2c0e061e5.The numerical corpus was run at validation commit
e282562543484311954c050f481fe20e22f28f69, treeb7c0e36c87a4ecb33baa50eb1602018b0e963d24. Git Data publication retained that exact tree as46c1ae5ca784dd0c0e3c2c00fbc0691d018521ad; only the commit message's trailing newline representation changed. Author names/emails/instants and the prior stack remain intact. Final head adds documentation and a test-only Clippy correction, with unchanged inference source. Source aliases, raw exits, failed attempts and full outputs are retained privately.Unrun/unverified: internal first-difference layer trace, branch-topology/JPEG repair, reference reuse/Graphs, other GPUs/Metal, whole real-weight Cua/Open-Jev models, performance. Full #114 numerical acceptance is failed, not unrun.
Demo / evidence
Maintained commands, supported-mode boundaries and numerical results are in
recipe/omnijev/README.md,recipe/omnijev/native.md, and the model README. Tests/helpers remain under roottests/; actual private tensors use explicit opt-in paths and hash guards. The contributor retains the exact source bundles, original requests, actual tensor captures, full raw/rounded/protected gates, HTTP transcripts and logs in a private archive. Images/checkpoint tensors and one-off run output are not committed or publicly redistributed. A video is N/A for this numerical/serving slice.Merge Danger
Door: two-way. The artifact and profile selection are explicit.
Blast Radius: bounded. Shared Qwen gains owned hidden and an explicit reference projection seam; OmniJev owns mapping, consumers and worker selection. Existing fixed/reuse consumers pass their named regressions. Keep draft while the full numerical gate remains failed.
Self-review
Agent-assisted incremental architecture, numerical, lifetime, mode, test and artifact review completed. This does not replace contributor or maintainer review. The human checklist remains for the contributor: