Repository navigation
Conversation
hsliuustc0106
left a comment
There was a problem hiding this comment.
DRAFT review — PR #206 "test(vela): clarify #169 span coordinates and contract acceptance"
Verdict: COMMENTED. The PR is in draft state (draft=true, author psv666, first-time author). Per policy, an explicitly draft PR receives a COMMENTED review regardless of findings; this review does not approve, request changes, or alter the draft state, and does not fill the contributor's self-review checkboxes.
Template and draft state: All template sections present (Purpose / Test Plan / Test Result / Demo-evidence / Merge Danger / Self-review). All four self-review checkboxes are unchecked, consistent with draft ("keep draft" stated in the body).
Pinned metadata (observed via GitHub API and local git)
- Base:
main@f1e674f2d324("Add a Valen-Decision-1.5 reference worker (#204)") — verified an ancestor of current main (614b0abc, #214). Main has advanced since creation, but the PR reports mergeable=MERGEABLE / CLEAN. - Head:
codex/l20-b-vela-contract@bfc766c103cc; head repo InferMatrixLab (PRs live in the InferMatrixLab parent). Two commits, both authoredpsv666 <2693925048@qq.com>. - Files: 6, +616/−10 —
recipe/vela/example-request-unicode.json(new, +46),tests/vela/fixtures/oracle-example-request-unicode.json(new, +215),tests/vela/test_oracle_fixtures.py(new, +76),tests/vela/test_runtime_contract.py(new, +154),recipe/vela/reference.md(+110/−3),src/models/vela/README.md(+15/−7).
Body gate (post-#138 required answers)
All four mandatory answers present and specific, "Is a PR needed? Yes." present, plus an "Observed gaps" paragraph in the bug-fix-evidence slot (records the #169 conflict and the incorrect tail-coverage recipe advice; no runtime defect claimed). Gate status: present and complete.
Diff review — new tests vs docs, and what they pin about span coordinates
Both. +616 lines split between new tests (230 Python test lines) and documentation, with one consumed 215-line frozen oracle fixture.
New tests (all weight-free; verified to run locally):
tests/vela/test_oracle_fixtures.py(ordinary, run everywhere): binds the new Unicode oracle to the maintained recipe request (key order preserved) and to the production envelope gate (finish_responsefromsrc/models/vela/postprocess.py— verified present at main). What it pins about coordinates, at test_oracle_fixtures.py:45–63:EMAIL_ADDRESSspan[10, 30)with textzhang.wei@example.cnpositioned after a non-BMP emoji, and for every span that slicing the named fieldstate["request"][start:end] == span["text"], with explicit negative checks that offsets are neither UTF-8 bytes (len(field[:start].encode("utf-8")) != start) nor UTF-16 code units (len(field[:start].encode("utf-16-le")) // 2 != start) — exactly the discriminator #169's acceptance-fixture checklist demands ("Include CJK and non-BMP characters so byte/UTF-16 offsets cannot pass"). It also pins set independence (sum(probabilities) > 1.3, impossible after cross-label softmax), preserved per-label noul views, threshold echo, and single-sequence usage155/0— matching #169's items on set semantics, Noul views, and single-rendered-sequence token counting.tests/vela/test_runtime_contract.py(opt-in; skips explicitly withoutVELA_CONTRACT_PACKAGE, test_runtime_contract.py:25 — observed 8+3+1+1+1 = 14 skips locally): 8 parametrized upstream validation/error-precedence cases withlocpath assertions; the coordinate-origin test (test_runtime_contract.py:80) pinning the claimed triad —over: "request"→ field-local code-point offsets,over: ["request"]andover: "user"→ rendered-part offsets including field headers, with a controlledpredictstub asserting the upstreamspan_limitscomputation; and the actual 8,192-token assembly boundary, unprotected-before-protected cut accounting, and schema-only overflow as a distinct error. These execute the hash-pinned upstream runtime (no model forward), with controlled outputs explicitly distinguished from model-forward evidence.
Docs:
recipe/vela/reference.md:283+new "Contract decisions and acceptance" section: a Vela-policy table (five-kind forwarding incl. span/set; span identity/offsets; independent set selection; length limits; confidence; rendered-row usage; validation), four separate acceptance gates, the opt-in command, fixture provenance and regeneration command, and a "Pending contract decisions" list (reference.md:365+). The −3 lines replace incorrect advice: the old "checkusage.input_tokensagainst the expected budget" tail-coverage suggestion becomes "usage.input_tokens is work accounting and cannot prove that the complete state was examined" (reference.md:401–402) — a correction, not a gate change; the assembly policy indeed cannot guarantee tail coverage.src/models/vela/README.md:108–113mirrors the coordinate-origin clarification and the unresolved cross-part case; also fixes genuinely broken relative links (../../→../../../fromsrc/models/vela/— the old targets do not exist) and adds the recipe anchor link (README.md:211).
Accuracy of #169 references against actual merged code at main
- #169 is an OPEN issue ("[Contract]: Reconcile Vela span/set and truncation semantics before native parity"), not a merged PR. Its acceptance-fixture checklist (CJK+non-BMP span offsets, set independence, Noul views, invalid span/set + precedence + schema-vs-state + truncation boundary, pinned 0.3B CPU FP32 oracle) is delivered item-by-item by the new tests/fixture; the PR correctly does not claim #61 adoption or #169 resolution and keeps cross-part behavior recorded as unresolved rather than blessed as a client-error contract (reference.md:365+, body).
- The merged carriers it cites are verified merged: #151 (0.3B reference worker, merged 2026-10-25) and #174 (0.8B hybrid, merged 2026-10-27); #137 is an open issue; #61 open with the linked comment. Accurate.
- The field-local claim is consistent with the pre-existing README wording at main ("Unicode code-point offsets into the targeted field's text (clipped and re-trimmed to the field when it shares a part)"). The part-target nuance (
["request"]≠"request"when fields share a part) is new documentation; the rendering/span-limit logic lives in the hash-pinned upstream package (vela2_inference.pypinned insrc/models/vela/engine.py:46, sha256d3e140c7…), not in-repo, so it is pinned by the opt-in tests only (see P3 below). The repo-side plumbing the tests rely on (verify_package,verify_no_foreign_runtime,put_first_on_sys_pathin engine.py:200/218/233) is verified present at main. The cross-part_to_generic→_prepTypeError is body-reported repro only, with no committed test — deliberately, since the PR refuses to codify it as a contract (disclosed).
Gate preservation (extends, does not weaken)
Additive only: no existing test or fixture modified (the original four oracle files untouched per diffstat), no tolerance or gate touched, and the one semantic doc edit removes a false assurance rather than loosening a requirement. The 14 new skips are explicit, package-gated opt-ins matching the repository's existing opt-in conventions (same pattern as --ignored GPU tests), and are disclosed in the body with the exact count. Extends the approved-family surface (#195's native acceptance is explicitly left separate).
Findings
- P3 — the span-coordinate origin triad is pinned only by opt-in tests that ordinary CI never executes. Trigger:
test_runtime_contract.pyrequiresVELA_CONTRACT_PACKAGE; locally and in CI without that env the 14 checks skip (observed). Consequence: the reference.md/README statements about["request"]vs"request"offsets rest on author-reported opt-in runs (body: 14 cases included in 161 passed on L20), not on any gate that runs unattended. Disclosed, but before this becomes the documented contract, at least one recorded run at the eventual merge head should be linked, or the package env wired into an opt-in CI lane. - P3 — cross-part span-list failure has no committed regression. Trigger:
over: ["request","source"]upstreamTypeErroris described in the body and reference.md:365+ but has no test. Consequence: the eventual contract decision (error status? rejection?) will lack a pinned repro; acceptable for now because the PR deliberately does not bless it, but track it under the pending-decision list. - Notes (informational):
- Main advanced from
f1e674f2to614b0abc(workspace version bumps); no conflict observed for these files, but a rebase/retarget is normal hygiene before ready. "span_heads" not in response(test_oracle_fixtures.py:59) covers the 0.3B's single-span-head case; hybrid router/broad identity remains with the pre-existing hybrid fixtures — coverage note, not a gap introduced here.- The committed oracle retains a partial PERSON prediction and the body explicitly disclaims task-accuracy meaning — correct framing.
- Main advanced from
Provenance (separated)
- Source inspected: PR body/metadata via API; full diff
f1e674f2d324..bfc766c103cc; #169 issue text; #151/#174/#137/#61 states;src/models/vela/{engine,postprocess,preprocess}.pyat main; README/reference.md at main and head; CONTRIBUTING.md. - Commands run (this review, macOS, no GPU; Python 3.12.8 / pytest 8.4.1 / mkdocs 1.6.1): worktree at
bfc766c103cc(removed after):PYTHONPATH=src python -m pytest tests/vela -q→ 147 passed, 14 skipped (all 14 verified to be the disclosedVELA_CONTRACT_PACKAGEskips; consistent with the body's 135 original + 12 new ordinary = 147, and 161 = 147+14 with package);PYTHONPATH=. python -m pytest tests/benchmarks -q→ 133 passed;mkdocs build --strict→ exit 0, 0 warnings. - Observed CI (head
bfc766c103cc, verified per-SHA in BOTH repos): ThinkFlowLab Actions: CI (ci.yml) success + Docs (docs.yml) success, event=pull_request, 2 runs; InferMatrixLab: same 2 runs success;gh pr checks: rust, benchmarks, build, cuda-compile, cuda-contract, kev-export pass, deploy skipping. This is the only PR of the three with current-head CI evidence. - Author-reported (not verified here): the L20 opt-in package run (161 passed), the reference-vs-worker HTTP comparisons, the byte-equality frontend loop, the cross-part TypeError repro, fixture capture provenance.
Bottom line
A well-scoped contract-documentation and fixture PR that accurately records #169's current state, adds 12 ordinary + 14 opt-in tests without touching existing gates, and corrects incorrect tail-coverage advice. COMMENTED. Before ready: link a recorded opt-in contract run at the merge head (P3), and keep the cross-part decision tracked (P3).
Required human contributor answers
The contributor confirmed this single stage on 2026-11-19: Vela #169 contract audit and acceptance wording. #137 and the native PRs provide background/evidence; implementation of the native stack and other models is outside this draft's scope. Detailed human confirmation is retained in the contributor's task record.
Is a PR needed? Yes. The contributor authorized necessary contract documentation/tests and a stage draft PR.
Observed gaps: #169 records the conflict between #137's old span/set rejection/no-truncation proposal and the shipped reference. The recipe suggested token usage as a tail-coverage check, which the actual assembly/window policy does not guarantee. Source inspection and a preparation-only repro also show distinct span coordinate origins and an unresolved cross-part-list failure. This PR changes no inference or transport behavior.
Summary
Document Vela's existing five-kind policy, clarify span coordinate origins and pending decisions, and bind a real non-BMP CPU oracle to maintained acceptance tests.
Preserve span/set extensions, upstream state truncation, model-defined confidence and actual rendered-row token accounting. The shared RFC #61 remains open. Related: #169, #137, merged #151/#174. This does not mark native/family acceptance complete.
Evidence
[10,30)after an emoji; slicing the named request field giveszhang.wei@example.cn. Independent set probabilities sum to more than1.3. The actual partial PERSON prediction is preserved without a task-accuracy claim.requestandsource; a span usesover: ["request","source"]. Mapping resolvestarget_part=["user","context"]; upstream_to_generic->_prepraisesTypeError: unhashable type: 'list'before a model forward. This is recorded as unresolved scope/error behavior, not blessed as a client-error contract.Personally executed on L20 Linux, baseline
f1e674f2d324f5bb4e1ba2162551d0b633f20be0, final candidatebfc766c103ccf529b3f277cba38550c42f04ae2e:The real-model/HTTP runs preceded the final documentation/controlled-coordinate follow-up; their inference/transport source is unchanged. With no package env, the14 upstream checks skip explicitly; ordinary tests need no weights/torch/GPU and perform no download.
Purpose
Provide #169's Vela-specific current-state/acceptance record, correct the tail-coverage advice and relative recipe links, and name the remaining decisions without changing external behavior.
Pending decisions
Test Plan
vllm-sr/Vela-2.0-0.3B@d6f03aa9baca4f9017fa8f8e9c2764d3bed3f409; repository package gate plus full model LFS SHA256bd37dd0db177d766bc554eded78ae8493bc248172d0480f6a543b30932a0cc8e.a56fc7bebdd3bfbb4edbbb80083c6a0421a7d51ab232533e869a14348f02bd0f.vela03-evidence.tar.gz, SHA256bcdaf65b82cfc10e9fd0ca8e91cae4e03b9ff077c960e8b5c1914e97ad12b1a2, is retained on contributor-local and L20 storage, with final coordinate repro/test/docs output retained separately. Raw machine logs remain outside product Git.Test Result
Scoped CPU/reference/HTTP/documentation checks pass. The earlier Windows ordinary suite had128 pass/7 failures from Linux-extension assumptions, symlink privileges and default CJK decoding; the original suite passed on Linux. New fixture consumers use explicit UTF8; unrelated platform tests are unchanged.
No external behavior, calibration or numerical gate is changed. Formal contract adoption, issue closure and native acceptance are not claimed.
Demo / evidence
The committed oracle is an actual CPU prediction captured by the documented exporter. Real direct/frontend comparisons preserve complete response bodies with no timing-field exemption. Video and performance A/B are inapplicable to this docs/contract-test change.
Merge Danger
Door: two-way.
Blast Radius: contract.
230 added Python test lines; total+616/-10, including a215-line consumed oracle. Tests import only a package accepted by the existing hash gate. The docs describe current behavior and pending policy; they do not standardize other workers.
Self-review
Agent precheck reviewed the six-file diff, owning upstream functions, production response gate, fixture provenance/consumers, ordinary-test independence and docs links. All test bodies/fixtures are under root tests. The contributor checklist remains for human review; keep draft.