Repository navigation
perf(kolibri1 TT): drop the two blocking device round-trips per expert fill from the P150 decode step - #3517
Merged
Conversation
The kolibri-1 P150 decode step carries two blocking device round-trips per touched expert: the fill-time D2H readback + Synchronize, and the memo-miss device re-read. The shadow already stores full slot bytes host-side and proves device==shadow at fill time, so the memo-miss re-read is redundant for compute and the fill verification can defer its drain to one point per layer without weakening the byte-exact gate. The spec scopes both levers, keeps the verification itself untouched, and names the red-first measurement (byte-identical decode steps, churn-heavy schedule) and the stop conditions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…host bytes The B2b-ii memo-miss path re-read the staged slot from the device with its own D2H copy plus a blocking Synchronize before it verified and dequanted — a second blocking round trip per miss on the decode critical path. The shadow already holds the authoritative slot bytes host-side: the fill-time readback verification proved device == shadow when the slot was staged, the eviction hook invalidates the shadow on every swap, and the slot-pool policy refuses a SlotFor on a slot the shadow does not hold. Kolibri1TTSlotShadow::Bytes(layer, slot) now serves those bytes to the dequant directly, and the memo-miss device re-read is deleted. The verification itself is not weakened: every fill still runs its D2H byte-exact compare against the shadow at fill time; only the memo-miss re-read of bytes the host already holds goes away. Memo-miss VerifyReadback accounting moves with it (the re-read no longer happens, so its verified-byte charge no longer accrues). P150 evidence (docs/bench-evidence/tt-kolibri1-readback-2026-10-28/): decode median 205.903 s vs baseline 214.919 s over 3 idle-gated runs each, -141 ms/step (-4.2%), CHAIN byte-identical to baseline 3/3. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
Each B2b-ii slot fill ended in a blocking Synchronize to settle its D2H readback, so the H2D of the next fill serialized behind the D2H of the last, mid-layer, on the decode critical path. The fill path now enqueues the D2H without the per-fill Synchronize and ONE drain per layer — after the fetch loop, before the compute loop — settles every readback and runs the unchanged byte-exact compares. The tt-metal queue accepted N outstanding D2H copies into distinct host destinations, so no K-fill batching fallback was needed. Two structural notes. The shared slot_image buffer becomes one buffer per fill: a deferred drain needs the previous fill's destination intact. And the shadow now Records the HOST image before the D2H enqueues, so the drain-point compare is device bytes vs host image — the H2D integrity gate itself. That is strictly stronger than the pre-change ordering, which recorded the post-sync device bytes and compared the shadow against the same buffer. The on-demand fallback fill inside the compute loop drains immediately, still before anything computes over the slot. The reviewer mutation proves the moved gate still fires: flipping one byte of the staged DEVICE slot after the H2D lands aborts at the drain-point VerifyReadback (docs/bench-evidence/ tt-kolibri1-readback-2026-10-28/mutation-red-drain-verify.log). The mutation was reverted byte-for-byte after the RED capture. P150 evidence (same directory): decode median 203.812 s vs lever-1 205.903 s (-33 ms/step) and baseline 214.919 s (-173 ms/step, -5.2%) over 3 idle-gated runs each, CHAIN byte-identical to baseline 3/3. The B2b-ii unit suites (test_kolibri1_tt*) are green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
Three corrections from the fresh review, all documentation in the evidence README: 1. The mutation-RED log (mutation-red-drain-verify.log) was taken on LEVER-1 code, predating the lever-2 commit by ~2h; the reviewer reproduced the equivalent RED on the true head d9b55d5 (one device byte corrupted before the readback enqueue, abort at first fill, kolibri1_tt_stream.cpp:272). 2. The "all test_kolibri1_tt* suites green" claim was green with device legs skipped (0.29s ctest runs, no VT_KOLIBRI1_TT_B2II_MODEL); the device golden gate is red at base AND head with byte-identical signatures (flip-record md5 59746f64... both), pre-existing environment drift, filed as ISSUE-LOCAL-01M62AJ8XNK23FG6ZVATKEJ0D9 on row MODEL-TEXT-kolibri-1-tenstorrent. The reviewer re-verified the CHAIN byte-identity claims from the raw logs. 3. Added the gate-history nuance: the pre-change fill-time verify was a tautology (Record-after-sync then compare the same buffer) and lever 2's Record-before-enqueue ordering replaces it with a strictly stronger device-vs-host compare. Net: strengthened. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
The README cites ISSUE-LOCAL-01M62AJ8XNK23FG6ZVATKEJ0D9 for the device-golden-drift finding, but the issue file itself was left untracked in the worktree -- so CI refused the reference (must resolve to exactly one local record; found 0). The file is the row's evidence of ownership: the drift red is pre-existing at the base commit and out of this row's scope. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5-3-flash [maki]
lu-zero
added a commit
that referenced
this pull request
Oct 29, 2026
Both rows landed 2026-10-29: the readback levers as 90fe546 (PR #3517) and the NUMA interleave degrade as db1d71c (PR #3519). Each issue closes with dated resolution evidence and each spec gains its ## Outcome: what was measured, what the review's mutations proved (the tautology finding; the load-bearing active_ flag; the doctest 2.5.2 SUBCASE sibling-loss), and what stays open (the absolute anchoring and first green device-golden under the kolibri-1-tenstorrent row; the stderr-count gate candidate; the env-doc semantics-drift characterization). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5-3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The kolibri-1 P150 decode step carried two blocking device round-trips per
touched expert: the fill-time D2H readback + Synchronize (verifying the H2D
staged the slot byte-exactly) and the memo-miss device re-read (supplying
bytes to the host dequant). ~19 fills per step, each draining the queue
mid-layer. This row removes both costs without weakening verification --
and, as the fresh review established, actually strengthens it.
Lever 1 (51a8221): the memo-miss path dequants from the slot shadow's
stored host bytes (new Kolibri1TTSlotShadow::Bytes()) instead of re-reading
the device. The second D2H + Synchronize is deleted.
Lever 2 (d9b55d5): the fill path keeps every byte-compare but stops
draining per fill -- one drain per layer, after the fetch loop and before
any compute over a slot; on-demand fallback fills still drain immediately.
Record() now stores the host image BEFORE the D2H enqueues, so the
drain-point VerifyReadback compares device bytes against a host image
independent of the copy. The review found the pre-change fill-time verify
was a tautology (Record-after-sync then compare the same buffer -- it could
never fail); the only genuine compare lived in the now-deleted memo-miss
re-read. The new ordering replaces it with the first real device-vs-host
compare this path has had: strictly stronger, not weaker.
Measurements (P150, host thalia, same-session A/B, 3-run medians,
idle-gated starts, greedy CHAIN byte-identical 3/3 on every arm):
baseline 3358.1 ms/step, lever 1 3217.2 (-4.2%), lever 1+2 3184.6 (-5.2%).
The historical 548.5 ms/step figure did not reproduce on this session;
all comparisons are same-session A/B, and the absolute-state anchoring is
filed separately (ISSUE-LOCAL-01M62AJ8XNK23FG6ZVATKEJ0D9, with the device
golden gate's base==head red signatures).
Fresh review (MERGEABLE): five mutations on the true head -- gate deletion
is silent except the stage-stats counter (load-bearing, proven), per-fill
vs per-layer timing behaviorally identical, drain call site deletion caught
only by the counter (counter-assertion candidate recorded), device-byte
corruption aborts at the first fill, wrong-slot memo dequant caught by the
correctness gate. Evidence corrections recorded in the evidence README
(ad4ba30): the implementer's mutation RED was lever-1 code (reviewer
reproduced on head), the suites-green claim was device-skipped, and the
gate-history nuance above.
Known non-goals, recorded: the tt-metal worker-spinner fleet these
host-only suites initialize is untouched (pinned in test runs); the
device golden gate's environment drift is a separate row; the
counter-assertion checker idea is recorded as a candidate, not landed.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]