Remove redundant CUB barriers from blockwise dequantization - #2053
Open
heiheiha798 wants to merge 2 commits into
Open
Remove redundant CUB barriers from blockwise dequantization#2053heiheiha798 wants to merge 2 commits into
heiheiha798 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remove the two external CTA barriers immediately before the one-shot CUB
BlockLoadandBlockStoreinkDequantizeBlockwise.The two collectives already use distinct, non-aliased shared
TempStorageobjects. The launch geometry gives each CTA one loop iteration, so neither
storage object is reused after its collective. This leaves CUB's internal
synchronization intact and does not change the loop, grid, launch geometry,
decode arithmetic, formats, dispatch, streams, or public API.
Tracking: fork issue #10
and engineering draft PR #11.
Changes
csrc/kernels.cu: delete exactly the two redundant external barriers.tests/test_ops.py: cover full/tail General8bit, NF4, and FP4 tiles for alloutput dtypes, repeated allocating and
.outcalls, and unchanged inputs.benchmarking/dequantize_blockwise_barriers.py: build two Git revisions inisolated source trees and emit complete interleaved CUDA-event samples,
medians, p10/p90, round ratios, and effective bandwidth as JSONL.
The production CUDA diff is two deletions. There is no architecture guard
because the storage-lifetime property is not SM103-specific.
B300 Results
Primary evidence is from an official CUDA 13 x64 compatible build with targets
75;80;86;89;90;100;120, executed on one NVIDIA B300 SXM6 AC (CC 10.3,148 SMs) in Slurm job 4974. Software: driver 580.126.09, CUDA/nvcc 13.0.88,
PyTorch 2.13.0+cu130, Python 3.12.3. The isolated libraries were built from
baseline
95f9af309d4d5793847169c39288dcd3fcbdf564and exact candidate1f033cccf1a47e6cf054c7083e06ead723c559b7; the test process asserted that itloaded the temporary candidate library rather than a checkout artifact.
Each cell used 20 warmups per round, seven interleaved CUDA-event rounds, and
size-scaled repetitions (4-100 per round). Effective bandwidth counts packed
input, absmax, the General8 map where applicable, and output bytes.
Across the exact-head matrix:
max-minus-min round-ratio spread below the measured effect.
check.
0.9820x.
were 3,922,064 bytes.
The broader pre-commit calibration in job 4963 exercised 213 direct cells,
198 allocating/
.outpublic correctness cells, three non-contiguous controls,packed sweeps, custom maps, extreme absmax values, and nested/backward paths.
It also found zero raw-bit differences and no greater-than-5% regression.
The original raw-log paths and full engineering record remain listed in the
fork draft PR.
Reproduction
Run inside a single-GPU Slurm allocation from a clean candidate checkout. The
build root must not already exist.
Focused synchronization checks can be reproduced against the candidate build
with the permanent full/tail test:
Validation
Exact-head job 4974, using the path-asserted temporary candidate library:
tests/test_ops.py -k 'dequantize_blockwise or dequantize_4bit':99 passed, 306 deselected.
tests/test_functional.py -k 'dynamic_blockwise_quantization or test_4bit_quant or test_4bit_compressed_stats':165 passed, 1408 deselected.
tests/test_autograd.py -k test_matmul_4bit:384 passed, 384 deselected.
compute-sanitizer --tool racecheck: 0 hazards, 0 errors, 0 warnings.compute-sanitizer --tool synccheck: 0 errors.targets.
pre-commit run --all-files: all hooks passed on the login CPU.tests/test_ops.py: 405 tests collected without animport/collection failure; the 18 CUDA synchronization cases skipped as
expected.
The contribution preflight also reran
pre-commit run --all-filesagainst theexact submitted head; all hooks passed without changing the worktree.
Limits
official-compatible CUDA 13 image path. No native-SM103 performance result
is substituted for that deployed-code evidence.
hipcc. The requested HIP/hipCUB compile check and HIP runtime validationwere therefore not run. This PR makes no HIP behavior claim; downstream HIP
CI remains required before merge.
were not benchmarked on hardware. The change is architecture-independent
and does not add or alter runtime dispatch.