Crescent Island (cri) support for the XPU/SYCL build by jiqing-feng · Pull Request #734 · huggingface/kernels

jiqing-feng · 2026-07-29T07:15:07Z

Adds Intel Crescent Island (cri) as an AOT target for SYCL kernels.

Why it isn't just one more device in the list

cri needs a newer toolchain: ocloc 25.35 doesn't know the device, and
sycl-tla 0.9.1 has no cri kernels. Both are bumped (ocloc → 26.22.38646.4,
sycl-tla → 0.9.2), and AOT for cri is gated on DPC++ 2026.0.

The bigger problem is that a kernel compiled with __SYCL_TARGET_INTEL_GPU_CRI__
is an Xe35 unit throughout, and its device code must not be lowered for pvc or
bmg. Since SYCL merges every TU that shares a target triple into one image, and
ocloc compiles that image once per device in its -device list, such a
translation unit can't share one AOT image across devices. The build now detects
that flag and picks the layout:

without it — a single spir64_gen image serves pvc,bmg-g21,cri, which keeps
the number of device compilations per TU down;
with it — the generic image is restricted to cri, and pvc/bmg get separate
intel_gpu_* alias targets.

Either way the .so also carries a spir64 JIT fallback, so a GPU that isn't
in the AOT list still runs.

Why the flag handling was rewritten

The SYCL flags used to be two hardcoded strings that dependencies patched with
string(REPLACE ...). That stops working once the target list is dynamic, and
it hid three failure modes that all produce a binary that builds fine and then
misbehaves:

backend options must be repeated per target, and CMake deduplicates the
identical -options strings, silently dropping every target but the first —
SHELL: prevents this;
-spirv-ext only reaches the image whose triple matches the exact
-fsycl-targets alias, so it has to be emitted once per target;
the SPIR-V translator replaces the extension list instead of merging, so the
driver defaults are now probed with icpx -### and sycl-tla's extras appended.
A failed probe is a hard error rather than a silent fallback.

Flags are now built from a few variables through a xpu_compose_sycl_flags()
macro, so a dependency changes a variable and recomposes.

Validation

Built flash-attn2 and megablocks from kernels-community against this
branch (torch213-cxx11-xpu20260-x86_64-linux) and checked numerics on both
BMG hardware and the CRI TBX simulator. Each case runs only the kernel on XPU;
data generation, the fp32 reference, MXFP4/MXFP8 dequantization and error
statistics stay on CPU.

flash-attn2 — 8/8 pass (fix / varlen × fp16 / bf16 × causal / non-causal),
max abs error 5.4e-04 (fp16) / 4.0e-03 (bf16).
megablocks grouped GEMM — 5/5 pass (bf16 ±bias, fp16, mxfp4, mxfp8), max
relative error 2.6e-03.

Both AOT layouts were also checked in the shipped binaries: flash-attn2, which
has no CRI translation unit, carries one spir64_gen image covering all three
devices, while megablocks carries per-device images. On the simulator the
runtime loads the AOT image rather than falling back to JIT.

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

ocloc 25.35 rejects the 'cri' device (Failed to parse target: cri - invalid device), which breaks the pvc,bmg,cri fat-binary link. Bump compute-runtime to 26.22.38646.4 (IGC v2.36.3, gmmlib 22.10.0) whose ocloc recognises cri.

The v0.9.1 tag predates the Crescent Island (cri) reverse-kernel support; pin to the upstream commit that landed it (sycl-tla d54352d, 'Upstream CRI code huggingface#833') so the Nix build can compile the cri AOT images. Revert to a released tag once one containing cri is available.

Build one .so with three AOT images. fp16/bf16 kernels compile for intel_gpu_pvc,intel_gpu_bmg_g21,spir64_gen; MXFP kernels are gated behind __SYCL_TARGET_INTEL_GPU_CRI__ and compile spir64_gen-only, AOT-lowered to cri via -device cri. Emit one -Xspirv-translator=<alias> per offload target so the custom -spirv-ext reaches every image (translator replaces, not merges).

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

The tri-target AOT-only build dropped the spir64 image, so sycl-tla kernels only loaded on pvc/bmg/cri. Re-add spir64 alongside the AOT targets to restore JIT coverage for other Intel GPUs. MXFP is absent from the JIT image, which is harmless since its dispatch is gated on device IP version >= 35 at runtime. Also align the sycl-tla revision across both paths. The non-nix FetchContent fallback pinned v0.9.1, which carries cri only as a placeholder constant and none of the Xe35 block-scaled kernels. nix-builder already pinned d54352d; v0.9.2 is exactly that commit, so both now use the tag and the source hash is unchanged.

…2026.0 The previous cri work passed -Xsycl-target-backend=spir64_gen only, which reaches the generic image alone, so the intel_gpu_pvc and intel_gpu_bmg_g21 alias images silently lost their backend options. Emitting one option per target then hit CMake's link-option deduplication, which dropped every repeated -options and made ocloc parse the next flag as its value, so use SHELL: to keep each pair intact. cri has no intel_gpu_* alias and is only reachable as a -device of the generic image. Only a kernel with an Xe35-only TU needs alias targets to keep MXFP code away from pvc/bmg; every other kernel can lower one generic image to all three devices, halving the device compilations per TU. Also gate the AOT device list on DPCPP 2026.0, since cri needs sycl-tla v0.9.2 and ocloc 26.22, and fail the build when the SPIR-V extension probe fails instead of silently dropping the compiler's default extensions.

jiqing-feng · 2026-08-07T03:22:05Z

Hi @danieldk . Would you please review this PR? Thanks!

sayakpaul · 2026-08-13T14:23:26Z

Daniel is away this week. Please expect some delay.

Resolved conflicts in the torch and tvm_ffi CMake preambles: keep the composable xpu_compose_sycl_flags() macro from this branch and fold the -foffload-fp32-prec-div/-sqrt flags from main into both the compile and the link flag sets.

danieldk · 2026-08-17T18:06:47Z

Thanks for the PR, I have it on my list for tomorrow!

danieldk

Looks good, thanks! Merging when CI is happy.

github-actions · 2026-08-19T06:29:57Z

Coverage report — `kernels/`

Measured on: Python 3.10 / Torch 2.13.0.
Other CI configurations are not included in this number.
Hardware-gated code paths (ROCm/XPU/NPU/Darwin/Windows) are excluded or unreachable on the Linux+CUDA runner.

Total coverage: 84.3% — threshold: 80% — ✅

Per-file breakdown

Name	Stmts	Miss	Cover	Missing
src/kernels/__init__.py	13	0	100%
src/kernels/_system.py	6	1	83%	10
src/kernels/_versions.py	63	7	89%	46, 49, 52-53, 56-57, 100
src/kernels/backends.py	212	62	71%	40, 44, 48-51, 68, 90, 108, 117, 121, 125-127, 148, 157, 161, 165-167, 188, 199, 201, 208-211, 224, 228, 232-252, 260, 283-303
src/kernels/compat.py	8	1	88%	5
src/kernels/deps.py	58	4	93%	59-60, 101, 104
src/kernels/hf_hub.py	62	5	92%	18, 20, 114, 136-137
src/kernels/importer.py	48	3	94%	106, 110, 113
src/kernels/install.py	50	5	90%	78, 116-117, 155, 173
src/kernels/layer/__init__.py	6	0	100%
src/kernels/layer/_interval_tree.py	103	4	96%	23, 52, 147, 150
src/kernels/layer/device.py	48	14	71%	42, 47-49, 91, 96-98, 101, 149, 152, 155-157
src/kernels/layer/func.py	82	7	91%	83, 113, 185, 303, 309, 322, 340
src/kernels/layer/globals.py	5	0	100%
src/kernels/layer/kernelize.py	74	8	89%	255, 281, 289-290, 296, 300, 316-318
src/kernels/layer/layer.py	211	16	92%	169, 212, 218, 231, 339, 419-420, 432, 441, 449, 460, 489, 493, 506, 559, 589
src/kernels/layer/mode.py	14	0	100%
src/kernels/layer/repos.py	144	42	71%	27, 33, 36-43, 63-64, 70, 73-76, 90, 94, 103-104, 110, 113-116, 123-124, 130, 133-136, 143-144, 150, 153-156, 163-164, 170, 173-176, 257
src/kernels/load.py	74	9	88%	218, 224, 230-231, 252-264
src/kernels/locking.py	108	70	35%	42-109, 113-136, 144, 148-155, 159-169, 173-180
src/kernels/status.py	49	2	96%	23, 81
src/kernels/variants.py	278	22	92%	64, 95, 116, 146, 255-256, 298-301, 303, 387-394, 400-406, 437-443, 455-461, 611-613
src/kernels/verify.py	88	1	99%	32
TOTAL	1804	283	84%

Updated by the Test kernels workflow on commit a95b66d50d7748d26d19cc1c7a5645003fcac129.

Two independent problems make the published XPU kernels slow. Both are in `dep-sycl-tla.cmake`. End-to-end result for `flash-attn2`, built through the normal `nix build .#redistributable.torch214-xpu20261-x86_64-linux` path. Intel Arc Pro B60, torch 2.14.0+xpu, fp16, causal, cold compiler cache. Baseline is the same build with sycl-tla pinned back to v0.8, i.e. the behaviour before the regression; all three columns are full 86-TU builds with identical AOT targets, so the only difference is what this PR changes. Throughput relative to that baseline, higher is better: | B x S x H x D | before the regression | Hub today | this PR | |---|---|---|---| | 4x512x8x64 | 1.00x | 0.56x | 0.90x | | 4x1024x8x64 | 1.00x | 0.32x | 0.87x | | 4x2048x8x64 | 1.00x | 0.12x | 0.81x | | 4x2048x16x64 | 1.00x | 0.11x | 0.80x | | 8x1024x8x64 | 1.00x | 0.21x | 0.83x | | 1x4096x32x128 | 1.00x | 0.11x | 1.06x | | 4x2048x16x96 | 1.00x | 0.08x | 0.78x | | 4x2048x8x192 | 1.00x | 0.42x | 1.00x | | 1x4096x32x192 | 1.00x | 0.16x | 1.01x | | first call | 1.00x | 0.07x | 0.93x | That is a 2.4x to 9.3x gain over the artifact users get today, and lands between 0.78x and 1.06x of pre-regression throughput. The remainder is v0.9.3 itself, which is still slightly behind v0.8 at head_dim 64 and 96; it is not something this PR can close, and v0.9.3 wins at the larger head dims. The rest of this description explains the two causes separately. ## 1. AOT is silently disabled on 2026.1 Two gates match `DPCPP_VERSION` against explicit lists that do not include `2026.1` — the version used by every current XPU variant (`torch214-xpu20261`, `torch215-xpu20261`). Missing an entry does not fail the build, it silently drops the AOT images and the `SPV_INTEL_2d_block_io` / `subgroup_matrix_multiply_accumulate` extensions. Kernels that use `XE_2D_*` fail to link (exit code 18); kernels that don't, like `flash-attn2`, just ship JIT-only. ```diff -if(DPCPP_VERSION STREQUAL "2026.0") +set(DPCPP_AOT_MIN_VERSION "2026.0") +if(DPCPP_VERSION VERSION_GREATER_EQUAL DPCPP_AOT_MIN_VERSION) -if(DPCPP_VERSION STREQUAL "2025.2" OR ... STREQUAL "2026.0" OR SYCL_TLA_REVISION STREQUAL "v0.5") +set(DPCPP_BLOCK_IO_MIN_VERSION "2025.2") +if(DPCPP_VERSION VERSION_GREATER_EQUAL DPCPP_BLOCK_IO_MIN_VERSION OR SYCL_TLA_REVISION STREQUAL "v0.5") ``` Both lists were already contiguous ranges, so this is behaviour-preserving for every version in `build-variants.json` and additionally fixes 2026.1. New DPC++ releases now opt *out* of AOT rather than opt in. `preamble.cmake` enables AOT by default for XPU; this gate is the only place that turns it back off. I added the AOT gate in #734, when `2026.0` was the only XPU version that existed. 2026.1 landed six days later in #784, which didn't touch this file. Probe kernel on `torch214-xpu20261-x86_64-linux`, before and after: ``` before: RequiresExtension: SPV_INTEL_2d_block_io icpx: error: llvm-spirv command failed with exit code 18 after: [16] __CLANG_OFFLOAD_BUNDLE__sycl-spir64_gen <- AOT [17] __CLANG_OFFLOAD_BUNDLE__sycl-spir64 <- JIT fallback ``` For `flash-attn2` this is a first-call effect: **2023.9 ms -> 147.8 ms** on a cold compiler cache. It does not change steady-state kernel time — that is problem 2. ## 2. sycl-tla v0.9.2 has a flash-attention forward regression v0.9.2 rewrote the FMHA forward mainloop and regressed throughput on Intel GPUs by up to 9x. It is fixed in v0.9.3. Same flash-attn2 sources, same compiler, same flags, same card (Arc Pro B60), cold cache. The only variable is the sycl-tla include path. fp16, causal, head_dim 64. Throughput relative to v0.8, higher is better: | B x S x H x D | v0.8 | v0.9.2 | v0.9.3 | |---|---|---|---| | 4x512x8x64 | 1.00x | 0.63x | 0.91x | | 4x1024x8x64 | 1.00x | 0.43x | 1.52x | | 4x2048x8x64 | 1.00x | **0.12x** | 0.91x | | 4x2048x16x64 | 1.00x | **0.11x** | 0.89x | | 8x1024x8x64 | 1.00x | 0.19x | 0.90x | v0.9.2 drops to a ninth of v0.8 throughput at 2048, and the gap grows with sequence length. v0.9.3 is back to roughly v0.8 and is 1.5x faster at 1024. These three builds are a reduced configuration — forward only, head_dim 64 — so that the sycl-tla revision is the single variable. The published `kernels-community/flash-attn2` `torch214-xpu20261` artifact pins kernel-builder at `d054906`, which resolves to v0.9.2, and measures in line with the v0.9.2 column. The revision is pinned in two places, both bumped to v0.9.3: - `kernel-builder/src/pyproject/templates/xpu/dep-sycl-tla.cmake` - `nix-builder/pkgs/xpu-packages/sycl-tla.nix` `cri` kernel support is retained — v0.9.3 was verified to compile for the `pvc`, `bmg-g21` and `cri` AOT targets. The sycl-tla revision gate keeps its explicit list — its fall-through is `FATAL_ERROR`, so it can't be forgotten silently. ## Notes Numerics unchanged (`max|diff|` ~1.2e-4 vs torch SDPA flash backend). AOT images cost build time and artifact size; the `spir64` JIT image is kept as a fallback for devices outside the AOT list. Existing kernels won't pick this up until they bump their `flake.lock` — `kernels-community/flash-attn2` currently pins kernel-builder at `d054906` — and are rebuilt.

jiqing-feng added 5 commits July 15, 2026 10:05

add cri

5e51564

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

update cri compile

18c964e

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

jiqing-feng marked this pull request as draft July 29, 2026 07:15

Merge branch 'main' into cri

50b4413

jiqing-feng force-pushed the cri branch from 953d3b4 to 50b4413 Compare July 29, 2026 08:36

jiqing-feng added 3 commits July 29, 2026 01:47

fix

482d627

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

jiqing-feng changed the title ~~Cri~~ Crescent Island (cri) support for the XPU/SYCL build Aug 5, 2026

jiqing-feng marked this pull request as ready for review August 5, 2026 08:01

Merge branch 'main' into cri

dbaef55

This was referenced Aug 6, 2026

flash-attn2: fix forward epilogue reduction for sycl-tla 0.9.2 huggingface/kernels-community#1065

Merged

Megablocks mxfp8 cri huggingface/kernels-community#1066

Merged

Merge branch 'main' into cri

a95b66d

Resolved conflicts in the torch and tvm_ffi CMake preambles: keep the composable xpu_compose_sycl_flags() macro from this branch and fold the -foffload-fp32-prec-div/-sqrt flags from main into both the compile and the link flag sets.

danieldk approved these changes Aug 19, 2026

View reviewed changes

danieldk merged commit 9d9b399 into huggingface:main Aug 19, 2026
42 of 45 checks passed

jiqing-feng mentioned this pull request Oct 22, 2026

Fix XPU flash-attention performance on DPC++ 2026.1 #921

Merged

Navigation Menu

Repository navigation

Crescent Island (cri) support for the XPU/SYCL build - #734

Crescent Island (cri) support for the XPU/SYCL build#734
danieldk merged 11 commits into
huggingface:mainfrom
jiqing-feng:cri

jiqing-feng commented Jul 29, 2026 •

edited

Loading

Uh oh!

jiqing-feng commented Aug 7, 2026

Uh oh!

sayakpaul commented Aug 13, 2026

Uh oh!

danieldk commented Aug 17, 2026

Uh oh!

danieldk left a comment

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jiqing-feng commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why it isn't just one more device in the list

Why the flag handling was rewritten

Validation

Uh oh!

jiqing-feng commented Aug 7, 2026

Uh oh!

sayakpaul commented Aug 13, 2026

Uh oh!

danieldk commented Aug 17, 2026

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 19, 2026

Coverage report — kernels/

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jiqing-feng commented Jul 29, 2026 •

edited

Loading

Coverage report — `kernels/`