Repository navigation
Conversation
|
Hi @ashuie! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
|
@malfet @zhili03 @fbarchard - could you take a look at this PR? Thanks! |
nSircombe
left a comment
There was a problem hiding this comment.
This will pick up V3 but miss V3AE. Could you add the MIDR for v3ae? (0x410FD830, I think - Implementor = 0x41, part = 0xD83).
|
@nSircombe Thanks, I've added V3AE. |
|
Might want to update the PR body as well. |
Requires `is_arm_neoverse()` to recognize Neoverse V3: pytorch/cpuinfo#435 On Arm Neoverse systems built with the oneDNN ACL backend, decline BF16 CPUBlas GEMMs when `n == 1` and `alpha == 1` so the existing native GEMV fallback is used. `n == 1` makes the operation a matrix-vector product, and these shapes are common during single-token transformer decode. In official Linux AArch64 wheels, the fallback is OpenBLAS `sbgemm_` when available. Otherwise, CPUBlas uses ATen `gemm_stub`. `is_arm_neoverse()` is required because this fallback only benefits Arm. Tested on Neoverse V3 - c8g.16xlarge and Neoverse V2 - c9g.16xlarge. GCC 13.3 and oneDNN 3.12.0 Workload: GPT-2 BF16 single token decode on one pinned thread | CPU | Before | After | Speedup | |---|---:|---:|---:| | Graviton 5 | 56.0842 ms | 10.5462 ms | **5.32x** | | Graviton 4 | 75.9550 ms | 14.1522 ms | **5.37x** | Across other thread counts: | CPU | Threads | Before | After | Speedup | |---|---:|---:|---:|---:| | Graviton 5 | 8 | 11.2110 ms | 4.0912 ms | **2.74x** | | Graviton 5 | 16 | 8.2988 ms | 4.5431 ms | **1.83x** | | Graviton 5 | 32 | 7.0627 ms | 4.2545 ms | **1.66x** | | Graviton 5 | 64 | 6.1616 ms | 4.2753 ms | **1.44x** | Correctness: CPUBlas results were compared against the existing oneDNN results across every transpose layout, `beta=0` and `beta=1`, and both aligned and non-aligned dimensions. Pull Request resolved: #198411 Approved by: https://github.com/aditew01, https://github.com/Skylion007
fbarchard
left a comment
There was a problem hiding this comment.
nit: Consider adding a unit test at the end of test/arm-cache.cc (line 2152) to exercise cpuinfo_arm_decode_cache for
cpuinfo_uarch_neoverse_v3 (0xD84) and V3AE (0xD83):
```cpp
TEST(ARM, neoverse_v3) {
const struct cpuinfo_arm_chipset chipset = {
.vendor = cpuinfo_arm_chipset_vendor_unknown,
.series = cpuinfo_arm_chipset_series_unknown,
};
struct cpuinfo_cache l1i = {0};
struct cpuinfo_cache l1d = {0};
struct cpuinfo_cache l2 = {0};
struct cpuinfo_cache l3 = {0};
cpuinfo_arm_decode_cache(
cpuinfo_uarch_neoverse_v3, 4, UINT32_C(0x410FD841), &chipset, 0, 4, &l1i, &l1d, &l2, &l3);
EXPECT_EQ(64 * 1024, l1i.size);
EXPECT_EQ(4, l1i.associativity);
EXPECT_EQ(64, l1i.line_size);
EXPECT_EQ(64 * 1024, l1d.size);
EXPECT_EQ(4, l1d.associativity);
EXPECT_EQ(64, l1d.line_size);
EXPECT_EQ(1024 * 1024, l2.size);
EXPECT_EQ(8, l2.associativity);
EXPECT_EQ(64, l2.line_size);
EXPECT_EQ(CPUINFO_CACHE_INCLUSIVE, l2.flags);
EXPECT_EQ(0, l3.size);
}
```
Support the Arm Neoverse V3 and V3AE architecture.
Verified on a Neoverse V3
gv5host:0x410fd841(r0p1)For reference: Adding Neoverse V2 support: #194