Skip to content

simx: reject LMEM configurations that are rejected in RTL - #433

Open
eyoon1131 wants to merge 1 commit into
vortexgpgpu:masterfrom
eyoon1131:simx-lmem-size-check
Open

eyoon1131 wants to merge 1 commit into
vortexgpgpu:masterfrom
eyoon1131:simx-lmem-size-check

Conversation

@eyoon1131

Copy link
Copy Markdown
Contributor

Summary

SimX currently builds and runs with any VX_CFG_LMEM_LOG_SIZE while VX_mem_unit.sv statically rejects two LMEM sizes: one that the LMEM base address is not aligned with, and, with WGMMA, one above 64 KB, the most the 16-bit smem descriptor offset can address. VX_MEM_LMEM_BASE_ADDR (0xFFFF0000 / 0x1FFFF0000) is a multiple of 64 KB but not of anything larger, so LMEM_LOG_SIZE > 16 is an invalid configuration. rtlsim fails under this config at startup, but SimX builds and still runs, leading to silent errors.

This PR mirrors those asserts in SimX, so an invalid configuration fails to build in both models.

Symptom

On xlen32 SimX with LMEM_LOG_SIZE=17, the CTA dispatcher grants slots up to 128 KB but slot bases above 64 KB (0xFFFF0000 + 64 KB = 0x1_0000_0000) truncate to 0x0000xxxx, which is the IO window. Those CTAs' shared-memory traffic overwrites the console rings.

Repro from a configured xlen32 build (rerun SimX with LMEM_LOG_SIZE=16 after):

CFG="-DVX_CFG_NUM_THREADS=16 -DVX_CFG_NUM_WARPS=16 -DVX_CFG_LMEM_LOG_SIZE=17"
make -C tests/regression/sgemm2 clean
CONFIGS="$CFG" make -C tests/regression/sgemm2
CONFIGS="$CFG" ./ci/blackbox.sh --driver=rtlsim   --app=sgemm2 --perf=7 --args="-n256 -t4 -c256"
CONFIGS="$CFG" ./ci/blackbox.sh --driver=simx   --app=sgemm2 --perf=7 --args="-n256 -t4 -c256"

-c256 gives 8 KB of LMEM per CTA. With 16 threads each CTA is one warp, so 8 CTAs are resident at 64 KB LMEM and 16 at 128 KB.

driver LMEM_LOG_SIZE result cycles LMEM reqs
simx 16 PASSED 33.5M 41.9M
simx 17 PASSED, plus 64 garbage [#N: lost <fp32 bits> bytes] console lines 49.9M 21.0M
rtlsim 17 VX_mem_unit.sv:69: Assertion failed ... invalid parameter - -

Note how SimX still reports a PASS at 17, because slots 8-15 wrap to the IO window which behaves like ordinary global memory. This explains the significant increase in cycles and the halving in LMEM reqs, as well as the console output.

Change

sim/simx/core.cpp: static_asserts mirroring VX_mem_unit.sv:59-69:

  • with TCU_WGMMA_ENABLE: LMEM is enabled, and LMEM_LOG_SIZE <= 16
    (descriptor offset field);
  • with LMEM_ENABLE: the LMEM size is a multiple of MEM_BLOCK_SIZE, and
    VX_MEM_LMEM_BASE_ADDR is aligned to the LMEM size.

They are in the Core constructor beside LocalMem::Create. SimX has no mem-unit object, so the core is where the RTL's VX_mem_unit sub-blocks (LMEM, LMEM switch, LSU adapters, coalescers) are instantiated.

Testing

xlen32, build/, SimX:

  • -DVX_CFG_LMEM_LOG_SIZE=17: fails to build with core.cpp:133: static assertion failed: LMEM base address must be aligned to the LMEM size.
  • -DVX_CFG_EXT_TCU_ENABLE -DVX_CFG_TCU_WGMMA_ENABLE -DVX_CFG_LMEM_LOG_SIZE=17: fails with core.cpp:129: static assertion failed: LMEM exceeds the 16-bit smem descriptor offset field.
  • Default and -DVX_CFG_LMEM_LOG_SIZE=16: build. sgemm2 passes as before.
  • The asserts generate no code, so every configuration that builds produces the same result as before. The catalog's widest configuration is NUM_THREADS=32, which gives 64 KB, so no catalog configuration trips an assert.
  • There's no regression test for the assert itself since a test would be an expected build failure.

VX_mem_unit.sv statically rejects two LMEM sizes: one that the LMEM base
address is not a multiple of, and, with WGMMA, one above 64 KB, the most
the 16-bit smem descriptor offset can address. SimX did not do this, so
it built and ran configurations such as VX_CFG_LMEM_LOG_SIZE > 16.

On xlen32 this caused silent failures. At LMEM_LOG_SIZE=17, CTA slots
above 64 KB start at 0xFFFF0000 + 64 KB, which wraps to 0x0000xxxx.
Those CTAs' shared-memory traffic lands on the console rings and the
exit-code word. sgemm2 still reports PASSED but prints garbage and half
of its LMEM requests leave LMEM. rtlsim correctly fails to run.

SimX now mirrors all four VX_mem_unit LMEM asserts as static_asserts in
the Core constructor so an invalid configuration fails to build.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant