fix(kernels): never launch an unresolved optional kernel symbol (#180)

[dndungu]

Closes #180.

Why

RepeatInterleaveF32 SIGSEGVs the whole process on the GB10, taking down every
GQA model on GPU (Llama, Mistral, Qwen, Gemma). zerfoo currently ships a
mitigation that disables the fused path entirely (zerfoo#980), paying a
performance cost.

Root cause

The null pointer is the function pointer, not a buffer. The issue guessed
an unallocated dst; the trace says otherwise. In

cuda._Cfunc_ccall_wrapper(0x0, ...)
kernels.RepeatInterleaveF32(0xf06420000900, 0xf06420000a00, 0x1, 0x2, 0x2, 0x8, 0x2, 0x21abf240)

0x0 is Ccall's fn argument. The input and output device pointers one
frame up are both non-null.

launch_repeat_interleave_f32 is an optional symbol. purego.go's
optionalSyms set exists so a library missing a newer kernel is not fatal; on
dlsym failure openKernelLib leaves the pointer at 0 and continues, on the
stated contract that "callers check before use"
(internal/cuda/kernels/purego.go:496-497).

fused_repeat_interleave_purego.go:19-26 checked only k == nil. It never
checked k.launchRepeatInterleaveF32 == 0, so it called cuda.Ccall(0, ...),
a jump to address 0 — SIGSEGV PC=0x0 inside cgo. That is a process kill, not
an error return, which is exactly why zerfoo's Reshape -> Repeat -> Reshape
fallback never got a chance to run.

Why the symbol is missing there, measured rather than assumed. The standing
gate pod (zerfoo docs/bench/manifests/validate-arm64.yaml) does not build
kernels from source; it sets LD_LIBRARY_PATH=/usr/local/cuda/lib64:/opt/zerfoo/lib
and kernelLibPaths[0] ("libkernels.so") resolves to the prebuilt library there:

$ nm -D --defined-only /opt/zerfoo/lib/libkernels.so | grep -i repeat
0000000000015f50 T launch_repeat
# launch_repeat_interleave_f32 is absent

That library (built 2026-07-03) exports 79 symbols; a build of current main
exports 130. The kernel source, the header, and the Makefile SRCS entry were
all fine — only the deployed binary was behind.

Scope: this was a class, not a one-off

An AST gate over the package found 34 cuda.Ccall sites launching optional
symbols with no zero-check, across 14 wrappers. Cross-referencing against the
pre-refresh deployed library, 19 of those symbols were absent from it, i.e.
they were live process kills rather than latent ones; the other 15 are latent
(that library happened to export them). 20 sibling wrappers already had
the guard (flash_decode_purego.go:40, gemv_warp_purego.go:20,
fp4_gemv_purego.go:42, ...), so this restores the intended, already-established
pattern rather than inventing one. All 34 sites now check; the diff is the same
three lines each time.

What changed

• All 34 unguarded optional launches now return an error naming the missing
  symbol instead of jumping to 0.
• IsRepeatInterleaveF32Supported() (matching the existing
  IsQ4KSm121Supported() pattern), because GPUEngine.RepeatInterleave falls
  back on any error and otherwise gives callers and tests no way to tell "the
  fused kernel ran" from "we silently took the slow path".
• The optionalSyms comment now states the invariant and names the gate.

Verification (ADR 091)

Red first, then green. Against the unfixed wrapper,
TestRepeatInterleaveF32_UnresolvedSymbol_ReturnsError does not fail — it
crashes the test binary, reproducing #180 exactly, with no GPU required:

=== RUN   TestRepeatInterleaveF32_UnresolvedSymbol_ReturnsError
SIGSEGV: segmentation violation
PC=0x0 m=0 sigcode=1 addr=0x0
signal arrived during cgo execution
github.com/zerfoo/ztensor/internal/cuda._Cfunc_ccall_wrapper(0x0, 0x460686809eb8, ...)
github.com/zerfoo/ztensor/internal/cuda/kernels.RepeatInterleaveF32(0x460686809eb8, 0x460686809db8, 0x1, 0x2, 0x2, 0x8, 0x2, 0x0)
	internal/cuda/kernels/fused_repeat_interleave_purego.go:23 +0xe0

The class gate was red with all 34 sites named:

=== RUN   TestOptionalKernelSymbolsAreGuarded
    checked 113 cuda.Ccall sites against 62 optional symbols
    fused_repeat_interleave_purego.go: RepeatInterleaveF32 launches optional symbol
      launch_repeat_interleave_f32 (k.launchRepeatInterleaveF32) without checking it
      is non-zero; a missing symbol makes this a SIGSEGV, not an error (ztensor#180)
    ... 33 more ...
--- FAIL: TestOptionalKernelSymbolsAreGuarded

After the fix, on the GB10:

--- PASS: TestRepeatInterleaveF32_UnresolvedSymbol_ReturnsError
--- PASS: TestRepeatInterleaveF32_NoKernelLib_ReturnsDistinctError
--- PASS: TestOptionalKernelSymbolsAreGuarded
    checked 113 cuda.Ccall sites against 62 optional symbols
--- PASS: TestGPUEngine_RepeatInterleave_CPUParity
    --- PASS: /issue180_repro_1x2x2x8_rep2
    --- PASS: /identity_rep1
    --- PASS: /llama_8kv_rep4
    --- PASS: /batch3_rep3
    --- PASS: /ragged_tail
--- PASS: TestGPUEngine_RepeatInterleave_UnsupportedFallsBack

These assertions were checked for sensitivity, not just for green. This repo
has a documented habit of gates that pass with real bugs present (a parity
harness that loaded the same file per row, #182's tolerance 1000x looser than
the measured error, a softmax that saturated until the test was position-blind
— lore L-0009/L-0018), so each assertion was made to fail on purpose:

1. RepeatInterleave falls back to the generic chain on any error, so a
   value-only parity test would go green while the fused kernel never ran. The
   test asserts the kernel is present first and fails rather than skips.
   Verified by pointing LD_LIBRARY_PATH at the stale /opt/zerfoo/lib:
   launch_repeat_interleave_f32 is not in the loaded libkernels.so; ... this test would prove nothing about the fused kernel.
2. Head expansion is a mapping problem, so every element encodes its own
   (b, kv, s, d) and a self-test asserts that encoding actually distinguishes
   heads before the comparison is trusted. Mutating the reference mapping from
   q/rep to q%numKV turns both red (12544/14336 elements differ, plus
   reference is position-blind: output heads 0 and 2 are identical).
3. Comparison is exact, not toleranced: repeat-interleave is a pure gather
   and performs no arithmetic, so any nonzero difference is a real defect.
4. The missing-symbol test is paired with a no-library test that must produce a
   distinct error, so an unconditional return error cannot satisfy it.
5. Both static gates refuse to pass if their own extraction finds nothing
   (if len(fieldForSym) < 100 { t.Fatalf(...) }, if callSites == 0 {...}).

No regressions. go test ./internal/cuda/... on the GB10 host produces a
failure set byte-identical to clean main in the same environment (118
failures both sides, diff clean). Those failures are environmental — this
host is not the gate pod, and several are tests that assert error returns "without
CUDA" on a machine that has CUDA. The standing gate pod remains the authority.

gradcheck (ADR 091 harness 1) is deliberately not extended: it is defined over
graph.Node implementations, and RepeatInterleave is an engine-level forward
gather with no graph.Node and no analytic Backward in ztensor. The
closed-form reference used here is a stronger oracle for a pure gather than
torch would be, since there is no floating-point convention to diverge on.

Risk

Behaviour only changes where the process previously died. Every new branch
converts a crash into an error return on a path that already had an error
return, and every affected caller already handles it (GPUEngine.RepeatInterleave
falls back to Repeat). The cuda-tagged build is unaffected apart from the
new always-true IsRepeatInterleaveF32Supported.

Follow-ups, not done here

• The deployed /opt/zerfoo/lib/libkernels.so on the DGX is 7 weeks behind
  main (79 exported symbols vs 130). This fix makes a stale library safe,
  but the fused GQA kernel cannot actually run until that library is
  refreshed. Refreshing it re-activates roughly 50 currently-dormant optional
  kernels at once, which is a much wider blast radius than this PR and wants
  its own validated rollout.
• nsa_attention_f32, kv_dequant_*, iq_dequant_* are declared and resolved
  in purego.go but have no .cu in SRCS and no Go caller. Dead entries;
  left alone.