[dpsoft]

Fixes #215.

The bug

With PC sampling on, a PyTorch process using the CUDA 13 wheels segfaulted at context creation on any host that also had a CUDA toolkit installed: 20 of 20 runs, in both tiers. cuptiPCSamplingEnable loads libnvperf_host.so by name, and the wheel's CUPTI has no RUNPATH, so the loader found the toolkit's copy through LD_LIBRARY_PATH or the ldconfig cache. CUPTI 13.0.85 then called into a 13.3 libnvperf_host and crashed in NVPW_CUDA_LoadDriver. The process died, not just the profile.

The change

When a PC tier is selected, InitializeInjection now calls pin_nvperf_host() before anything else reads the tier. It finds the libcupti in use with dladdr, checks whether a libnvperf_host.so is already mapped, and then does one of four things:

• Nothing mapped yet (the normal case): it dlopens the libnvperf_host.so next to libcupti, and keeps it loaded. A by-name dlopen returns an object already mapped under that soname, so CUPTI's own lookup gets this copy. The report shows nvperf=pinned.
• The sibling is already mapped: nothing to do (matched). Files are compared by device and inode, not path, so /usr/local/cuda/lib64 and /usr/local/cuda-13.3/lib64 are one copy.
• A different copy is already mapped: it cannot be unloaded, so PC sampling is turned off for the process, tier_refused is incremented, and both paths are logged with the LD_PRELOAD workaround (mismatched). The process runs unsampled instead of crashing.
• No sibling next to libcupti: there is nothing to compare against, so it behaves as before and logs a warning (no-sibling).

If the sibling exists but fails to load, PC sampling is turned off (pin-failed). The sibling is looked up in the directory libcupti was opened from, then in the directory its real path resolves to, so a symlink farm that links only libcupti still finds it.

The decision is in shim/core/nvperfpin.h, with no dl calls, and nvperfpin_test.cc covers it in make test without a GPU. The test includes a hard link standing in for the two-names-one-directory toolkit layout. Both exit report lines gain an nvperf= field. bench/cmd/scenario/gpupc.go parses those lines as key=value pairs, so it is unaffected. docs/gpu-injection.md describes the requirement and the report values.

The check runs only when a PC tier is selected. With PC sampling off, nothing changes and the report reads nvperf=unchecked.

Verification

• make -C shim test and make -C shim check-cupti-guard pass.
• RTX 3090, driver 610.57.04, torch 2.14.0+cu130 (CUPTI 13.0.85), CUDA 13.3 toolkit on LD_LIBRARY_PATH, shim only, the same setup that crashed 20 of 20:
  • 10 of 10 runs completed (5 continuous, 5 serialized), each with nvperf=pinned to the wheel's copy, ctx_enabled=1 and about 119k PCs per continuous run.
  • With the toolkit's copy forced in with LD_PRELOAD: the run completed with pc_sampling=off tier_refused=1 nvperf=mismatched, and the log names both paths.
  • A toolkit-only process (toolkit libcupti, serialized tier): nvperf=pinned to the toolkit's own copy, ctx_enabled=1, about 9.8k PCs.

Not in this PR

The continuous tier still deadlocks a process that unloads modules from more than one thread. That is #214.