gpu: the adapter deadlocks the profiled process when a lazy module load computes the cubin CRC during a flush #211

Summary

The NVIDIA adapter (libperfagent-gpu-nvidia.so) can permanently deadlock the profiled process. The agent does not need to be attached: the shim alone, injected via CUDA_INJECTION64_PATH, froze a PyTorch workload in 1 of 8 runs. Every thread ends up in a futex wait, GPU utilisation drops to 0%, and nothing is traced (TracerPid is 0).

With the agent attached, the symptom is a capture showing gpu: batches=0 records=0 and "no GPU activity was observed in this window". That was 4 of 7 runs at PERFAGENT_GPU_SAMPLE_PERIOD=1.

The cycle

From gdb -p <pid> -batch -ex 'thread apply all bt' on a hung process (shim only, no agent):

1. Application thread (pt_autograd_0): cudaLaunchKernel → cuLaunchKernel → libcuda → libcupti → on_callback → CubinQueue::capture → cupti_cubin_crc → perfagent::cupti::GetCubinCrc → blocked on the CUPTI guard's mutex. Under lazy module loading, the launch is what loads the module, so this thread is inside the driver and holds a driver lock when the resource callback fires.
2. Adapter timer thread: Drainer → on_timer_tick → cuptiActivityFlushAll, holding the CUPTI guard, waiting on a semaphore for CUPTI's worker thread.
3. CUPTI worker thread: blocked in libcuda pthread_rwlock_rdlock, on the lock the application thread holds.

Thread 1 waits for the guard, the guard's holder waits for thread 3, and thread 3 waits for thread 1.

Why it exists

This came in with the single-guard design from #99. cupti_cubin_crc deliberately takes the guard, and its comment accepts that it "can wait behind one of the timer thread's CUPTI calls". That trade misses that the waiter is inside the driver, holding a lock that the flush it is waiting behind needs. nvidia/cupti_guard.h already names this exact shape (a guarded call made from inside a CUPTI callback while the flusher holds the guard) as the reason the buffer-completed calls are exempt from the guard. The CRC call made from inside the resource callback is a second instance of it.

Fix direction

Take the CRC off the application thread. CubinQueue already holds its own copy of the cubin, and the drain thread that offers it already runs under the guard, so the CRC can be computed there. After that, no application thread takes the guard from inside a CUDA callback, and every CUPTI call stays guarded.

Environment

RTX 3090, driver 610.57.04, CUDA 13.3 CUPTI (nvidia/cu13/lib/libcupti.so.13 from the torch 2.14.0+cu130 wheel), CPython 3.12.14, kernel 6.19.10, main at c1a43c7.