[dpsoft]

Fixes #214.

The bug

Under continuous PC sampling, a process that unloads modules from more than one thread deadlocked on every run (15 of 15). There were three threads in the cycle:

1. Application thread: cuModuleUnload holds a driver lock. CUPTI's MODULE_UNLOAD_STARTING callback then calls pc_drain_all, which blocks on the CUPTI guard.
2. Adapter timer: holds the guard across cuptiActivityFlushAll, which waits on a CUPTI pool thread.
3. CUPTI pool thread: waits for the driver lock that the application thread holds.

The unload drain cannot be skipped or deferred: in continuous mode, CUPTI requires a PC-data flush after every load-unload-load, or two different instructions end up sharing a PC identity.

Which edge closes the cycle

I tested this with temporary switches on the reproducer (now shim/nvidia/testdata/cuda_unload.cu):

Timer change	Result
None (control)	3 of 3 hung
Periodic PC drain removed, flush kept under the guard	5 of 5 hung
Activity flush removed	5 of 5 clean
Flush kept, but outside the guard	10 of 10 clean

The flush is the only guarded call that waits on CUPTI's workers. cupti_pcsampling.h says cuptiPCSamplingGetData "never does device synchronization", and the timer's periodic PC drain under the guard never hung. cuptiActivityFlushPeriod is no substitute for the flush, because it only returns full buffers.

The change

• The flush wrapper drops the guard. ActivityFlushAll becomes ActivityFlushAll_NeverUnderGuard, which doesn't take the guard. It counts any call made while the calling thread holds the guard, and the exit report prints that count as cupti_flush_under_guard, which must be 0. Every other CUPTI call stays guarded, and the raw cuptiActivityFlushAll name still doesn't compile.
• The guard's rule changes. It goes from "every call" to "no thread of ours holds the guard across a call that waits on CUPTI's workers". That rule, and why the flush is its one exception, is the new "The flush" section in nvidia/cupti_guard.h. The adapter's comments on the unload drain and on the PC-context lock are updated to match.
• check-cupti-guard. Its compliant example now flushes after the guarded scope closes, not inside it.
• New hardware check. make -C shim check-nvidia-unload runs the reproducer under both PC tiers, 3 times each. It fails on a hang (SIGTERM only, never SIGKILL), on an undrained unload, or on a flush made under the guard. It needs a GPU and CUPTI, so it is not part of make test or CI.

The risk

The flush now runs at the same time as an application thread's guarded PC-sampling calls. That only happens on the unload, context-create and context-destroy callbacks, and on Tier A's start/stop at launch. CUPTI doesn't document whether that pairing is safe. The alternative is a measured permanent hang, and nothing went wrong in any run below.

Reading #99's trace again, it had the same shape: one of our locks held across a call that needed the driver lock, with the flush as the third party, not the cause. The counter means that a future change that puts the flush back under a CallScope shows up in the exit report, not as a hang.

Verification

All runs are shim only, on an RTX 3090 with driver 610.57.04, CUDA 13.3 CUPTI and kernel 6.19.10.

• make -C shim test and make -C shim check-cupti-guard pass.
• check-nvidia-unload: 6 of 6 pass.
  • Mutation check: with the flush wrapped back in a CallScope, it fails on the first continuous run with a hang.
• Reproducer, continuous, 500 iterations × 2 threads: 10 of 10 clean. Each run had module_unload_drains=1000, about 154k PCs, getdata_failed=0 and cupti_flush_under_guard=0.
• PyTorch (torch 2.14.0+cu130):
  • Continuous: 3 of 3 clean.
  • Serialized: 3 of 3 clean, with bursts, start_failed=0 and stop_failed=0.
  • Every run delivered all 41,304 kernel activity records. Buffer counts were 287 to 292, against 294 with the guarded flush.
• The agent was not attached for these runs; the change is confined to the shim.