[Bug] RTX LiDAR GPU pagefault / ERROR_DEVICE_LOST with auxiliary output FULL near robot geometry (6.2.0-rc.14)

[junminkyung]

Title: [Bug] RTX LiDAR GPU pagefault / ERROR_DEVICE_LOST with auxiliary output FULL near robot geometry (6.2.0-rc.14)

Description

An RTX LiDAR created with isaacsim.sensors.experimental.rtx.Lidar.create intermittently crashes the GPU (GPU pagefault → VkResult: ERROR_DEVICE_LOST, process exits) right after rendering starts. It happens when the auxiliary output level is FULL and the sensor sits at a particular pose next to our mobile-robot geometry. The same scene with aux output BASIC or EXTRA did not crash in our trials.

It also reproduces with the shipped Example_Rotary profile set to FULL, so it does not depend on our custom emitter-state profile. (Not related to the emitter-state progression in #828 as far as we can tell; mentioned only for context.)

Detailed Description

• One LiDAR prim at /World/L (outside the robot hierarchy), placed with a copied world transform. Physics, cameras and other sensors disabled. No ROS, no Docker, no custom ray tracing.
• Each trial = one fresh process, run one at a time, nothing else on the GPU. Crash = log contains GPU pagefault / ERROR_DEVICE_LOST.

Condition (each 5 trials)	GPU crashes
Rear mount pose, our profile, aux FULL	4/5
Rear mount pose, Example_Rotary, aux FULL	4/5
Rear mount pose, aux BASIC	0/5
Rear mount pose, aux EXTRA	0/5
Four sensors on two robots, aux FULL	4/5
Four sensors on two robots, aux BASIC	0/5
Front mount pose of the same robot, aux FULL	0/5
Rear pose, robot prims deactivated (rest of scene kept)	0/5
Rear pose, only the robot kept (environment and vehicle removed)	2/5
Rear pose, only the sensor housing/bracket meshes kept	0/5
Rear position with identity rotation / front position with 180° yaw	4/5 / 0/5
Rear pose, /renderer/raytracingMotion/enabled=false	4/5
Rear pose, multi-tick rate and per-sensor tick TLAS disabled	4/5
Rear pose, on the second GPU	3/5
Rear pose, origin raised 1 mm	3/5
Asset-free scenes (cube; 97,200-triangle box with no/unindexed/indexed normals), FULL	0/5 each

What we could not establish: whether HitNormal or Velocity (both FULL-only) is involved, which mesh/triangle triggers it, and whether 6.1 behaves differently (not tested). The nearest robot triangle is ~10.4 mm from the sensor origin; we did not find the origin lying on a surface. Adding/expanding authored normals on the robot meshes did not remove the crash.

Steps to Reproduce

1. Load a scene containing the robot geometry. We could not reproduce it with simple geometry (see table). The scene and robot are proprietary company assets, so we cannot share them.
2. Create a LiDAR from the default profile with FULL auxiliary output at the rear-mount pose and render:

from isaacsim import SimulationApp
app = SimulationApp({"headless": True, "enable_motion_bvh": True})
import omni.usd
from pxr import UsdGeom, Gf
from isaacsim.core.api import SimulationContext
from isaacsim.sensors.experimental.rtx import Lidar, LidarSensor

stage = omni.usd.get_context().get_stage()
# ... load the robot scene here ...
lidar = Lidar.create("/World/L", config="Example_Rotary", tick_rate=10, aux_output_level="FULL")
xf = UsdGeom.Xformable(stage.GetPrimAtPath("/World/L")); xf.ClearXformOpOrder()
xf.AddTransformOp().Set(REAR_MOUNT_WORLD_MATRIX)   # Gf.Matrix4d of the rear sensor mount
sensor = LidarSensor(lidar, annotators=["generic-model-output"])
sim = SimulationContext(physics_dt=0.01, rendering_dt=0.1, stage_units_in_meters=1)
app.update(); sim.play()
for _ in range(30):
    app.update()
    sensor.get_data("generic-model-output")
app.close()

3. In about 4 of 5 launches the process crashes within the first rendered frames. With aux_output_level="BASIC" or "EXTRA" it completes.

Error Messages

crash_log_excerpt.txt

[Error] [omni.rtx] GPU crash is detected. Trying to write shader debug info into: ...nvdbg
[Warning] [omni.rtx] GPU pagefault occured on virtual address(0x0000007800087000)
[Warning] [omni.rtx] GPU pagefault no associated data found closest allocations were: ...
[Error] [omni.rtx] VkResult: ERROR_DEVICE_LOST
[Error] [omni.usd.multitick.render] Sensor beginFrame() failed
[Error] [omni.rtx] A GPU crash occurred. Exiting the application...

The crash information identifies the shader as rtx/raytracing/rtxsensor/RtxSensorv30.rgs.hlsl, rtxSensorRaygen ("Ray generation shader for RTXSensor V2.0"). We have .nv-gpudmp / .nvdbg files and can share the dumps (not the scene) if useful.

Questions

1. Does FULL auxiliary output (HitNormal / Velocity) have extra requirements on scene geometry or mesh attributes, or is this a known issue in this build?
2. Can you identify the faulting buffer from the GPU dump if we send it?
3. Until a fix, is BASIC/EXTRA the recommended way to avoid it? We would need hit normals for an incidence-dependent return model.

Isaac Sim version

6.2.0-rc.14+release.52805.02189a84 (Kit 110.4.0, omni.sensors.nv.lidar 3.4.0)

Operating System (OS)

Ubuntu 22.04.5 LTS

GPU Name

2× NVIDIA GeForce RTX 3090 Ti (one active GPU per process)

GPU Driver and CUDA versions

Driver 580.159.04, CUDA 13.0 (nvidia-smi)

Logs

Full logs and GPU dumps available on request.

[vick-yu]

Thanks for the detailed report and systematic testing matrix -- that's exactly the kind of data that helps narrow down these issues.

---

 

Reproduction attempt

 
I ran several test configurations on a DGX Spark with aux_output_level="FULL" and Example_Rotary profile. All test runs completed successfully.

What's likely happening

 
Looking at your crash log more closely:
 

GPU pagefault occured on virtual address(0x00000006075b8000)
GPU pagefault no associated data found closest allocations were:
1: AddressRange(0x0000000000000000 - 0x0000000000000000) with name(dummy to ignore)
2: AddressRange(0xffffffffffffffff - 0xffffffffffffffff) with name(dummy to ignore)

 
The pagefault address (~25.8 GB into GPU virtual space) isn't a NULL pointer -- it's a valid-looking address, but the allocator has no record of it. The shader (RtxSensorv30.rgs.hlsl, hash 2f90a8e9940632b8) is trying to write to a buffer that was never allocated or was freed before use.
 
The FULL aux level adds two extra output buffers:
 

• hitNormals (surface normal at each hit point)
• velocities (object velocity at each hit point)
   
  The documented behavior for hitNormal output is:
   

Provided if outputHitNormal is set to true, the OmniLidar's aux_output_level is FULL, and --/app/sensors/nv/lidar/publishNormals=true is set.
 
Since publishNormals defaults to false, there may be a mismatch between the shader expecting to write normals and the backing buffer not being allocated. The geometry and pose dependency you observed fits this: the crash only triggers when rays actually hit surfaces (requiring a buffer write) rather than missing all geometry.
 

Suggested workaround

 
If you need hit normals for your incidence-dependent return model, try explicitly enabling publishNormals:
 

import carb.settings
carb.settings.get_settings().set("/app/sensors/nv/lidar/publishNormals", True)

 
Or via command line:

./python.sh your_script.py --/app/sensors/nv/lidar/publishNormals=true

 
This may resolve the crash -- or at least confirm the hypothesis.
 
If this doesn't help or isn't practical, using aux_output_level="EXTRA" is the reliable workaround until a fix lands.
 

Driver note

 
Your driver (580.159.04) is the DGX Spark version. For x86_64 desktop GPUs like the RTX 3090 Ti, Isaac Sim 6.2.0 recommends driver 595.58.03. While likely not the root cause of this specific issue, it's worth updating if you continue to see GPU stability problems.
 

Next steps

 

1. Try publishNormals=true and let us know if it changes the crash behavior.
    
2. Minimal reproduction scene: Since I couldn't reproduce with NovaCarter or synthetic geometry, could you try to create a minimal scene using public Isaac Sim assets (e.g., robots from /Isaac/Robots/) that triggers the crash? This would help determine whether the issue is specific to your mesh topology or a more general bug. If you can find a public asset + pose combination that crashes, that would make debugging much easier.

[junminkyung]

repro_rectlight_full.py

Thanks again for the suggestions. We found the trigger and have a minimal repro that uses only public content.

Root cause in our scene: two small RectLights (70 × 64 mm marker backlights, visibleInPrimaryRay = true) mounted 0.12 m and 0.64 m beside the rear LiDAR. Turning them off, or setting only visibleInPrimaryRay = false, takes our scene from 4/5 crashes to 0/5. Every other change we tried (mesh normals, materials, transforms, removing robot meshes, publishNormals=true) left the crash in place.

Minimal repro (attached repro_rectlight_full.py): empty stage, stock Example_Rotary profile (converted from the bundled json), aux output FULL, and one RectLight at 0.12 m from the sensor. No other geometry.

Condition (fresh process each run)	GPU crash
FULL + RectLight	5/5
FULL + RectLight, visibleInPrimaryRay = false	0/3
EXTRA + RectLight	0/3
FULL, no light	0/3
FULL + RectLight, enable_motion_bvh = False	3/3
FULL + RectLight, supportMultiTickRate / perSensorTickTlas off	3/3
FULL + RectLight, multi_gpu = True	0/3

The crash happens on the first frame after play() (GPU pagefault ... no associated data found, then VK_ERROR_DEVICE_LOST in rtxSensorRaygen, same as in the original post). It needs multi_gpu = False: with multi-GPU enabled we have not seen it on this machine (2 × RTX 3090 Ti, driver 580.159.04, Isaac Sim 6.2.0-rc.14). We have not tried the recommended 595 driver yet.

Commands:

./python.sh repro_rectlight_full.py                 # crashes
./python.sh repro_rectlight_full.py --primary 0     # ok
./python.sh repro_rectlight_full.py --aux EXTRA     # ok
./python.sh repro_rectlight_full.py --multi-gpu 1   # ok on our machine

We use EXTRA for now. Since the lights must stay visibleInPrimaryRay for our camera simulation, a fix on the FULL path (or a way to exclude lights from RTX LiDAR rays) would help us. We can share the GPU dump from a crashing run of this public repro if useful.

[vick-yu]

Thanks for the outstanding isolation work. Reproducing with just an empty stage, Example_Rotary, and a single RectLight is exactly what we needed. We have reproduced the issue and will get back to you as soon as we have an update.

[junminkyung]

Thank you. I will waiting for your update.

[vick-yu]

An internal ticket (7009783) has been created to resolve this issue.

[junminkyung]

Thank you for the update and for creating an internal ticket to address this issue. I appreciate your support and look forward to future updates!