[V80][service shell] `06_dcmac` stalls while programming the service-layer PDI and leaves the board requiring reset

[think-gaoxing]

Summary

I've tried to get example 06 work, but failed for several commits, including main branch and dev branch. Can someone provide a branch or release to get a working example? Any help will be grateful.

Details:

Programming the hardware VBIN produced by examples/06_dcmac on an Alveo V80 fails during the first (service-layer) partial PDI, before the slash partial PDI is sent and before either traffic-producer kernel starts.

vrtd submits the entire 12,285,968-byte PDI as one H2C MM QDMA request to the SBI keyhole at 0x102100000. After the fixed 10-second QDMA wait, only 6,320,128 bytes have transferred and the request returns EIO. At the same time, the AMC heartbeat expires and the upstream PCIe root port reports Completion Timeout AER. AER recovery fails for PF0, PF1, and PF2, and the kernel reports that a hot reset is required.

This is disruptive: the application cannot simply be rerun. The V80 must be reset/re-enumerated, and in some cases a cold reboot is required to return PLM/SBI to a known state.

Expected behavior

Both partial PDIs in the DCMAC VBIN should load successfully, the device should remain responsive, and 06_dcmac should start and complete both producer kernels.

Actual behavior

The first service-layer PDI fails partway through programming. The host application reports an internal vrtd/I/O error (or appears hung on an unbounded kernel wait with the baseline host application), and the board enters a state with AMC heartbeat failure and PCIe Completion Timeout errors.

SLASH revisions

Baseline/upstream revision used to build the shell and original example:

63c7000cac6706de7e64f3aca8ea21ae2e8f2c30
Fix 200Gb/s interface indexing (#209)

The shell reports the same revision with a dirty marker because it was built locally:

Shell build commit: 0x63c7000cac6706d (dirty)

The exact captured failure was run from commit bc22480a after adding an RM NoC-aperture compatibility guard and a bounded kernel wait. Those changes did not modify vrt/vrtd/src/design_writer.c, and the service-layer PDI that failed is byte-for-byte identical to the one in the original baseline VBIN:

c40f5bd03de906188a3cf5060836f0a927dee6499e3dc321ea08580a666cbf1a  service-layer partial PDI in dcmac_hw.vbin
c40f5bd03de906188a3cf5060836f0a927dee6499e3dc321ea08580a666cbf1a  service-layer partial PDI in captured failing package

The failure happens on this first service-layer PDI, so the changed slash RM in the captured package is never programmed.

Environment

Board: AMD/Xilinx Alveo V80 PQ
BDF: 0000:38:00 (PF0 38:00.0, PF1 38:00.1, PF2 38:00.2)
OS: Ubuntu 24.04 (Noble)
Kernel: Linux 6.8.0-22-generic
Vivado: 2025.1, SW Build 6140274, IP Build 6138677
AMI driver: 1.0.0
SLASH driver: 1.0.0
Shell: service, 200 MHz

v80-smi list before reproduction

Board 0000:38:00 OK (PF0: OK) (PF1: OK) (PF2: OK) (VRTD: OK) Shell: service

The board was healthy before programming. examples/01_aximm could be programmed and completed its 1,024-element data check with Test passed on the same shell. It also passed again after recovering the board, which helps isolate the failure to the larger/backpressured DCMAC service-layer programming path rather than general PCIe, QDMA, or BAR access.

Original VBIN

5021c9b31e328e4b8b2d579f48384b7f3f03997eea25b292b97a13a80d1daac5  examples/06_dcmac/build/dcmac_hw.vbin

Archive members:

  8,166,704  images/top_i_slash_slash_dcmac_hw_inst_0_partial.pdi
      5,493  report_utilization_dcmac_hw.xml
      3,355  system_map.xml
 12,285,968  images/top_i_service_layer_service_layer_dcmac_hw_inst_0_partial.pdi
        243  images/top_i_slash_slash_dcmac_hw_inst_0_hw_probes.ltx

v80-smi inspect on the original VBIN

Command:

v80-smi inspect examples/06_dcmac/build/dcmac_hw.vbin

Output:

Vbin examples/06_dcmac/build/dcmac_hw.vbin:
    Platform: HARDWARE
    Clock frequency: 200000000
    Shell required: service
    Utilization:
        slash: LUTs: 24712 (2.95%), FFs: 1973 (0.12%), LUTRAM: 0 (0%), SRL: 1 (0.01%), RAMB36: 0 (0%), RAMB18: 0 (0%), URAM: 0 (0%), DSP: 0 (0%)
            Cells:
                traffic_producer_0 (slash_base_traffic_producer_0_1): LUTs: 195 (0.02%), FFs: 303 (0.02%), LUTRAM: 0 (0%), SRL: 0 (0%), RAMB36: 0 (0%), RAMB18: 0 (0%), URAM: 0 (0%), DSP: 0 (0%)
                traffic_producer_1 (slash_base_traffic_producer_1_1): LUTs: 194 (0.02%), FFs: 303 (0.02%), LUTRAM: 0 (0%), SRL: 0 (0%), RAMB36: 0 (0%), RAMB18: 0 (0%), URAM: 0 (0%), DSP: 0 (0%)
            Slash logic:
                smartconnect_0 (slash_base_smartconnect_0_1): LUTs: 755 (0.09%), FFs: 861 (0.05%), LUTRAM: 0 (0%), SRL: 1 (0.01%), RAMB36: 0 (0%), RAMB18: 0 (0%), URAM: 0 (0%), DSP: 0 (0%)
    Kernel:
        Name: traffic_consumer_0
        Physical address: 0x20200000000
        Argument:
            Index: 0
            Name: rx_flits
            Type: scalar
            Offset: 16
            Range: 32
            Direction: Read
    Kernel:
        Name: traffic_consumer_1
        Physical address: 0x20200010000
        Argument:
            Index: 0
            Name: rx_flits
            Type: scalar
            Offset: 16
            Range: 32
            Direction: Read
    Kernel:
        Name: traffic_producer_0
        Physical address: 0x20200020000
        Argument:
            Index: 0
            Name: flits
            Type: scalar
            Offset: 16
            Range: 32
            Direction: Write
        Argument:
            Index: 1
            Name: dest
            Type: scalar
            Offset: 24
            Range: 8
            Direction: Write
    Kernel:
        Name: traffic_producer_1
        Physical address: 0x20200030000
        Argument:
            Index: 0
            Name: flits
            Type: scalar
            Offset: 16
            Range: 32
            Direction: Write
        Argument:
            Index: 1
            Name: dest
            Type: scalar
            Offset: 24
            Range: 8
            Direction: Write

Steps to reproduce

1. Check out the baseline revision and initialize its submodules:

   git checkout 63c7000cac6706de7e64f3aca8ea21ae2e8f2c30
   git submodule update --init --recursive

2. Build/install the SLASH driver, vrtd, VRT, and v80-smi following the repository README. Start vrtd.

3. Build or install a service static shell from the same revision. The reproduced shell reported:

   Shell loaded: service
   Shell build commit: 0x63c7000cac6706d (dirty)
   Clock frequency: 200000000
   

4. Build example 6 with Vivado/Vitis 2025.1 and a DCMAC license:

   source /mnt/tools_disk/Xilinx/2025.1/Vivado/settings64.sh
   export XILINXD_LICENSE_FILE=/path/to/dcmac.lic
   cmake -S examples/06_dcmac -B examples/06_dcmac/build -G Ninja -DSLASH_USE_REPO=ON
   cmake --build examples/06_dcmac/build --target dcmac_hw -j 32
   cmake --build examples/06_dcmac/build --target 06_dcmac -j 32

5. Confirm the board is healthy and inspect the package:

   v80-smi list
   v80-smi query --device 38:00
   v80-smi inspect examples/06_dcmac/build/dcmac_hw.vbin

6. In separate terminals, capture daemon and kernel logs:

   sudo journalctl -fu vrtd.service
   sudo dmesg -wT

7. Run the example:

   examples/06_dcmac/build/06_dcmac \
     38:00 examples/06_dcmac/build/dcmac_hw.vbin

8. Observe that programming fails during the first service-layer PDI. Neither producer kernel is started. Do not immediately rerun the command: the partial SBI stream leaves the device in an unknown state and the kernel requests a hot reset.

vrtd output from the failure

2026-09-02T00:12:04+00:00 vrtd[2577]: Design write submitted dev=0 uid=1006 conn_id=11
2026-09-02T00:12:04+00:00 vrtd[2577]: Design writer transfer starting
2026-09-02T00:12:04+00:00 vrtd[2577]: Transferring design writer payload to device offset 0x102100000 (bytes=12285968 buffer_bytes=12288000 keyhole_bytes=4096)
2026-09-02T00:12:14+00:00 vrtd[2577]: Failed to transfer design writer payload: Input/output error
2026-09-02T00:12:14+00:00 vrtd[2577]: Failed to transfer design writer payload
2026-09-02T00:12:14+00:00 vrtd[2577]: Design writer transfer failed: Input/output error
2026-09-02T00:12:14+00:00 vrtd[2577]: Design write failed (error=5) for uid=1006 conn_id=11

Relevant dmesg output

2026-09-02T00:12:04+00:00 ami: CRITICAL WARNING: cmd id: 14 timed out(timeout), hot reset is required
2026-09-02T00:12:04+00:00 ami 0000:38:00.0: ERROR: Submitted command timed out
2026-09-02T00:12:04+00:00 ami: ERROR: Failed to get the heartbeat msg!
2026-09-02T00:12:04+00:00 ami: ERROR: AMC Heartbeat expired event received
2026-09-02T00:12:06+00:00 pcieport 0000:37:01.0: AER: aer_status: 0x00004000, aer_mask: 0x00188020
2026-09-02T00:12:06+00:00 pcieport 0000:37:01.0:    [14] CmpltTO                (First)
2026-09-02T00:12:06+00:00 pcieport 0000:37:01.0: AER: aer_layer=Transaction Layer, aer_agent=Requester ID
2026-09-02T00:12:06+00:00 pcieport 0000:37:01.0: AER: aer_uncor_severity: 0x00463010
2026-09-02T00:12:06+00:00 ami 0000:38:00.0: AER: can't recover (no error_detected callback)
2026-09-02T00:12:06+00:00 slash_qdma 0000:38:00.1: AER: can't recover (no error_detected callback)
2026-09-02T00:12:06+00:00 slash_ctl 0000:38:00.2: AER: can't recover (no error_detected callback)
2026-09-02T00:12:06+00:00 pcieport 0000:37:01.0: AER: device recovery failed
2026-09-02T00:12:10+00:00 ami: ERROR: Heartbeat fail count above threshold! Raising fatal event...
2026-09-02T00:12:10+00:00 ami: ERROR: AMC Heartbeat fatal event received, stopping GCQ...
2026-09-02T00:12:14+00:00 slash:qdma_request_wait_for_cmpl: qdma38001-MM-0: req 0x00000000b0735fa4, W,6320128,0/12285968,0x102100000, done 0, err 0, tm 10000.
2026-09-02T00:12:14+00:00 slash:qdma_descq_dump: qdma38001-MM-0: 0x0/0x0, desc sz 1536/0, pidx 511, cidx 512

Diagnostic observations

• At the baseline revision, vrt/vrtd/src/design_writer.c::transfer_all_at_pos() allocates a buffer for the complete PDI and calls slash_qdma_qpair_transfer() once with the full PDI length.
• The design-writer queue uses ring-size index 9. CPM5's default table maps index 9 to 1537 entries (1536 usable descriptors).
• The failed transfer stops at 6,320,128 bytes, approximately 1543 4-KiB pages, and the dump reports a 1536-descriptor ring with no free entries. This is close enough to the ring boundary to suggest that SBI/PLM backpressure plus a whole-PDI synchronous request exhausts the descriptor ring before the request can complete.
• AMD's XVSEC Versal downloader source explicitly notes that SBI backpressures at some PDI locations while PLM decodes/processes the image. A robust VRT design writer likely needs a backpressure-aware bounded submission strategy rather than one request spanning the entire multi-SLR service-layer PDI.
• The smaller AXIMM programming path succeeds on the same board, shell, driver, and QDMA interface.
• Offline implementation/timing/DFX checks do not surface this runtime failure. The relevant service-layer design routed with no unrouted nets, met timing, and passed pr_verify; the failure occurs during transport/programming.

Recovery impact

After failure, normal management operations may fail because AMI/GCQ is halted. Recovery required stopping vrtd, removing PF0/PF1/PF2 through the existing slash_hotplug interface, toggling secondary-bus reset, rescanning PCIe, restarting vrtd, and waiting for AMC/GCQ readiness. A cold reboot was needed after later diagnostic attempts left PLM/SSIT state uncleared by PCIe hot reset.

[think-gaoxing]

dupliate of #214

[think-gaoxing]

dupliate of #214

[think-gaoxing]

dupliate of #214

[quetric]

Closing this duplicate in favor of #214