Skip to content

Unit_hipGraphDestroyNode_DestroyDependencyNode segfault (iGPU, OpenCL) #654

Closed
#2188
Closed
Unit_hipGraphDestroyNode_DestroyDependencyNode segfault (iGPU, OpenCL)#654
#2188
Labels
category:correctnessIncorrect results, crashes and behavioral bugsopenclIssues affecting only the OpenCL backend
Milestone
1.1

Description

@pjaaskel
pjaaskel
opened on Oct 16, 2023
Thread 1 "hipGraphDestroy" received signal SIGSEGV, Segmentation fault.
0x0000555557227010 in ?? ()
(gdb) bt
#0  0x0000555557227010 in ?? ()
#1  0x00007ffff7d3e195 in CHIPGraph::CHIPGraph (this=0x5555572a37b8, OriginalGraph=...) at /home/pjaaskel/src/chipStar/src/CHIPGraph.cc:72
#2  0x00007ffff7d9bf80 in CHIPGraphExec::CHIPGraphExec (this=0x5555572a37b0, Graph=0x5555564de460)
    at /home/pjaaskel/src/chipStar/src/CHIPGraph.hh:682
#3  0x00007ffff7d6060f in hipGraphInstantiate (pGraphExec=0x7fffffffce90, graph=0x5555564de460, pErrorNode=0x0, pLogBuffer=0x0, 
    bufferSize=0) at /home/pjaaskel/src/chipStar/src/CHIPBindings.cc:481
#4  0x00005555555a03a3 in ____C_A_T_C_H____T_E_S_T____43 ()
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/unit/graph/hipGraphDestroyNode.cc:128
#5  0x00005555555c0573 in Catch::TestInvokerAsFunction::invoke (this=0x55555570d9c0)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:14261
#6  0x00005555555bab84 in Catch::TestCase::invoke (this=0x5555564de9a0)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:14100
#7  0x00005555555baaba in Catch::RunContext::invokeActiveTestCase (this=0x7fffffffd6d0)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:12959
#8  0x00005555555b975e in Catch::RunContext::runCurrentTest (this=0x7fffffffd6d0, Python Exception <class 'gdb.error'>: There is no member named _M_p.
redirectedCout=, Python Exception <class 'gdb.error'>: There is no member named _M_p.
redirectedCerr=)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:12932
#9  0x00005555555b8be5 in Catch::RunContext::runTest (this=0x7fffffffd6d0, testCase=...)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:12693
#10 0x00005555555bd283 in Catch::(anonymous namespace)::TestGroup::execute (this=0x7fffffffd6c0)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:13287
#11 0x00005555555bc8ab in Catch::Session::runInternal (this=0x7fffffffd9e0)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:13493
#12 0x00005555555bc697 in Catch::Session::run (this=0x7fffffffd9e0)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:13449
#13 0x00005555555ed23a in Catch::Session::run<char> (this=0x7fffffffd9e0, argc=2, argv=0x7fffffffdc98)
    at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:13171
#14 0x00005555555d23ef in main (argc=2, argv=0x7fffffffdc98) at /home/pjaaskel/src/chipStar/HIP/tests/catch/external/Catch2/catch.hpp:17448

Activity

  1. pjaaskel
    added
    openclIssues affecting only the OpenCL backend
    on Oct 16, 2023
  2. pjaaskel
    added this to the 1.1 milestone on Oct 16, 2023

  3. pvelesko commented on Oct 18, 2023

    @pvelesko
    pvelesko
    on Oct 18, 2023
    Collaborator

    Known to fail on all platforms - seems like bitrot and will not be relevant once Michal's graphs PRs are merged.

  4. pvelesko
    closed this as completedon Oct 18, 2023

  5. pvelesko commented on Oct 14, 2026

    @pvelesko
    pvelesko
    on Oct 14, 2026
    Collaborator

    Reproduced again on current main (c8e69d1) on Aurora PVC with the OpenCL backend: Unit_hipGraphDestroyNode_DestroyDependencyNode_GraphsTest1 segfaults 10/10 when run alone, and it also crashes on Level Zero in a full check.py run. The test only stopped showing up because it was excluded.

    Cause: hipGraphDestroyNode (src/CHIPBindings.cc:1465) only deletes the node. It never calls CHIPGraph::removeNode (src/CHIPGraph.cc:343), and the destructor (src/CHIPGraph.hh:64) does not unlink neighbours. The freed node stays in Nodes_ and in the kernel node's Dependencies_. The test then calls hipGraphInstantiate (CHIPBindings.cc:1547), whose graph copy constructor calls the virtual OriginalNode->clone() on every entry of Nodes_ (src/CHIPGraph.cc:80), so it calls into freed memory.

    gdb backtrace (OpenCL):

    #0  0x0000000001ea9dd0 in ?? ()
#1  CHIPGraph::CHIPGraph (OriginalGraph=...) at src/CHIPGraph.cc:80
#2  hipGraphInstantiate at src/CHIPBindings.cc:1547
#3  hip-tests/catch/unit/graph/hipGraphDestroyNode.cc:148

    At the clone() call the freed node's vptr reads main_arena + 96 (the glibc free-list pointer) instead of the CHIPGraphNodeMemcpy vtable. On Level Zero the freed chunk happens to keep its old vptr, so a single run can pass by luck.

    CUDA defines the counterpart (CUDA Runtime API, Graph Management, cudaGraphDestroyNode): "Removes node from its graph. This operation also severs any dependencies of other nodes on node and vice versa." So both the node removal and the edge removal are required.

    Draft #1864 calls removeNode from hipGraphDestroyNode and also erases the node from the other nodes' dependency lists. Calling the existing removeNode alone would not be enough, since it only erases Nodes_. Related but separate: #1852 (graph destroy leak) and #1868 (clone map). This is a chipStar host-code bug, not LLVM, IGC or a runtime. Reopening.

    This comment was generated using AI

  6. pvelesko
    reopened this on Oct 14, 2026
  7. pvelesko
    added
    category:correctnessIncorrect results, crashes and behavioral bugs
    on Oct 21, 2026

  8. pvelesko commented on Oct 24, 2026

    @pvelesko
    pvelesko
    on Oct 24, 2026
    Collaborator

    Also hit on the AMD rusticl lane (OpenCL, radeonsi) on a fresh build: https://github.com/CHIP-SPV/chipStar/actions/runs/40082797768/job/120839657224, the only failure out of 1430.

    This issue was generated using AI

  9. pvelesko commented on Oct 26, 2026

    @pvelesko
    pvelesko
    on Oct 26, 2026
    Collaborator

    Seen again on the AMD rusticl lane: Unit_hipGraphDestroyNode_DestroyDependencyNode_GraphsTest1 hit SEGFAULT in https://github.com/CHIP-SPV/chipStar/actions/runs/40408611968/job/121872734534 on #2129, which only changes a test script under tests/compiler/promoteInt. The same lane passed on another branch in https://github.com/CHIP-SPV/chipStar/actions/runs/40407849045, so it is intermittent.

    This comment was generated using AI

  10. pvelesko commented on Oct 28, 2026

    @pvelesko
    pvelesko
    on Oct 28, 2026
    Collaborator

    Also reproduces on macOS (Apple M4) with the native Vulkan backend on the Mesa KosmicKrisp driver, chipStar main 5afe718: GraphsTest1 Unit_hipGraphDestroyNode_DestroyDependencyNode exits with SIGSEGV (rc 139) when run alone. The same test passes on the Linux native Vulkan lane (https://github.com/CHIP-SPV/chipStar/actions/runs/40682259076/job/122796067856), so whether it crashes depends on heap layout.

    A likely cause in src/CHIPBindings.cc: hipGraphDestroyNode only does delete on the node. It never calls CHIPGraph::removeNode, and it does not remove the node from the Dependencies_ / Dependendants_ lists of its neighbours (~CHIPGraphNode only clears its own lists). The graph's Nodes_ and the kernel node's dependency list keep a dangling pointer, and hipGraphInstantiate then copies the graph (CHIPGraph::CHIPGraph(const CHIPGraph &), the frame in the backtrace above) and dereferences it.

    This issue was generated using AI

  11. pvelesko
    added 9 commits that reference this issue on Oct 29, 2026

    known_failures: list the macOS KosmicKrisp lane's failures with their…

    23ac18d

    known_failures: list the macOS KosmicKrisp lane's failures with their…

    36572ee

    tests: add reproducer for hipGraphDestroyNode leaving the node in its…

    37e7389

    runtime: hipGraphDestroyNode removes the node from its graph and its …

    ea0ad15

    known_failures: drop Unit_hipGraphDestroyNode_DestroyDependencyNode_G…

    0dc9a14

    tests: add reproducer for hipGraphDestroyNode leaving the node in its…

    dbccf28

    runtime: hipGraphDestroyNode removes the node from its graph and its …

    bfa0e6c

    known_failures: drop Unit_hipGraphDestroyNode_DestroyDependencyNode_G…

    3fd2cbe

    tests: add reproducer for hipGraphDestroyNode leaving the node in its…

    33f0126
  12. pvelesko
    closed this as completedin #2188on Oct 29, 2026
  13. pvelesko
    added a commit that references this issue on Oct 29, 2026

    known_failures: drop Unit_hipGraphDestroyNode_DestroyDependencyNode_G…

    a4eab9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    category:correctnessIncorrect results, crashes and behavioral bugsopenclIssues affecting only the OpenCL backend

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions