gh-149816: #96 Fix a race condition in invoke_gc_callback with free threading by dzaramelcone · Pull Request #150029 · python/cpython

dzaramelcone · 2026-05-18T21:22:52Z

The optimization that avoided allocating an info dict when no gc callbacks were registered introduces a time of check to time of use race in free threaded mode (lineno 1943-1955). I just pulled out the check. Since info is always non-null, I also replaced XDECREF use with DECREF. (lineno 1959, lineno 1982).

I implemented a stress test for this and uncovered what I believe to be a bug with _PyList_GetItemRef. I am following suit with python/cpython#149909 and not including the test with my PR for two reasons:

My implementation was resource intensive and not deterministic.
I didn't want to scope creep fixing the bug with _PyList_GetItemRef (but I'm happy to open an issue for it!)

also, this is my first PR against CPython, I am doing it from PyCon US as part of the sprint! :) Hello world!

Issue: 22 free-threading race conditions #149816

python-cla-bot · 2026-05-18T21:22:57Z

All commit authors signed the Contributor License Agreement.

encukou · 2026-05-19T18:41:56Z

Could you add the test, since you already wrote it? It should be deterministic if it the bug if fixed, ritght?

If it's resource-intensive, you can mark is with @support.requires_resource('cpu').

dzaramelcone · 2026-05-20T21:46:15Z

Could you add the test, since you already wrote it? It should be deterministic if it the bug if fixed, ritght?

If it's resource-intensive, you can mark is with @support.requires_resource('cpu').

Thank you for the review Petr!

That test is flaky and relies on basically random permutations of thread ordering to reproduce it.

however, now that I think of it, the original issue included a test that reliably reproduced the issue such that TSAN would flag it, so I will just include that one.

… with free threading

dzaramelcone · 2026-05-29T03:29:46Z

Could you add the test, since you already wrote it? It should be deterministic if it the bug if fixed, ritght?

If it's resource-intensive, you can mark is with @support.requires_resource('cpu').

@encukou added!

encukou · 2026-06-01T13:38:25Z

Thank you!

Did you consider if (PyList_GET_SIZE(gcstate->callbacks) == 0) { return; } just after the PyList_CheckExact?
The optimization seems worth keeping: the callbacks list will commonly be empty. Tuple & string creation are fast, but still much heavier than checking the size.

dzaramelcone · 2026-07-11T13:12:35Z

Thank you!

Did you consider if (PyList_GET_SIZE(gcstate->callbacks) == 0) { return; } just after the PyList_CheckExact? The optimization seems worth keeping: the callbacks list will commonly be empty. Tuple & string creation are fast, but still much heavier than checking the size.

Hey Petr! Sorry for the delay.

That looks like it should be fine to me, so I added the early return.

Thank you for the review! Hope you are well!

bedevere-bot · 2026-08-19T08:36:55Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit 73d0954 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F150029%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

encukou · 2026-08-20T14:21:43Z

Thank you for the fix!

miss-islington-app · 2026-08-20T14:23:00Z

Thanks @dzaramelcone for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington-app · 2026-08-20T14:23:00Z

Thanks @dzaramelcone for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

miss-islington-app · 2026-08-20T14:23:03Z

Sorry, @dzaramelcone and @encukou, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 91d71dd67074d4599b6bd49cc933f41f8bd57058 3.14

bedevere-app · 2026-08-20T14:23:21Z

GH-156119 is a backport of this pull request to the 3.15 branch.

bedevere-bot · 2026-08-20T14:49:18Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable 3.x (tier-2) has failed when building commit 91d71dd.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/125/builds/9186) and take a look at the build logs.
Check if the failure is related to this commit (91d71dd) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/125/builds/9186

Failed tests:

test_dtrace

Failed subtests:

test_gc - test.test_dtrace.SystemTapNormalTests.test_gc
test_gc - test.test_dtrace.SystemTapOptimizedTests.test_gc
test_function_entry_return - test.test_dtrace.SystemTapNormalTests.test_function_entry_return
test_function_entry_return - test.test_dtrace.SystemTapOptimizedTests.test_function_entry_return

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 441, in test_function_entry_return
    self.run_case("call_stack")
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_yjx6hwy3/tmpl2dwhy4o.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/call_stack.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 441, in test_function_entry_return
    self.run_case("call_stack")
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_x1oi_exf/tmp474ax30_.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python -O -O /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/call_stack.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 470, in test_gc
    self.run_case("gc")
    ~~~~~~~~~~~~~^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_x1oi_exf/tmpsd1smwvj.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python -O -O /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/gc.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 470, in test_gc
    self.run_case("gc")
    ~~~~~~~~~~~~~^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_x1oi_exf/tmpq39cr703.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/gc.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 441, in test_function_entry_return
    self.run_case("call_stack")
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_yjx6hwy3/tmp46pc980_.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python -O -O /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/call_stack.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 470, in test_gc
    self.run_case("gc")
    ~~~~~~~~~~~~~^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_yjx6hwy3/tmp9ipauu79.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python -O -O /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/gc.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 441, in test_function_entry_return
    self.run_case("call_stack")
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_x1oi_exf/tmpe9btg6dk.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/call_stack.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 470, in test_gc
    self.run_case("gc")
    ~~~~~~~~~~~~~^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 436, in run_case
    actual_output, expected_output = self.backend.run_case(
                                     ~~~~~~~~~~~~~~~~~~~~~^
        name, optimize_python=self.optimize_python)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 138, in run_case
    actual_output = normalize_trace_output(self.trace_python(
                                           ~~~~~~~~~~~~~~~~~^
        script_file=abspath(name + self.EXTENSION),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        python_file=abspath(name + ".py"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        optimize_python=optimize_python))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 180, in trace_python
    return self.trace(script_file, subcommand, timeout=60,
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      check_returncode=True)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 241, in trace
    return super().trace(
           ~~~~~~~~~~~~~^
        generated_script_file,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        check_returncode=check_returncode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_dtrace.py", line 169, in trace
    raise AssertionError(
    ...<2 lines>...
    )
AssertionError: Command "stap -g /tmp/test_python_yjx6hwy3/tmp4du4raij.stp -c '/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python /var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/dtracedata/gc.py'" failed with exit code 1: output='/var/lib/buildbot/worker/cstratak-fedora-stable-aarch64/3.x.cstratak-fedora-stable-aarch64/build/build_oot/python: error while loading shared libraries: libpython3.16d.so.1.0: cannot open shared object file: No such file or directory\nWARNING: Child process exited with status 127\nWARNING: /usr/bin/staprun exited with status: 1\nPass 5: run failed.  [man error::pass5]\n'

bedevere-bot · 2026-08-20T15:09:39Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Stable Clang Installed 3.x (tier-3) has failed when building commit 91d71dd.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1645/builds/3249) and take a look at the build logs.
Check if the failure is related to this commit (91d71dd) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1645/builds/3249

Summary of the results of the build (if available):

Click to see traceback logs

Note: switching to '91d71dd67074d4599b6bd49cc933f41f8bd57058'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 91d71dd67074 gh-149816: #96 Fix a race condition in invoke_gc_callback with free threading (GH-150029)
Switched to and reset branch 'main'

configure: WARNING: s390x-ibm-linux-gnu/clang is not supported
configure: WARNING:

Platform "s390x-ibm-linux-gnu" with compiler "clang" is not supported by the
CPython core team, see https://peps.python.org/pep-0011/ for more information.


Objects/unicodeobject.c:4906:1: warning: unused function 'ctz' [-Wunused-function]
 4906 | ctz(size_t v)
      | ^~~
1 warning generated.
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:285:5: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  285 |     CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |     ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:292:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  292 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:438:5: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  438 |     CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |     ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:445:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  445 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:551:5: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  551 |     CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |     ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:560:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  560 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:582:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  582 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:593:9: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  593 |         CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |         ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:690:11: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  690 |           CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |           ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:735:5: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  735 |     CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |     ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:761:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  761 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:772:9: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  772 |         CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |         ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:785:11: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  785 |           CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |           ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:930:5: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  930 |     CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |     ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:943:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  943 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:960:5: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  960 |     CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
      |     ^
./Modules/expat/xmltok_impl.c:118:5: note: expanded from macro 'CHECK_NMSTRT_CASES'
  118 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:967:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
  967 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:1202:7: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
 1202 |       CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |       ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
In file included from ./Modules/expat/xmltok.c:312:
./Modules/expat/xmltok_impl.c:1222:11: warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
 1222 |           CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
      |           ^
./Modules/expat/xmltok_impl.c:89:5: note: expanded from macro 'CHECK_NAME_CASES'
   89 |     EXPAT_FALLTHROUGH;                                                         \
      |     ^
./Modules/expat/fallthrough.h:47:33: note: expanded from macro 'EXPAT_FALLTHROUGH'
   47 | #      define EXPAT_FALLTHROUGH __attribute__((fallthrough))
      |                                 ^
19 warnings generated.

make: [Makefile:2954: libinstall] Error 1 (ignored)
ln: failed to create symbolic link 'python3': No space left on device
make: *** [Makefile:2662: bininstall] Error 1

bedevere-bot · 2026-08-20T15:11:08Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Stable LTO + PGO 3.x (tier-3) has failed when building commit 91d71dd.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1627/builds/3227) and take a look at the build logs.
Check if the failure is related to this commit (91d71dd) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1627/builds/3227

Summary of the results of the build (if available):

Click to see traceback logs

Note: switching to '91d71dd67074d4599b6bd49cc933f41f8bd57058'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 91d71dd67074d4 gh-149816: #96 Fix a race condition in invoke_gc_callback with free threading (GH-150029)
Switched to and reset branch 'main'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:3476: clean-retain-profile] Error 1 (ignored)
/tmp/ccAXCyFh.s: Assembler messages:
/tmp/ccAXCyFh.s: Fatal error: can't write 3951 bytes to section .gnu.lto_.ipa_modref.9de3007e39d434a9 of Modules/expat/xmlparse.o: 'No space left on device'
/tmp/ccAXCyFh.s: Fatal error: Modules/expat/xmlparse.o: No such file or directory
make[2]: *** [Makefile:1615: Modules/expat/xmlparse.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/tmp/ccm4Pxse.s: Assembler messages:
/tmp/ccm4Pxse.s: Fatal error: can't write 3951 bytes to section .gnu.lto___jisxcommon_encmap.15.3c48bb678fc154af of Modules/cjkcodecs/_codecs_jp.o: 'No space left on device'
/tmp/ccm4Pxse.s: Fatal error: Modules/cjkcodecs/_codecs_jp.o: No such file or directory
make[2]: *** [Makefile:3765: Modules/cjkcodecs/_codecs_jp.o] Error 1
Assembler messages:
Fatal error: can't create Modules/_posixsubprocess.o: No space left on device
make[2]: *** [Makefile:3781: Modules/_posixsubprocess.o] Error 1
Assembler messages:
Fatal error: can't create Modules/cjkcodecs/multibytecodec.o: No space left on device
make[2]: *** [Makefile:3771: Modules/cjkcodecs/multibytecodec.o] Error 1
Assembler messages:
Fatal error: can't create Modules/mmapmodule.o: No space left on device
make[2]: *** [Makefile:3779: Modules/mmapmodule.o] Error 1
Assembler messages:
Fatal error: can't create Modules/expat/xmltok.o: No space left on device
make[2]: *** [Makefile:1621: Modules/expat/xmltok.o] Error 1
Assembler messages:
Fatal error: can't create Modules/selectmodule.o: No space left on device
make[2]: *** [Makefile:3785: Modules/selectmodule.o] Error 1
Assembler messages:
Fatal error: can't create Modules/unicodedata.o: No space left on device
make[2]: *** [Makefile:3773: Modules/unicodedata.o] Error 1
Assembler messages:
Fatal error: can't create Modules/_decimal/_decimal.o: No space left on device
make[2]: *** [Makefile:3722: Modules/_decimal/_decimal.o] Error 1
make[1]: *** [Makefile:997: profile-gen-stamp] Error 2
make: *** [Makefile:1009: profile-run-stamp] Error 2

bedevere-bot · 2026-08-20T15:12:08Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Stable LTO 3.x (tier-3) has failed when building commit 91d71dd.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1654/builds/3193) and take a look at the build logs.
Check if the failure is related to this commit (91d71dd) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1654/builds/3193

Summary of the results of the build (if available):

Click to see traceback logs

Note: switching to '91d71dd67074d4599b6bd49cc933f41f8bd57058'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 91d71dd67074 gh-149816: #96 Fix a race condition in invoke_gc_callback with free threading (GH-150029)
Switched to and reset branch 'main'

Assembler messages:
Fatal error: can't create Python/hamt.o: No space left on device
make: *** [Makefile:3382: Python/hamt.o] Error 1
make: *** Waiting for unfinished jobs....
Assembler messages:
Fatal error: can't create Python/importdl.o: No space left on device
make: *** [Makefile:2215: Python/importdl.o] Error 1
Assembler messages:
Fatal error: can't create Python/interpconfig.o: No space left on device
make: *** [Makefile:3382: Python/interpconfig.o] Error 1
Assembler messages:
Fatal error: can't create Python/codegen.o: No space left on device
make: *** [Makefile:3382: Python/codegen.o] Error 1
Assembler messages:
Fatal error: can't create Objects/typeobject.o: No space left on device
make: *** [Makefile:3382: Objects/typeobject.o] Error 1
Assembler messages:
Fatal error: can't create Python/initconfig.o: No space left on device
make: *** [Makefile:3382: Python/initconfig.o] Error 1
Assembler messages:
Fatal error: can't create Python/import.o: No space left on device
make: *** [Makefile:3382: Python/import.o] Error 1
Assembler messages:
Fatal error: can't create Python/ceval.o: No space left on device
make: *** [Makefile:3392: Python/ceval.o] Error 1
Assembler messages:
Fatal error: can't create Python/Python-ast.o: No space left on device
make: *** [Makefile:3382: Python/Python-ast.o] Error 1
Assembler messages:
Fatal error: can't create Objects/unicodeobject.o: No space left on device
make: *** [Makefile:3382: Objects/unicodeobject.o] Error 1

bedevere-app Bot mentioned this pull request May 18, 2026

22 free-threading race conditions #149816

Open

24 tasks

dzaramelcone marked this pull request as ready for review May 19, 2026 00:23

bedevere-app Bot added the awaiting review label May 19, 2026

dzaramelcone added 2 commits May 28, 2026 23:19

pythongh-149816: python#96 Fix race condition in invoke_gc_callback…

2306df8

… with free threading

add gc callback mutation race regression test

c2b7dbe

dzaramelcone force-pushed the gh-149816-96 branch from ec90630 to c2b7dbe Compare May 29, 2026 03:26

Merge branch 'main' into pythongh-149816-96

5c71bc1

dzaramelcone and others added 2 commits July 11, 2026 08:37

Merge branch 'main' into pythongh-149816-96

a33bd91

pythongh-149816: Avoid allocating callback info for empty list

062d75a

dzaramelcone and others added 2 commits July 11, 2026 09:14

Merge branch 'main' into pythongh-149816-96

f67ea2b

Merge branch 'main' into pythongh-149816-96

73d0954

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 19, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 19, 2026

encukou merged commit 91d71dd into python:main Aug 20, 2026
138 of 139 checks passed

bedevere-app Bot removed the awaiting review label Aug 20, 2026

encukou added needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Aug 20, 2026

miss-islington-app Bot assigned encukou Aug 20, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Aug 20, 2026

Uh oh!

Uh oh!

Conversation

dzaramelcone commented May 18, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented May 19, 2026

Uh oh!

dzaramelcone commented May 20, 2026

Uh oh!

dzaramelcone commented May 29, 2026

Uh oh!

encukou commented Jun 1, 2026

Uh oh!

dzaramelcone commented Jul 11, 2026

Uh oh!

bedevere-bot commented Aug 19, 2026

Uh oh!

encukou commented Aug 20, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Aug 20, 2026

Uh oh!

miss-islington-app Bot commented Aug 20, 2026

Uh oh!

miss-islington-app Bot commented Aug 20, 2026

Uh oh!

bedevere-app Bot commented Aug 20, 2026

Uh oh!

bedevere-bot commented Aug 20, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Aug 20, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Aug 20, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Aug 20, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dzaramelcone commented May 18, 2026 •

edited by bedevere-app Bot

Loading

python-cla-bot Bot commented May 18, 2026 •

edited

Loading