Skip to content

gh-156132: Relax memory ordering of shared refcount atomics - #156135

Open
kddnewton wants to merge 1 commit into
python:mainfrom
kddnewton:relaxed-acqrel-refcounts
Open

gh-156132: Relax memory ordering of shared refcount atomics#156135
kddnewton wants to merge 1 commit into
python:mainfrom
kddnewton:relaxed-acqrel-refcounts

Conversation

@kddnewton

@kddnewton kddnewton commented Aug 20, 2026

Copy link
Copy Markdown

In the free-threaded build, all operations on ob_ref_shared use sequentially consistent atomics. Full ordering is stronger than the biased reference counting protocol requires, and on ARM64 the ordered instructions are measurably slower under contention.

Increfs need atomicity but no ordering: a thread incrementing ob_ref_shared already holds a valid reference, so the increment only extends the object's lifetime and neither publishes nor consumes any of the object's data. Visibility of the object's contents is provided by whatever operation gave the reference to this thread. Switch the incref paths to relaxed ordering:

  • Py_INCREF and _Py_RefcntAdd: relaxed fetch_add on ob_ref_shared, via a new _Py_atomic_add_ssize_relaxed.
  • _Py_TryIncRefShared and _Py_NewRefWithLock: relaxed CAS, via a new _Py_atomic_compare_exchange_ssize_relaxed.

Decrefs do need ordering, but acquire/release rather than seq_cst. The successful CAS in _Py_DecRefShared releases this thread's accesses through the dying reference, and acquires other threads' released accesses in case this decref is the one that makes the object dead and proceeds to deallocate (or queues it to the owning thread). Switch it to a new _Py_atomic_compare_exchange_ssize_acq_rel with a relaxed failure ordering, since the loop reloads and retries.

On x86-64 the generated code is unchanged (locked RMW instructions are always fully ordered). On ARM64 with clang the decref change is also codegen-neutral (both seq_cst and acq_rel CAS lower to casal); the incref sites lower to ldadd/cas instead of ldaddal/casal. The MSVC ARM64 backend uses the _nf Interlocked variants for the relaxed operations and the _rel variant plus a __dmb(ISHLD) fence for the acq_rel CAS.

I used these benchmarks to verify that it was a performance win on ARM hardware. Note that I saw a pretty significant difference on neoverse v2 (up to 16% on the stress benchmark) but only a minor one on M3 (about 2%). Note that in the webbench you have to be careful for benchmarking that the benchmark doesn't get fully saturated just waiting on put, because that overwhelms the refcount operations. I found that the biggest difference was actually on only 2 threads.

webbench.py
stress.py

pythongh-156132: Relax memory ordering of shared refcount atomics

In the free-threaded build, all operations on ob_ref_shared use
sequentially consistent atomics. Full ordering is stronger than the
biased reference counting protocol requires, and on ARM64 the ordered
instructions are measurably slower under contention.

Increfs need atomicity but no ordering: a thread incrementing
ob_ref_shared already holds a valid reference, so the increment only
extends the object's lifetime and neither publishes nor consumes any
of the object's data. Visibility of the object's contents is provided
by whatever operation gave the reference to this thread. Switch the
incref paths to relaxed ordering:

* Py_INCREF and _Py_RefcntAdd: relaxed fetch_add on ob_ref_shared,
  via a new _Py_atomic_add_ssize_relaxed.
* _Py_TryIncRefShared and _Py_NewRefWithLock: relaxed CAS, via a new
  _Py_atomic_compare_exchange_ssize_relaxed.

Decrefs do need ordering, but acquire/release rather than seq_cst.
The successful CAS in _Py_DecRefShared releases this thread's accesses
through the dying reference, and acquires other threads' released
accesses in case this decref is the one that makes the object dead and
proceeds to deallocate (or queues it to the owning thread). Switch it
to a new _Py_atomic_compare_exchange_ssize_acq_rel with a relaxed
failure ordering, since the loop reloads and retries.

On x86-64 the generated code is unchanged (locked RMW instructions are
always fully ordered). On ARM64 with clang the decref change is also
codegen-neutral (both seq_cst and acq_rel CAS lower to casal); the
incref sites lower to ldadd/cas instead of ldaddal/casal. The MSVC
ARM64 backend uses the _nf Interlocked variants for the relaxed
operations and the _rel variant plus a __dmb(ISHLD) fence for the
acq_rel CAS.
@bedevere-app

bedevere-app Bot commented Aug 20, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants