Support building wheels for manylinux and OSX by aaronj0 · Pull Request #22 · compiler-research/cppjit

aaronj0 · 2026-08-17T22:52:17Z

Rebased on top of #21 since we need that patch for relocatable wheels.

github-actions · 2026-08-17T23:35:43Z

Test Results

Configuration	Result
macos-26-intel-llvm21-py3.14-cxx20	`= 439 passed, 44 skipped, 85 xfailed, 32 xpassed, 100 warnings in 90.00s (0:01:30) =`
macos-26-llvm21-py3.14-cxx20	`= 445 passed, 30 skipped, 95 xfailed, 30 xpassed, 100 warnings in 73.12s (0:01:13) =`
ubuntu-24.04-llvm21-py3.14-cxx20-vg	`====== 555 passed, 25 skipped, 18 xfailed, 2 xpassed in 95.78s (0:01:35) =======`
ubuntu-24.04-llvm22-py3.14-cxx20	`====== 555 passed, 25 skipped, 19 xfailed, 1 xpassed in 100.01s (0:01:40) ======`
ubuntu-24.04-llvm22-py3.14-cxx20-cling	`====== 545 passed, 25 skipped, 27 xfailed, 3 xpassed in 95.91s (0:01:35) =======`
ubuntu-24.04-llvm22-py3.14-cxx23	`====== 576 passed, 3 skipped, 20 xfailed, 1 xpassed in 107.93s (0:01:47) =======`

aaronj0 · 2026-08-24T15:28:23Z

@vgvassilev A note: The cleanest way to do this is to have wheel-builder images on ci-workflows that provision the image and LLVM build for manylinux and OS X. I will open a PR once I have a working draft locally and update the approach here to trigger that common workflow. That can then, in principle, be reused to build wheels for any other Python package based on LLVM under compiler-research

mcbarton · 2026-08-25T11:08:53Z

+archs = ["arm64"]
+# eigen and boost are test-suite build deps; the LLVM toolchain is the
+# llvm-wheel artifact wheels.yml stages at /opt/llvm before this runs.
+before-all = "brew install eigen boost"


Do this assume you are using homebrew? Not everyone does use it on MacOS. I use Macports for example.

mcbarton · 2026-08-25T11:10:05Z

+environment = { CMAKE_ARGS = "-DLLVM_DIR=/opt/llvm/lib/cmake/llvm -DClang_DIR=/opt/llvm/lib/cmake/clang" }
+
+[tool.cibuildwheel.macos]
+archs = ["arm64"]


I know its being supported less and less, but what about MacOS intel. There is clearly appetite for it by people wanting to use cppjit (see #39 (comment))

mcbarton · 2026-08-25T11:11:12Z

+test-command = "python -m pip install -r requirements.txt && python .github/wheel_smoke.py && cd test && make -j$(sysctl -n hw.ncpu) PYTHON=python && python -m pytest -ra"
+# 14.0 is the toolchain's own deployment floor (set at llvm-wheel build
+# time); nothing brew-built ships in the wheel.
+environment = { CMAKE_ARGS = "-DLLVM_DIR=/opt/llvm/lib/cmake/llvm -DClang_DIR=/opt/llvm/lib/cmake/clang", MACOSX_DEPLOYMENT_TARGET = "14.0" }


I'm a little confused. Where does this /opt/llvm/ build come from?

mcbarton · 2026-08-25T11:11:46Z

+# LLVM/Clang 21 archives, pinned by sha256) at /opt/llvm.
+before-all = "bash {project}/.github/cibw_before_all_linux.sh"
+# CMAKE_ARGS reaches the CppInterOp ExternalProject sub-configure.
+environment = { CMAKE_ARGS = "-DLLVM_DIR=/opt/llvm/lib/cmake/llvm -DClang_DIR=/opt/llvm/lib/cmake/clang" }


Same to MacOS quesion about the /opt/llvm/ build

mcbarton · 2026-08-25T11:19:06Z

+test-command = "python .github/wheel_smoke.py"
+
+[tool.cibuildwheel.linux]
+archs = ["x86_64"]


What about Linux arm?

vgvassilev · 2026-08-25T12:23:02Z

-# and linking LLVMSupport fails at generate time unless find_package(ZLIB)
-# resolves, which needs zlib.h from the zlib package.
-micromamba create -y -p /opt/llvm -c conda-forge 'llvmdev=21.*' 'clangdev=21.*' lld zlib
+asset="llvm-wheel-21.1.8-ubuntu-24.04-x86_64-12af669f7a4444de.tar.zst"


I am still not sure if I understand the special llvm-wheel. How is it different from what we vendor?

Only different in the fact that it is built on top of the manylinux pypa container to ensure it satisfies the glibc floor and "standard" build environment when building wheels. We then vendor it the same way. We will need to have a separate llvm-wheel because we can only support one LLVM version and build config per-wheel. Some projects just do it from scratch with an alma8 container such as Triton which we could adopt but this approach is more reliable since PyPA controls the image there which is well tested against Python package deployment environments.

Does that mean that the other vendored llvms do not satisfy the libc versions?

No because they are source built on ubuntu24 runners. We need to build on a manylinux image to publish the wheels for manylinux. Similarly we will need a new image for manylinux-aarch64

aaronj0 · 2026-08-25T13:01:22Z

@mcbarton A comment: this PR is in development and some of the decisions here are still in discussion. I can request your review at a point things have converged. To answer your current questions:

Yes, Intel mac support is planned but is best-effort since macOS 27 drops support for those platforms. Currently the wheels ship with macOS deployment target 14.0 (the llvm-wheel toolchain's own floor). Linux aarch64 is planned as well.

The goal is to land the current wheel-building infrastructure before extending to more platforms, instead of having the entire thing arrive in one PR. That gives me a chance to verify the system before incrementally extending for more architectures (and more Python versions).

Regarding homebrew for eigen/boost: that's a test-time dependency, not part of how the wheel is built. The suite's probes now cover /opt/homebrew (arm64 brew) and /opt/local (MacPorts) as of the last commit, so whichever package manager provides eigen/boost, the tests find them.

On /opt/llvm: We build the llvm-wheel toolchain from compiler-research/ci-workflows on a manylinux image and OS X runner. Those archives are published on the recipe cache release (https://github.com/compiler-research/ci-workflows/releases/tag/cache) and this toolchain is staged at /opt/llvm by the 'before-all' step on linux and by the setup-recipe step in wheels.yml on macOS. I am still trying to figure out how to optimise this process further so bear that in mind.

aaronj0 · 2026-08-25T15:40:19Z

+test-sources = ["test", "requirements.txt", ".github/wheel_smoke.py"]
+test-command = "python .github/wheel_smoke.py"
+
+[tool.cibuildwheel.linux]


@vgvassilev You can see here that the vendored LLVM is reused exactly as build args on the machine that builds the wheel. Those machines are configured in the matrix in wheels.yml:

- { os: ubuntu-24.04, label: manylinux-x86_64, arch: x86_64 } - { os: macos-26, label: macosx-arm64, arch: arm64 }

So for every new target platform we can add a new row and extend the wheels infra. I hope this addresses your concern of any redundancy here

aaronj0 · 2026-08-25T15:41:01Z

+      - uses: actions/checkout@v7
+
+      # ref pins the recipe content the cache key is computed from.
+      - uses: compiler-research/ci-workflows/actions/setup-recipe@main


Here we reuse the recipe we created with manylinux

aaronj0 · 2026-08-25T15:41:52Z

+      - uses: pypa/cibuildwheel@v4.2.0
+
+      - uses: actions/upload-artifact@v7
+        with:
+          name: wheels-${{ matrix.label }}
+          path: wheelhouse/*.whl
+
+  sdist:
+    name: sdist
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v7


These are standard PyPA cibuildwheel and sdist actions required to generate the Python wheel

vgvassilev

I do not think I am competent enough to review this, but lgtm... @henryiii, do you mind having a look?

aaronj0 changed the title ~~Wheels build test ci~~ Support building wheels for manylinux and OSX Aug 17, 2026

aaronj0 force-pushed the wheels-build-test-ci branch 3 times, most recently from 811d00b to 1b4e02d Compare August 18, 2026 09:03

aaronj0 mentioned this pull request Aug 18, 2026

[interop] Resolve libclangCppInterOp relative to libcppjit at runtime #24

Merged

aaronj0 marked this pull request as draft August 19, 2026 07:33

aaronj0 force-pushed the wheels-build-test-ci branch 2 times, most recently from 74bb3ff to ee821f3 Compare August 20, 2026 11:53

This was referenced Aug 20, 2026

[cppjit] Install valgrind from conda-forge on the valgrind cells compiler-research/ci-workflows#101

Merged

[interop] Detect the resource dir of the required system LLVM #32

Merged

aaronj0 force-pushed the wheels-build-test-ci branch from ee821f3 to 3afd4f7 Compare August 24, 2026 11:04

aaronj0 marked this pull request as ready for review August 24, 2026 11:04

Legend101Zz mentioned this pull request Aug 24, 2026

Brian2 downstream support: wheels, Windows, and migration from cppyy #39

Open

aaronj0 force-pushed the wheels-build-test-ci branch from 3afd4f7 to de3498f Compare August 24, 2026 13:43

aaronj0 requested review from guitargeek and vgvassilev August 24, 2026 14:24

mcbarton reviewed Aug 25, 2026

View reviewed changes

vgvassilev reviewed Aug 25, 2026

View reviewed changes

aaronj0 force-pushed the wheels-build-test-ci branch 3 times, most recently from e94d1e6 to afcfde2 Compare August 25, 2026 13:56

aaronj0 commented Aug 25, 2026

View reviewed changes

vgvassilev reviewed Aug 25, 2026

View reviewed changes

aaronj0 force-pushed the wheels-build-test-ci branch from afcfde2 to e327565 Compare August 25, 2026 16:51

aaronj0 force-pushed the wheels-build-test-ci branch 2 times, most recently from 7e0f1aa to e71d2df Compare August 25, 2026 17:18

aaronj0 added 2 commits August 26, 2026 16:19

[test] Find eigen and boost under the Homebrew and MacPorts prefixes

d4445dc

[ci] Build and test manylinux and macOS arm64 wheels

684e839

aaronj0 force-pushed the wheels-build-test-ci branch from e71d2df to 684e839 Compare August 26, 2026 14:32

Uh oh!

Conversation

aaronj0 commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

aaronj0 commented Aug 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaronj0 Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaronj0 commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vgvassilev left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aaronj0 commented Aug 17, 2026 •

edited

Loading

github-actions Bot commented Aug 17, 2026 •

edited

Loading

aaronj0 Aug 25, 2026 •

edited

Loading

aaronj0 commented Aug 25, 2026 •

edited

Loading