cleanup + Libc for gpu by ZuseZ4 · Pull Request #161565 · rust-lang/rust

ZuseZ4 · 2026-08-22T22:31:20Z

Disclaimer: I used an llm to implement the fixes, since I still don't like cmake.
I confirm that on a high-level these are the right fixes as far as I can tell, and I reviewed/refactored it.

After #161431 I rebuilt on another server, and got ld.lld: error: Invalid summary version 14. Version should be in the range [1-12]., since my lld was still on 22, but our submodule now is 23. The issue only shows up when we build the two libomptarget-nvptx and libomptarget-amdgpu bitcode files (or their libompdevice versions), which we have never been using, and results from https://arxiv.org/abs/2608.13759 also show that we don't need them for now. They are meant to be called from within GPU kernels, so their build was always a bit mroe complicated. The first commit gets rid of them.

In theory, we could now even get rid of our clang requirements, but that'd be too easy. So I instead used the second commit to add a new libc-for-gpu component, which allows running (most of?) the standard library on the GPU. Similar to the just removed libompdevice it's meant to be called from kernels, but this time not for optimized algorithms, but for libc functionality. I've tried that previously after our last update to LLVM 22 in #152011, but back then I never got it to work, only to build. This time I also managed to print hello-world from a GPU, so I'm more comfortable merging it. We are not wiring it up yet, and we're also not distributing libc-for-gpu yet, I leave those for a follow-up pr.

closes: #152011

cc @sgasho @Sa4dUs

r? kobzol

rustbot · 2026-08-22T22:31:23Z

This PR changes how LLVM is built. Consider updating src/bootstrap/download-ci-llvm-stamp.

rustc-dev-guide is developed in its own repository. If possible, consider making this change to rust-lang/rustc-dev-guide instead.

cc @BoxyUwU, @tshepang

rustbot · 2026-08-22T22:31:26Z

kobzol is not on the review rotation at the moment.
They may take a while to respond.

ZuseZ4 · 2026-08-22T22:32:27Z

@bors try jobs=dist-x86_64-linux

cleanup + Libc for gpu try-job: dist-x86_64-linux

rust-bors · 2026-08-23T00:01:31Z

💔 Test for 8052db3 failed: CI. Failed job:

try - dist-x86_64-linux (web logs, enhanced plaintext logs)

ZuseZ4 · 2026-08-23T00:27:05Z

File ".../libc/utils/hdrgen/hdrgen/yaml_to_classes.py", line 11, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

cc @jhuber6

@Kobzol It seems like libc is using yaml for code generation. How acceptable is it to add that dependency?

Once a build works, I'll measure the build times, but I'd expect libc-for-gpu to compile very quickly compared to the other components.

jhuber6 · 2026-08-23T01:59:08Z

File ".../libc/utils/hdrgen/hdrgen/yaml_to_classes.py", line 11, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'
cc @jhuber6

@Kobzol It seems like libc is using yaml for code generation. How acceptable is it to add that dependency?

Once a build works, I'll measure the build times, but I'd expect libc-for-gpu to compile very quickly compared to the other components.

Yeah, it's used to generate the headers from config files instead of tons of ifdef like other libc implementations. It used to use LLVM's Tablegen but that dependency made things too difficult to use.

Kobzol

Where should the yaml be installed, into the system Python? We do already use some Python dependencies for tidy, but that requires a non-trivial system for using a lockfile, managing a virtualenv, etc. That's probably overkill for this one. So I guess that just installing pyyaml into the job where you want to run this on CI would be enough for now.

View changes since this review

Kobzol · 2026-08-24T05:54:36Z

+    }
+}
+
+#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq)]


Could you please add a comment that briefly explains what this step does, or rather what it produces?

Kobzol · 2026-08-24T05:55:54Z

+
+        let out_dir = builder.offload_out(target).join(gpu_target);
+        let lib_dir = out_dir.join("lib").join(gpu_target);
+        let files = vec![lib_dir.join("libc.a"), lib_dir.join("libm.a"), lib_dir.join("crt1.o")];


This probably doesn't work on Windows? Is the whole thing Linux-only? If yes, then we should maybe print a warning and skip it on other OSes. Not sure if Offload is also Linux only.

ZuseZ4 · 2026-08-24T14:40:22Z

@bors try jobs=dist-x86_64-linux

ZuseZ4 · 2026-08-24T14:47:45Z

@bors try jobs=dist-x86_64-linux

cleanup + Libc for gpu try-job: dist-x86_64-linux

rust-bors · 2026-08-24T16:32:00Z

💔 Test for 77addc6 failed: CI. Failed job:

try - dist-x86_64-linux (web logs, enhanced plaintext logs)

ZuseZ4 · 2026-08-24T16:45:26Z

@bors try jobs=dist-x86_64-linux

cleanup + Libc for gpu try-job: dist-x86_64-linux

ZuseZ4 · 2026-08-24T17:20:09Z

@Kobzol you're going too fast with your bootstrap cleanups 😂

Kobzol · 2026-08-24T17:32:44Z

The last one wasn't me 😆 But yeah, a lot of refactoring going on in bootstrap right now 😅

rust-bors · 2026-08-24T18:14:11Z

💔 Test for 351bec1 failed: CI. Failed job:

try - dist-x86_64-linux (web logs, enhanced plaintext logs)

ZuseZ4 · 2026-08-24T19:28:26Z

oh fun.

hdrogen seems to use a | operator on two dict's, which is only supported from python 3.9 onwards: "PEP 584 dict union". Iirc we'll update our centos runners at the earliest in December, which should give us a newer Python for free. Not sure if it will be 3.9 though, and I don't want to let this pr hang around for 4 months.

@jhuber6 since it only seems to affect 3 lines, I guess we have a good chance of changing it in libc? Their official baseline is 3.8, but I hope for 10 lines total they wouldn't mind. Since the total PR would be 10 lines, it should also be easy enough to backport.

LLM found of the 3 uses:

Details

``` 3. The only incompatibility is PEP 584 dict union, at exactly three sites, all in hdrgen/header.py: - :142 — self.extra_standards |= other.extra_standards - :177 — descriptions = LIBRARY_DESCRIPTIONS | self.extra_standards - :219 — same expression as :177 ```

jhuber6 · 2026-08-24T19:54:02Z

oh fun.

hdrogen seems to use a | operator on two dict's, which is only supported from python 3.9 onwards: "PEP 584 dict union". Iirc we'll update our centos runners at the earliest in December, which should give us a newer Python for free. Not sure if it will be 3.9 though, and I don't want to let this pr hang around for 4 months.

@jhuber6 since it only seems to affect 3 lines, I guess we have a good chance of changing it in libc? Their official baseline is 3.8, but I hope for 10 lines total they wouldn't mind. Since the total PR would be 10 lines, it should also be easy enough to backport.

LLM found of the 3 uses:
Details
3. The only incompatibility is PEP 584 dict union, at exactly three sites, all in hdrgen/header.py: - :142 — self.extra_standards |= other.extra_standards - :177 — descriptions = LIBRARY_DESCRIPTIONS | self.extra_standards - :219 — same expression as :177

Sounds reasonable to me, I don't think it makes a big difference either way

…obzol Offload cmake cleanups r? kobzol Split out of rust-lang#161565, which requires (minor) LLVM changes. These cleanups are valuable on their own already, and shouldn't be blocked. So let's land it, before it get's stale with all the refactoring going on ^^ Copied the disclaimer: I used an llm to implement the fixes, since I still don't like cmake. I confirm that on a high-level these are the right fixes as far as I can tell, and I reviewed/refactored it.

Rollup merge of #162196 - ZuseZ4:offload-cmake-cleanups, r=Kobzol Offload cmake cleanups r? kobzol Split out of #161565, which requires (minor) LLVM changes. These cleanups are valuable on their own already, and shouldn't be blocked. So let's land it, before it get's stale with all the refactoring going on ^^ Copied the disclaimer: I used an llm to implement the fixes, since I still don't like cmake. I confirm that on a high-level these are the right fixes as far as I can tell, and I reviewed/refactored it.

Offload cmake cleanups r? kobzol Split out of rust-lang/rust#161565, which requires (minor) LLVM changes. These cleanups are valuable on their own already, and shouldn't be blocked. So let's land it, before it get's stale with all the refactoring going on ^^ Copied the disclaimer: I used an llm to implement the fixes, since I still don't like cmake. I confirm that on a high-level these are the right fixes as far as I can tell, and I reviewed/refactored it.

rustbot · 2026-11-02T16:20:49Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

rustbot added A-rustc-dev-guide Area: rustc-dev-guide S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Aug 22, 2026

rustbot assigned Kobzol Aug 22, 2026