Skip to content

CI: LLD and a Dev Drive for Windows builds (~11% faster) — stacked on #1077 - #1081

Open
katulevskiy wants to merge 20 commits into
zeronsh:mainfrom
katulevskiy:ci/faster-linkers
Open

CI: LLD and a Dev Drive for Windows builds (~11% faster) — stacked on #1077#1081
katulevskiy wants to merge 20 commits into
zeronsh:mainfrom
katulevskiy:ci/faster-linkers

Conversation

@katulevskiy

@katulevskiy katulevskiy commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Stacked on #1077. That PR is from a fork, so GitHub can't use it as the base here; review only the commits after it. Merge #1077 first.

What

Two changes to the Windows build, the critical path of CI. Everything else this PR tried was measured and dropped (below).

  • LLD instead of link.exe (.github/actions/fast-linker): rust-lld, which ships with the toolchain. It's used by windows-ci-setup and by voice's Windows runtime check.
  • A Dev Drive for ./target (windows-ci-setup): a fresh ReFS virtual disk mounted at target, via samypr100/setup-dev-drive, pinned. Every script and path keeps working.

Each has a switch (fast-linker, dev-drive) that brings back the old behaviour.

Results

Measured in linker-benchmark.yml, now "Windows build benchmark": cold, same commit, side by side, one runner each. Cold builds of the same code vary by several minutes between runs, so only same-run comparisons count.

Windows Test build App build (after the tests)
link.exe, C: (before) 31.7 min 23.1 min
LLD, C: 29.7 min (−6%) 21.5 min (−7%)
LLD, Dev Drive (this PR) 28.2–28.6 min (−11%) 20.6–20.8 min (−11%)

About 3.5 min off the tests job and 2.5 min off the app job, cold. Warm builds after merge save less in absolute terms.

Tried and dropped

  • mold on Linux: no faster than Rust's default LLD on the 4-core runners. Core build 11.1 vs 9.6 min, UI 18.3 vs 13.3 min.
  • Windows tests in the dev profile: the build was slower, 31.5 min vs 23.7 min for the CI release profile, which already compiles the workspace unoptimized.
  • cargo-hakari (workspace-hack):
    • Unifying features changed behaviour: three Cursor attachment tests (attachments_reach_the_shim_as_image_blocks and two others) fail with "image caps not applied", and pass without it.
    • Locally, the first build in a target directory got 11% slower; later builds of other packages got up to 40% faster.
    • Not worth it for CI. It would need its own investigation for local builds.

Notes

  • The benchmark only runs when the Windows build setup changes, or by hand.
  • The Linux core job failed on one run of this PR because the Ubuntu package mirror stalled (four download attempts over 10 min), not because of these changes.

@katulevskiy
CI: one entry workflow with run admission and a fast pull request tier
4779297 
Pull request checks took a median of ~3 hours from push to done, and ~110
minutes of that was jobs waiting for a runner. Each push asked for ~207
runner-minutes on 15-20 jobs, almost all of it compiling, so bursts of
pushes queued hundreds of jobs and every run held a few runners while the
rest of its jobs waited behind other runs.

- ci.yml is the only pull request / main / merge queue entry point. `plan`
  picks the suites a change needs; `admit` waits until every runner the
  run needs is free, then all its jobs start together and each runner is
  released when its job ends (scripts/ci/admit.py, first come first served
  with conservative backfilling, state read from the Actions API so fork
  pull requests work). `CI result` is the one check to require.
- Pull requests get a fast tier: Windows, macOS, voice and Android run only
  when the change looks platform specific (their files, platform
  conditionals or APIs in changed code); `ci:full` or CI_PR_TIER=full runs
  everything. Main and the merge queue run the full tier.
- Voice: 14 `cargo test` builds capped at 2 jobs (~40 min) become core-tests
  additions plus one nextest build for audio and the mobile voice core; the
  workflow no longer runs uncancelled on every push or saves caches from
  pull requests.
- Windows: three test jobs become one nextest build; one shared cache
  instead of four (~4.5 GB of the 10 GB repository cache).
- Linux UI and browser share one job and cache; edge coordinator and vault
  share one job; the macOS app is built with the test dependencies instead
  of a `cargo check` that recompiled everything; iOS no longer waits for
  the frame-source job, which only runs when the GPUI pin can change.
- Only main saves Rust caches.
@katulevskiy
CI admission: cap reservations read from other runs, expire unseen ones
0c73af5 
A pull request can edit its own ci.yml, so the reservation in its admit job
name is untrusted. Cap it at the largest plan.py makes, and stop counting an
admitted run's never-created jobs 90 minutes after admission, so one run
cannot hold more than a full run's runners or hold unused ones for long.
@katulevskiy
CI: reuse passes on unchanged inputs; ci:priority, ci:lock and an eme…
c4b356b 
…rgency workflow

Reuse. plan fingerprints every suite's inputs in the tree under test (the
merge with main for pull requests): workspace manifest, lockfile, toolchain,
CI files, the suite's workflow and non-Rust files, and for Rust suites the
packages it tests plus every workspace package they depend on, plus any file
outside those that their sources name with a ../ path (scanned every run).
Every job records its pass in the Actions cache under that fingerprint as its
last step; a suite whose jobs all passed is skipped and reserves nothing, and
single Windows and voice jobs are skipped on their own. Pull requests trust
their own passes and main's; main and the merge queue trust only main's;
ci:full and manual runs reuse nothing. Callers are split per suite so each
records under its own fingerprint.

Priority and locks (labels, so triage or above). ci:priority puts a pull
request's runs at the front of the admission queue; ci:lock keeps a full
run's runners out of everyone else's budget while the pull request is open
(until unlabelled, merged, closed or CI_LOCK_HOURS), and admits its runs
first. Runs name their pull request in the admit job; the claim counts only
when that pull request's head is the run's commit. ci-emergency.yml (write
access) locks a pull request, cancels the other pull requests' CI runs and
re-runs its own if needed.
@katulevskiy
CI reuse: fail closed
f1b768a 
The merge queue, the last gate before main, reuses nothing. A pull request
that changes a Rust build file no selected suite's fingerprint covers reuses
nothing. Sources that build a path from CARGO_MANIFEST_DIR's parent in code
make their suite read the whole workspace. concat!(.., "/../x") paths and
.cargo/ count as inputs.
@katulevskiy
CI result: fix the result listing (shell quoting)
e70a64d
@katulevskiy
CI: run less on pull requests, more on schedule, and merge tiny test …
c01b1ee 
…binaries

Measured in CI, test execution is a small part of every job; compiling and
starting jobs that run little is the cost. So:

- Voice: only the audio stack (zeron-audio, which nothing else builds yet)
  and the packaged Windows media runtime remain, for their own files. The
  mobile voice/orb tests were a subset of ios-core, the macOS/Windows
  `cargo check` of harness+engine duplicated the platform builds, and the
  voice-media tests join the Windows nextest build.
- Fast tier gating of jobs and steps to their own files: Windows packaging
  (`app`), the iOS simulator tests and their bindings, the Linux and macOS
  browser/preview fixtures. Android builds the APK on pull requests for the
  app, the FFI crate and dependency changes; the iOS mobile-core paths come
  from zeron-mobile's Cargo closure. Gated jobs and optional steps are part
  of the reuse fingerprint.
- GPUI frame-source runs only when the pinned zui revision moves.
- Version bumps (only `version = "..."` lines in Cargo.toml/Cargo.lock)
  select nothing and are left out of fingerprints.
- nextest `ci` profile for pull requests and main leaves out the slow/soak
  tests (48-chat and real-quantum doc_host sync, the codex dying-server
  backoff, preview leak, two sleep-bound discovery tests); the cursor_shim
  stress test runs 14 rounds (ZERON_STRESS_ROUNDS, 100 by default). Preview
  discovery tests run one at a time (fixed port).
- Nightly ci.yml schedule runs everything in full; weekly-tests.yml runs the
  engine integration binaries and crate tests no workflow ran.
- Linux core no longer builds the 8 test binaries that run nothing there
  (Windows-only or all-#[ignore]d). The small harness, engine and preview
  integration suites move into one `tests/misc` binary per crate. The orb
  geometry tests move to zeron-orb (they tested its re-export); the
  assertion-free microbenchmark is dropped. The macOS app builds with the
  tests instead of rebuilding zeron-ui without the fixture feature.
@katulevskiy
CI: pass the nextest config to the build-only steps too
94e579b 
NEXTEST_PROFILE=ci names a profile from scripts/ci/nextest.toml; the macOS and
Windows `--no-run` builds did not load that file, so nextest refused the
unknown profile.
@katulevskiy
Engine: keep the git identity setup inside the home it is given
a78f604 
configure_git_identity pinned HOME but inherited XDG_CONFIG_HOME, and
`git config --global` writes $XDG_CONFIG_HOME/git/config when that file
exists. With a sandbox or test home, the identity landed in the developer's
real git config: running the engine tests on a machine with XDG_CONFIG_HOME
set replaced the user's name and email with the test's. An XDG_CONFIG_HOME
outside the given home now gives way to the home's own, and the test pins it
for its own git calls too.
@katulevskiy
CI: iOS simulator tests in parallel with ios-core; Windows UI tests s…
75d2ed5 
…ide by side

- The C header and module map the simulator tests link are committed in
  apps/ios/CoreFFI (outside Xcode's synchronized folders) and refreshed by
  scripts/ios/build-core.sh like the Swift bindings; ios-core checks all
  three are current. ios-tests no longer waits for ios-core's artifact, so
  the iOS path is one job long instead of two.
- The simulator core for the unit tests builds in the dev profile, not the
  optimized `mobile` profile the app ships.
- Windows runs zeron-ui's tests side by side: nextest gives each its own
  process, so the in-process `--test-threads=1` restriction no longer applies.
@katulevskiy
CI admission: budget defaults at the organization's runner limits
338b5f1 
20 jobs in total and 5 on macOS (the organization's concurrency limits), and
at most 10 on Windows so Linux keeps half the pool. The repository variables
still override them.
@katulevskiy
CI: skip the browser steps when the Linux setup failed
66f2289 
They run after a failed UI test on purpose (a separate signal), but without
the system packages (an apt mirror stall) they can only fail too.
@katulevskiy
CI: retry the WebRTC terminal path test (a known race)
2c44a48 
remote_terminals' network-change test asserts no keystroke goes through the
relay once the path reports Direct; in a loaded run one did. Retried like the
other timing-sensitive tests in nextest.toml until the race is fixed.
@katulevskiy
Fix two races that failed CI on loaded runners
fe122d6 
- Muse: a crash usually shows first as "Muse stdout closed", racing the
  process exit; `try_wait` lost that race under load and users got the bare
  protocol error instead of "Muse exited unexpectedly" with Muse's last
  stderr lines. Wait up to 250 ms for the exit, as the Pi driver already
  does (a_crash_mid_turn_reports_muses_last_words failed this way).
- Remote terminals: a subscription reported TerminalPath::Direct before
  attaching the direct writer, so a keystroke typed by whoever saw Direct
  could still go through the relay (a_terminal_rides_a_real_webrtc_pair_
  through_a_network_change failed this way). Report Direct once writes
  ride the link. The CI retry for that test goes.

Both tests pass 25 runs in a row under CPU load.
@katulevskiy
CI reuse: count only the CI files a suite runs
f9e2650 
Every suite's fingerprint included all of scripts/ci and the composite
actions, so a change to the planner or the scheduler (which decide what runs,
not whether it passes) re-ran every suite. Each suite now counts the CI files
it executes: nextest.toml and the Linux or Windows setup for the Rust suites,
test-core.sh for core, the macOS fixture runner for macOS, apt-install.sh for
the installer. ci.yml still counts for all.
@katulevskiy
docs: changing the fingerprint inputs reruns everything once
58a2509
@katulevskiy
CI admission: no circular waits
176f7e4 
Two ways admission could deadlock (until the fail-open timeout):

- Each waiting run counted every other waiting run's admit job as a busy
  runner, including runs queued behind it. With a full run (16 of 20) at the
  head and five or more waiting behind it, the head waited for their runners
  and they waited for it. A run now counts only the admit runners ahead of it.
- Two ci:lock labels each reserved 16 of 20 runners and counted the other's
  reservation, so neither locked pull request (nor anyone else) could start.
  Locks are now ordered by when their label was added: an older lock ignores
  a newer one's standing reservation.

Every wait now points forward in one order, so no cycle can form. A test
simulates 3000 random queues (sizes, priority, overlapping locks, polling
order) and checks they all drain; the previous scheduler deadlocked in 309.
The fail-open timeout drops from 5 h to 2 h as the backstop for the one case
outside the model (waiting admit jobs taking every runner in the pool).
@katulevskiy
CI: faster linkers (mold on Linux, LLD on Windows), with a side-by-si…
ff2e1d3 
…de benchmark

- .github/actions/fast-linker: mold 3.0 on Linux (pinned, checksum-verified;
  Rust already links with LLD there, mold is faster still: the engine's
  150 MB test binary links in ~0.3 s against ~0.5 s locally), rust-lld
  instead of link.exe on Windows (ships with the toolchain). macOS keeps
  ld-prime. The Linux and Windows setup actions, iOS core and the voice
  Windows runtime use it, before their Rust cache (part of its key); an
  input turns it off.
- linker-benchmark.yml: the same builds (Linux core and UI test binaries;
  Windows test binaries, then the app) from cold, default and fast linker
  each on its own runner at the same time; times in the run summary.
  Cross-run comparisons can't see a few minutes: cold Windows builds of the
  same code took 18.7 to 24 min on different runs.
- test-core.sh builds only with --no-run.
@katulevskiy
CI: Windows Dev Drive and dev-profile tests; cargo-hakari workspace-hack
098ea23 
- Windows builds on a Dev Drive (ReFS on a fresh virtual disk, mounted at
  ./target before the Rust cache restores into it), via
  samypr100/setup-dev-drive (pinned).
- The Windows tests job builds the dev profile (as Linux and macOS do)
  instead of the optimized release build; 8 MB stacks for the main thread
  (/STACK) and test threads (RUST_MIN_STACK), since debug code takes more and
  Windows gives a main thread 1 MB. The app job keeps release; each has its
  own cache now.
- crates/workspace-hack (cargo-hakari): one feature set per third-party
  dependency across the workspace, so builds of different packages in one
  target directory (the Windows app after its tests, local `cargo test -p`
  then `cargo build -p`) reuse compiled dependencies instead of rebuilding
  them with other features. GPUI, its family, naga, criterion and ashpd stay
  out of it: the iOS and Android builds compile crates that depend on the
  hack, and those don't build there. core-tests checks it is current
  (scripts/ci/check-workspace-hack.sh, a pinned cargo-hakari).
- The linker benchmark compares Windows one change at a time: today's
  setup, then LLD, then the Dev Drive, then the dev profile.
@katulevskiy
katulevskiy marked this pull request as draft October 27, 2026 13:04
@katulevskiy
katulevskiy marked this pull request as ready for review October 27, 2026 13:05
@katulevskiy
Cargo.lock: zeron-proto's workspace-hack dependency
d86cf03 
Lost while testing the workspace-hack check; CI builds with --locked.
@katulevskiy
CI: keep what measured faster (LLD and a Dev Drive on Windows), drop …
bec39a8 
…the rest

From linker-benchmark.yml (cold, same commit, side by side) and CI runs:
- LLD instead of link.exe on Windows: test build 31.7 -> 29.7 min, app
  23.1 -> 21.5 min. Kept.
- Dev Drive for ./target on Windows: another 1.5 and 1 min (about 11% off
  in total with LLD). Kept.
- mold on Linux: no faster than Rust's default LLD (core 11.1 against 9.6
  min, UI 18.3 against 13.3 on the 4-core runners). Dropped.
- Windows tests in the dev profile: the build took 31.5 min against 23.7 for
  the CI release profile (which already compiles the workspace unoptimized).
  Reverted.
- cargo-hakari: the unified features changed behaviour (three Cursor
  attachment tests fail with "image caps not applied"; they pass without it),
  and the first build in a target directory got 11% slower. Removed.
The benchmark compares the Windows builds only, in the release profile CI uses.
@katulevskiy katulevskiy changed the title CI: faster linkers (mold on Linux, LLD on Windows) — stacked on #1077 CI: LLD and a Dev Drive for Windows builds (~11% faster) — stacked on #1077 Oct 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wingleeio wingleeio Awaiting requested review from wingleeio wingleeio is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@katulevskiy