diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 829b37a..78046a1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -61,6 +61,10 @@ Everything goes through one seam, so CI and your machine cannot disagree: scripts/task check # fmt:check, lint, test, build ``` +On macOS, Linux-only checks run in an [OrbStack](https://orbstack.dev) machine: +`scripts/lab setup` once, then prefix the command, as in +`scripts/lab scripts/task build:static`. Firecracker needs KVM, so boots stay in CI. + Never use `--no-verify`. If a check is wrong, fix the check in its own pull request. A bug fix arrives with the test that would have caught it. diff --git a/ROADMAP.md b/ROADMAP.md index c337d2d..cb25f9f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -29,7 +29,7 @@ Releases ship when their gate passes, not on a date. ## v0.1 — Static sites in a microVM -`#·········` 4% · 1 of 24 done +`#·········` 8% · 2 of 25 done Serve a static site from a Firecracker microVM over HTTP/1.1, from cold boot to first byte inside the boot budget, measured in CI. For sites behind a load balancer or CDN that already terminates TLS. @@ -65,6 +65,7 @@ Serve a static site from a Firecracker microVM over HTTP/1.1, from cold boot to ### done - [x] `0012` Scaffold the Cargo workspace and the task seam chore · p0 · s · infra +- [x] `0097` Linux lab for development on macOS chore · p0 · s · infra ## v0.2 — HTTPS and containers diff --git a/cairn/items/0097-linux-lab-for-development-on-macos.md b/cairn/items/0097-linux-lab-for-development-on-macos.md new file mode 100644 index 0000000..34f4385 --- /dev/null +++ b/cairn/items/0097-linux-lab-for-development-on-macos.md @@ -0,0 +1,75 @@ +--- +id: 97 +uid: fcf55ec2-86a4-4d81-9d93-8eb637cc9992 +title: Linux lab for development on macOS +type: chore +status: done +milestone: v0.1 +created: 2026-10-02 +updated: 2026-10-02 +closed_at: 2026-10-02 +priority: p0 +effort: s +area: infra +--- + +## Purpose + +The maintainer works on macOS (Apple silicon). Everything liam ships is Linux: static musl binaries, io_uring, and a kernel image. Without a Linux machine at hand, every Linux-only check (`scripts/task build:static`, io_uring tests, kernel builds) waits on a CI round trip. A local lab makes those a one-line command, and gives parallel agents a shared place to run them without sharing a target directory. + +## Approach + +- An OrbStack machine named `liam-lab` (Ubuntu, aarch64). OrbStack mounts `/Users` at the same path, so the worktree is visible inside the VM unchanged. +- `scripts/lab ` runs a command in the VM at the caller's directory. `CARGO_TARGET_DIR` is set under the VM user's home, derived from the worktree path, so macOS builds, Linux builds and parallel worktrees never share a target directory. +- `scripts/lab setup` provisions idempotently: build-essential, musl-tools, the x86_64 cross gcc and binutils, flex, bison, bc, libelf-dev, libssl-dev, cpio, file, curl, qemu-system-x86; rustup with the toolchain from `rust-toolchain.toml` plus the x86_64 and aarch64 musl targets. Cargo is configured so the x86_64 musl target links with the cross toolchain. +- The VM has no KVM, so nothing here boots Firecracker. That stays on CI. + +## Acceptance criteria + +- [x] `scripts/lab --help` describes the commands, and a missing `orb` or a missing `liam-lab` machine fails with an error naming the fix. +- [x] `scripts/lab setup` provisions a fresh machine, and a second run succeeds without reinstalling anything. +- [x] `scripts/lab cargo test --workspace` passes in the lab. +- [x] `scripts/lab scripts/task build:static` produces static x86_64 `liamd` and `liam-init`. +- [x] io_uring works in the lab: a program calls `io_uring_setup` and completes a request. +- [x] Two worktrees get different target directories, neither of them the macOS `target/`. +- [x] CONTRIBUTING.md tells macOS contributors how to use the lab. + +## 2026-10-02 + +Cross-compiling C for x86_64 musl on an aarch64 machine: x86_64-linux-gnu-gcc alone fails on mimalloc (it finds the aarch64 glibc headers: bits/libc-header-start.h missing). Ubuntu ships musl for the native architecture only, so setup builds musl 1.2.5 (the version in Rust's bundled x86_64 musl libc.a) from upstream source for x86_64 with --enable-wrapper=gcc, into ~/.local/x86_64-linux-musl. The tarball's sha256 a9a118bb...c75e4 matches Ubuntu's musl_1.2.5.orig.tar.gz on both archive.ubuntu.com and ports.ubuntu.com. Cargo config: linker x86_64-linux-gnu-gcc, CC_x86_64_unknown_linux_musl the musl-gcc wrapper. scripts/task build:static now honours CARGO_TARGET_DIR, which scripts/lab sets. + +## 2026-10-02 + +Evidence 1: scripts/lab --help prints usage, exit 0; no args exits 2. With PATH=/usr/bin:/bin: 'scripts/lab: orb not found. Install OrbStack (brew install orbstack), then: orb create ubuntu liam-lab', exit 1. With a stub orb listing another machine: 'scripts/lab: no OrbStack machine named liam-lab. Create it: orb create ubuntu liam-lab && scripts/lab setup', exit 1. A command's exit status passes through (sh -c 'exit 7' gave 7). + +## 2026-10-02 + +Evidence 2: a throwaway machine (orb create ubuntu:resolute liam-lab-fresh) provisioned from nothing by this script with only the machine name changed: exit 0 in 1m33s, installing all packages, rust 1.98.1 with both musl targets and building musl; build:static then passed there. The second setup on it: exit 0 in 0.74s, 'all installed', toolchain 'unchanged', musl not rebuilt. liam-lab itself: setup re-run several times, each exit 0 with nothing reinstalled. The throwaway machine was deleted. + +## 2026-10-02 + +Evidence 3: scripts/lab cargo test --workspace in liam-lab, exit 0: six test binaries, 'test result: ok' for each (2 + 2 + 2 passed, three with 0 tests). + +## 2026-10-02 + +Evidence 4: scripts/lab scripts/task build:static, exit 0: liamd 543392 bytes and liam-init 397880 bytes, both 'ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped'. CI's static job measured 543,400 and 397,880 for the same code. Both run in the lab under OrbStack's Rosetta binfmt: 'liamd 0.0.0', 'liam-init 0.0.0', which exercises mimalloc's C built by the cross musl-gcc. The aarch64-unknown-linux-musl target also builds both binaries statically and they run. + +## 2026-10-02 + +Evidence 5: kernel 7.0.14-orbstack, /proc/sys/kernel/io_uring_disabled = 0, 149 io_uring symbols in /proc/kallsyms including __arm64_sys_io_uring_setup. A 50-line C program (raw syscalls, no liburing) set up a 4-entry ring, submitted IORING_OP_READ of /etc/hostname and reaped it: 'io_uring: features=0x3ffff user_data=42 res=9 data=liam-lab', exit 0. + +## 2026-10-02 + +Evidence 6: CARGO_TARGET_DIR in the lab is /home/oddurs/.cache/liam-lab/target/Users/oddurs/Code/.worktrees/liam/chore/linux-lab for this worktree (and for its crates/liamd subdirectory), and /home/oddurs/.cache/liam-lab/target/Users/oddurs/Code/liam for the primary checkout. After the lab builds the worktree has no target/ directory at all. + +## 2026-10-02 + +Evidence 7: CONTRIBUTING.md, under Checks, says in three lines that on macOS Linux-only checks run in an OrbStack machine via scripts/lab setup and a prefixed command, and that Firecracker boots stay in CI. + +## 2026-10-02 + +Review follow-up: the musl prefix is now versioned (~/.local/x86_64-linux-musl-1.2.5), so bumping musl_version rebuilds rather than keeping old headers; rustup-init is downloaded to a file before it runs, so a failed download fails setup with curl's error. Re-verified: setup on liam-lab built the versioned prefix, the old one was removed, and a clean build:static gave liamd 543392 and liam-init 397880 bytes, static-pie x86-64. A second throwaway machine was provisioned from nothing by the revised script (exit 0), build:static passed on it, and it was deleted. + +## Result + +scripts/lab runs any command in the liam-lab OrbStack machine (Ubuntu 26.04 aarch64, kernel 7.0 with io_uring enabled) at the caller's directory, with CARGO_TARGET_DIR under ~/.cache/liam-lab/target/ so worktrees and macOS never share a target. scripts/lab setup provisions it idempotently in about 1.5 minutes from nothing. x86_64 static builds work: linker x86_64-linux-gnu-gcc, and C dependencies compile against an x86_64 musl 1.2.5 built from source, because Ubuntu has no cross musl. scripts/lab scripts/task build:static gives the same sizes as CI to within 8 bytes. x86_64 binaries run in the lab under Rosetta. No KVM: Firecracker boots stay in CI. Run scripts/task check on macOS, because site/node_modules belongs to macOS. diff --git a/scripts/lab b/scripts/lab new file mode 100755 index 0000000..04e15a8 --- /dev/null +++ b/scripts/lab @@ -0,0 +1,154 @@ +#!/bin/sh +# A Linux machine for the checks macOS cannot run: static musl builds, io_uring, +# kernel builds. It is an OrbStack machine, which mounts /Users at the same path, +# so a worktree is the same directory on both sides and nothing is copied. +# +# The VM has no KVM. Anything that boots Firecracker still runs in CI. +set -eu + +machine=liam-lab +repo=$(cd "$(dirname "$0")/.." && pwd) + +die() { + printf 'scripts/lab: %s\n' "$1" >&2 + exit 1 +} + +say() { + printf '▸ %s\n' "$1" +} + +usage() { + cat <<'USAGE' +usage: scripts/lab + scripts/lab setup + + run a command in the liam-lab machine, in the current directory + setup install the build toolchain in liam-lab; safe to run again + +Builds in the lab get a target directory of their own under the VM user's home, +one per worktree, so they never collide with the macOS target/ or with each +other. Example: + + scripts/lab scripts/task build:static + scripts/lab cargo test --workspace + +Firecracker needs KVM, which the lab does not have; that runs in CI. Run +scripts/task check on macOS: the site's node_modules in the worktree are macOS's. +USAGE +} + +require_machine() { + command -v orb >/dev/null 2>&1 || + die 'orb not found. Install OrbStack (brew install orbstack), then: orb create ubuntu liam-lab' + orb list --quiet | grep -qx "$machine" || + die "no OrbStack machine named $machine. Create it: orb create ubuntu $machine && scripts/lab setup" +} + +# Runs in the VM through `sh -c`, so $HOME is the VM user's home. The target +# directory is keyed on the worktree's path: parallel worktrees stay apart, and +# the same worktree reuses its incremental build. +run() { + require_machine + top=$(git rev-parse --show-toplevel 2>/dev/null || pwd) + # shellcheck disable=SC2016 # expanded in the VM, not here + exec orb -m "$machine" -w "$(pwd)" sh -c ' + key=$1 + shift + export PATH="$HOME/.cargo/bin:$PATH" + export CARGO_TARGET_DIR="$HOME/.cache/liam-lab/target$key" + exec "$@" + ' lab "$top" "$@" +} + +musl_version=1.2.5 +musl_sha256=a9a118bbe84d8764da0ea0d28b3ab3fae8477fc7e4085d90102b8596fc7c75e4 + +packages='build-essential musl-tools gcc-x86-64-linux-gnu binutils-x86-64-linux-gnu +flex bison bc libelf-dev libssl-dev cpio file curl qemu-system-x86' + +setup() { + require_machine + channel=$(sed -n 's/^channel *= *"\(.*\)"/\1/p' "$repo/rust-toolchain.toml") + [ -n "$channel" ] || die "no channel in $repo/rust-toolchain.toml" + + say "packages" + # shellcheck disable=SC2016,SC2086 # expanded in the VM; the list is split on purpose + orb -m "$machine" -u root sh -c ' + set -eu + missing= + for p in "$@"; do + dpkg-query -W -f="\${Status}" "$p" 2>/dev/null | grep -q "ok installed" || + missing="$missing $p" + done + if [ -z "$missing" ]; then + echo " all installed" + exit 0 + fi + echo " installing:$missing" + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -y -qq --no-install-recommends $missing + ' setup $packages + + say "rust $channel and an x86_64 musl toolchain" + # shellcheck disable=SC2016 # expanded in the VM + orb -m "$machine" sh -c ' + set -eu + channel=$1 musl_version=$2 musl_sha256=$3 + if [ ! -x "$HOME/.cargo/bin/rustup" ]; then + # Downloaded first: piped into sh, a failed download is an empty script + # that exits 0. + init=$(mktemp) + curl --proto =https --tlsv1.2 -sSf https://sh.rustup.rs -o "$init" + sh "$init" -y --quiet --no-modify-path --default-toolchain none >/dev/null + rm -f "$init" + fi + PATH="$HOME/.cargo/bin:$PATH" + rustup toolchain install "$channel" --profile minimal \ + --component clippy,rustfmt \ + --target x86_64-unknown-linux-musl,aarch64-unknown-linux-musl + rustup default "$channel" + + # The machine is aarch64, so the x86_64 musl target needs a cross linker, + # and C dependencies (mimalloc) a compiler that sees x86_64 musl headers. + # Ubuntu packages musl for the native architecture only, so musl is built + # from upstream source for x86_64: the version Rust bundles for the target, + # with the checksum Ubuntu publishes for the same tarball. At link time Rust + # supplies the libc and start files; gcc only drives the link. + # Versioned, so bumping musl_version rebuilds instead of keeping old headers. + musl="$HOME/.local/x86_64-linux-musl-$musl_version" + if [ ! -x "$musl/bin/musl-gcc" ]; then + echo " building musl $musl_version for x86_64" + src=$(mktemp -d) + curl -fsSL "https://musl.libc.org/releases/musl-$musl_version.tar.gz" \ + -o "$src/musl.tar.gz" + echo "$musl_sha256 $src/musl.tar.gz" | sha256sum -c --quiet + tar -xzf "$src/musl.tar.gz" -C "$src" --strip-components=1 + (cd "$src" && ./configure --prefix="$musl" --target=x86_64-linux-gnu \ + --disable-shared --enable-wrapper=gcc >/dev/null && make -s -j"$(nproc)" install >/dev/null) + rm -rf "$src" + fi + + mkdir -p "$HOME/.cargo" + cat >"$HOME/.cargo/config.toml" <" +} + +case "${1:-}" in +'') usage >&2 && exit 2 ;; +-h | --help | help) usage ;; +setup) setup ;; +*) run "$@" ;; +esac diff --git a/scripts/task b/scripts/task index 600b8de..65a8c1c 100755 --- a/scripts/task +++ b/scripts/task @@ -66,8 +66,9 @@ case "${1:-}" in exit 1 fi cargo build --release --locked --target "$target" -p liamd -p liam-init + # scripts/lab points CARGO_TARGET_DIR outside the checkout. for bin in liamd liam-init; do - path="target/$target/release/$bin" + path="${CARGO_TARGET_DIR:-target}/$target/release/$bin" kind=$(file -b "$path") case "$kind" in *'statically linked'* | *'static-pie linked'*) ;;