diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index 0eb98b7..0fd0bdb 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -69,14 +69,18 @@ required administration flows explicitly and permits established replies. | `pikvm01` | — | `10.10.70.20` | Direct untagged attachment to `gw01` | | `kvm01` | — | `10.10.70.21` | Direct untagged attachment to `gw01` | -`gw01` supplies DHCP on every client VLAN. Named hosts use DHCP reservations. -Gateway and managed-switch interface addresses and the local DNS mirror address -are static. Dynamic clients use `.200` through `.250` within each client VLAN. -Reservations use each endpoint's permanent hardware MAC address as recorded in -the version-controlled gateway configuration. - -IncusOS seed data enables DHCP on the 10GbE management interface. The 2.5GbE -vPro interface receives its OOB reservation independently from AMT firmware. +`gw01` supplies DHCP on every client VLAN. Dynamic clients use `.200` through +`.250` within each client VLAN. DHCP-served named hosts (`sandbox01`, +`pikvm01`, `kvm01`) use reservations keyed on each endpoint's permanent +hardware MAC address as recorded in the version-controlled gateway +configuration. + +Infrastructure endpoints do not depend on DHCP: gateway and managed-switch +interface addresses and the local DNS mirror address are static interface +configuration, IncusOS node management addresses are static in each node's +seed (bound to the management NIC's hardware MAC in `GilmanLab/fleet`), and +lab-node AMT addresses are static in MEBx so out-of-band access survives a +gateway outage. The AMT interfaces have no DHCP reservations. ## Gateway interface mapping diff --git a/docs/docs/runbooks/commission-lab-node.md b/docs/docs/runbooks/commission-lab-node.md new file mode 100644 index 0000000..ec95c09 --- /dev/null +++ b/docs/docs/runbooks/commission-lab-node.md @@ -0,0 +1,226 @@ +--- +title: Commission a lab compute node +description: Provision AMT, install seeded IncusOS from USB, join the cluster, and escrow recovery keys for an MS-02 lab node. +--- + +# Commission a lab compute node + +Bring a Minisforum MS-02 Ultra lab node (`lab01`–`lab03` class) from bare or +previously-used hardware to an ONLINE Incus cluster member. This procedure was +proven live on 2026-08-21 (session 009) against all three lab nodes and AMI +BIOS `2.22.0059` with Intel ME `19.0.5`. + +The [network address and VLAN plan](../reference/networking/address-plan.md) is +canonical for every address used here. Seed configuration lives in +`GilmanLab/fleet` `nodes//config.yaml`. + +## Preconditions and required access + +- A `GilmanLab/fleet` config for the node with its **verified** 10GbE + management NIC MAC (`strict_hwaddr` binding; a wrong MAC leaves the node + unreachable after install). See "Harvest the management MAC" below. +- `incusos-builder` at the pin recorded in the fleet README, and a sacrificial + USB stick. Install media MUST be a raw image written with `dd`; never boot + the installer through Ventoy, and never attempt an AMT IDER/USB-R boot (see + Safety impact). +- `lab-admin` AWS SSO for SOPS: the node's AMT credential + (`fleet//amt.sops.yaml`) and recovery-key escrow + (`fleet//incusos.sops.yaml`) in `GilmanLab/secrets`. +- The trusted `bootstrap-admin` Incus client keypair (public half is in every + seed). +- Physical access for MEBx configuration, power, and USB insertion. AMT has no + remote path until MEBx is done, and the chassis has no wired ATX control. +- An AMT client on VLAN 10 for post-provisioning verification and console + work. gw01 forwards management→OOB in full, but the home→OOB path does not + pass the AMT service ports; a client on the management VLAN (for example + the `meshcommander` container on `nas01`, `http://10.10.10.14:3000/`) is + required. + +## Safety impact + +- The seed sets `force_install: true`: everything on the node's 128GB system + drive is destroyed. The `install.target` selector (`bus: NVME`, + `max_size: 200GiB`) cannot select the 2TB data drive. +- The Secure Boot procedure removes the factory Microsoft/vendor keys. After + it, only IncusOS-signed media boots on the node with Secure Boot enabled. +- **Never issue "Reset to IDE-R CDROM/Floppy" or attempt an AMT storage- + redirection boot on this platform.** AMI `2.22.0059` wedges pre-boot with + black video and dead input (local and remote); only a cold AMT power cycle + recovers. Reproduced twice in session 009. + +## Procedure + +### 1. Provision AMT in MEBx + +Read the credential first (never store it in MeshCommander's device list): + +```sh +cd ~/code/glab/secrets +AWS_PROFILE=lab-admin sops -d --extract '["password"]' fleet//amt.sops.yaml +``` + +In BIOS setup → MEBx, all in one visit: + +1. Set the AMT admin password to the escrowed value. +2. Network Setup: **static** address per the address plan + (`10.10.70.11/12/13`, mask `255.255.255.0`, gateway `10.10.70.1`). AMT + addresses are deliberately not DHCP reservations. +3. Enable SOL, storage redirection, and KVM (Manageability Feature Selection). +4. User Consent → opt-in: **NONE** (headless KVM fails otherwise). +5. Network Access State: **Network Active**. + +Settings commit on **Save & Exit**, not live. In the same visit, prepare +Secure Boot (Security → Secure Boot): + +1. Secure Boot Mode: Custom. +2. Key Management → **Factory Key Provision: Disabled**. With it enabled the + firmware silently re-installs factory keys on every reset and the IncusOS + media keeps failing Secure Boot validation. +3. Key Management → **Reset to Setup Mode** (delete all keys / delete PK). +4. Keep Secure Boot enabled; Save & Exit. The installer auto-enrolls the + IncusOS PK/KEK/db and the system runs in User Mode afterward. + +Verify AMT from a VLAN 10 client: `ping 10.10.70.` and TLS digest auth +(AMT ≥16 is TLS-only — ports 16993/16995; 16992/16994 stay closed by design): + +```sh +curl -sk --digest -u "admin:" -o /dev/null \ + -w '%{http_code}\n' https://10.10.70.:16993/index.htm # expect 200 +``` + +### 2. Harvest the management MAC + +The seed binds the management interface by MAC. Two proven methods: + +- **PXE-entry read (works on blank machines):** BIOS → Advanced → Network + Stack: Enabled → Save & Exit → boot menu (F7). Each PXE entry is labeled + with its NIC's MAC. The entry equal to the AMT MAC is the shared 2.5GbE + NIC; the other copper entry is the 10GbE management NIC. Disable Network + Stack again afterward. +- **Old-OS lease (works when a previous OS remains):** boot the resident OS + and read `show dhcp server leases` on `gw01`; the VLAN 10 lease is the + management NIC. + +Observed pattern on all three MS-02s: management MAC = AMT MAC + 1. Treat it +as a prediction only; verify before building. + +Record the MAC in `GilmanLab/fleet` `nodes//config.yaml` through the +normal PR flow. + +### 3. Build and write install media + +```sh +incusos-builder validate -f nodes//config.yaml +incusos-builder build -f nodes//config.yaml -o .img +diskutil list # identify the stick, carefully +diskutil unmountDisk /dev/diskN +sudo dd if=.img of=/dev/rdiskN bs=16m +diskutil eject /dev/diskN +``` + +Pin `image.release` to the release the cluster is running +(`incus query nas01:/os/1.0 | jq -r .environment.os_version`); joining +members should match the existing members. + +### 4. Install + +Insert the stick and boot it (F7 if needed). The install is unattended; the +installer waits for media removal, then reboots into IncusOS. The console +shows the hostname, `Applying Secure Boot certificate update`, the +`Application TLS certificate fingerprint`, and the static management address. + +If the installer fails with `mkfs.vfat ... contains a mounted filesystem`, +old partition remnants (observed with Proxmox) are colliding: boot a rescue +Linux and wipe **only** the 128GB system drive, then retry: + +```sh +lsblk -o NAME,SIZE,MODEL # verify by size before wiping +wipefs -af /dev/<128GB-disk> +sgdisk --zap-all /dev/<128GB-disk> +``` + +### 5. Verify identity and escrow recovery keys + +Compare the console fingerprint line with the API certificate before trusting +anything: + +```sh +echo | openssl s_client -connect 10.10.10.:8443 2>/dev/null \ + | openssl x509 -noout -fingerprint -sha256 +incus remote add 10.10.10. # only after the fingerprints match +``` + +Escrow immediately — before the join and before any reboot test: + +```sh +incus query :/os/1.0/system/security +``` + +Write `config.encryption_recovery_keys[0]` and +`state.pool_recovery_keys.local` into `GilmanLab/secrets` +`fleet//incusos.sops.yaml` (keys `encryption_recovery_key` and +`zfs_pool_local_recovery_key`) and merge through the normal PR flow. Then +acknowledge retrieval — this clears the console warning and MUST use the +dedicated action endpoint (a plain PUT of the state field is silently +ignored): + +```sh +incus query ':/os/1.0/system/security/:retrieved' -X POST +``` + +### 6. Join the cluster + +```sh +TOKEN="$(incus cluster add nas01: | tail -1)" +jq -n --arg cert "$(cat ~/.config/incus/servercerts/nas01.crt)" --arg token "$TOKEN" '{ + server_name: "", + enabled: true, + server_address: "10.10.10.:8443", + cluster_address: "10.10.10.14:8443", + cluster_certificate: $cert, + cluster_token: $token, + member_config: [{entity: "storage-pool", name: "local", key: "source", value: "local/incus"}] +}' | incus query :/1.0/cluster -X PUT --data @- +``` + +`server_name` is required. The `local` ZFS pool's member source is the +OS-created `local/incus` dataset. Joining adopts the cluster certificate, so +remove the now-stale standalone remote: `incus remote remove `. + +## Verification + +- `incus cluster list nas01:` shows the node `ONLINE`, "Fully operational". +- Smoke test on the new member: + + ```sh + incus launch images:alpine/3.22 nas01:smoke- --target + incus exec nas01:smoke- -- ping -c 2 10.10.10.1 + incus delete -f nas01:smoke- + ``` + +- `incus query 'nas01:/os/1.0/system/security?target='` reports + `encryption_recovery_keys_retrieved: true`. + +## Rollback / recovery + +- Boot prompts for a recovery key: use the escrowed + `encryption_recovery_key` from `GilmanLab/secrets`, then rebind the TPM via + the system security API. +- A wrong seeded MAC leaves the node unreachable with no shell: fix the fleet + config and reinstall from step 3. +- AMT/firmware wedge (black video, dead input): AMT Power Actions → power + down, wait, power up. Do not retry the action that caused it. +- A failed join leaves the node standalone; it can be re-joined with a fresh + token. If the node joined partially, remove it with + `incus cluster remove --force` before retrying. + +## Escalation + +- Secure Boot violations that survive the Factory Key Provision + Setup Mode + procedure: stop and inspect the firmware's key state; do not disable Secure + Boot to work around it. +- TPM unseal failures on first boot: clear the fTPM and reinstall (see the + [nas01 rebuild runbook](rebuild-nas01.md), which documents the same failure + class). +- Installer refuses or fails after a clean disk wipe: capture the console + error and check `lxc/incus-os` upstream before retrying. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 78ed0b6..6c2f99f 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -48,6 +48,7 @@ nav: - Tailscale policy: reference/networking/tailscale-policy.md - Runbooks: - Change the Tailscale policy: runbooks/tailscale-policy-change.md + - Commission a lab compute node: runbooks/commission-lab-node.md - Rebuild nas01: runbooks/rebuild-nas01.md - Manage the sw-core01 configuration: runbooks/sw-core01-configuration.md - Manage the sw-mgmt01 configuration: runbooks/sw-mgmt01-configuration.md