golden image cutover - #22
Merged
Merged
Conversation
vpzed-dev-lux
commented
Aug 28, 2026
vpzed-dev-lux
commented
Collaborator
- add scripts/build-image.sh and point at the golden image
- rename modules/vm to modules/vm-pve
- move cloud-init from a snippet to PVE's native settings
- drop the hypervisor SSH credential from the root config
- add the ansible base role, and the collections CI needs to lint it
- delete the cloud-init layer and rewrite the docs around it
- import the disk over the API, not over SSH
- take back two guarantees the cutover had quietly outsourced
- add AGENTS.md as a symlink to CLAUDE.md
- correct the docs against the finished code
The bpg provider can only upload cloud-init snippets over SSH, which is what forces a root SSH key for the hypervisor into providers.tofu. Everything that snippet does can move to PVE's native cloud-init settings or to ansible - except installing qemu-guest-agent, because apply blocks until the agent reports an address and the ansible inventory reads that address from the agent. So the agent gets baked into the image instead. build-image.sh downloads a dated upstream serial, verifies it against SHA256SUMS, installs the agent with virt-customize, and uploads the result to the node over the API - deliberately not scp, since losing the SSH path is the point of the exercise. Two things the build has to get right and neither is obvious: - The image's root filesystem has ~366 MB free, and apt's universe index is ~120 MB unpacked. APT::Snapshot would pull a second full index set alongside the configured one, which fills the guest disk mid-unpack. The sources file is rewritten to the snapshot mirror for the duration instead, and only Packages indexes are fetched. - PVE streams everything after the multipart file part straight into the file, so filename=@ must be the last -F. With checksum after it, the checksum was never parsed as a parameter and its 286 bytes were appended to the image - which qemu-img tolerates silently. In the right order the node verifies the upload itself. There is no `systemctl enable qemu-guest-agent`: the unit ships an empty [Install] and is started by a udev rule on the virtio-serial port. The build asserts that rule exists, since without it apply hangs for the full 30m timeout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
The module is specific to ProxMox VE - it speaks the bpg/proxmox provider's resource schema throughout - and the bare name did not say so. The module call is renamed too, not just the directory, so resource addresses read module.vm-pve["name"]. That is only free right now: state currently holds nothing but the three data sources, so there is no address to migrate. Once Step 7 provisions a VM, the same rename would need a moved block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
The snippet was the only reason the provider needed SSH to the hypervisor. Everything it carried now lands in one of two places: identity at first boot goes into the VM's own cloud-init config over the API, and everything softer - packages, apt pinning, timezone - becomes ansible's job. So this drops the fqdn/ci_packages/runcmd/user_data locals and the proxmox_virtual_environment_file resource, and adds user_account plus an explicit upgrade to the initialization block. upgrade is stated rather than left unset because PVE defaults ciupgrade to 1 while spec.package_upgrade defaults to false, and the provider's schema default is Computed - so an unset value would silently take PVE's side. The dns block stays as it was: dns_domain still drives searchdomain, which is what PVE turns into the guest's generated fqdn, so dropping the local fqdn computation changes nothing the guest sees. packages and archive_snapshot become module outputs, since the ansible inventory is what applies them now. spec.packages gains a name validation to replace the protection jsonencode() used to give it inside the YAML template. vms.tofu drops the three arguments those variables backed; the rest of the root config is a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
With the snippet gone there is nothing the provider cannot do over the API, so the ssh block goes, and with it local.pve_host and pve_ssh_private_key_path. Root on the hypervisor is no longer a credential this project holds. snippet_datastore goes the same way - nothing writes snippets any more. The root vms output gains packages, archive_snapshot and timezone, which is how the three settings that used to be baked into the cloud-init drive reach the ansible inventory instead. ci_timezone survives for that reason, but it now describes a re-appliable setting rather than a first-boot one. variables.tofu is a symlink to site-specific values outside the repo, so the same edits are mirrored into variables.tofu.example by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
base carries what the cloud-init snippet used to do at first boot: the apt snapshot pin, the APT::Periodic zeros, the apt-daily timers, the timezone and the spec's packages. Moving them to layer 2 makes them re-appliable - change a VM's archive_snapshot or the fleet timezone and the next playbook run reaches the running guest, where before it needed a rebuild. It is applied via roles: rather than the spec's include loop, so no inventory file can opt out and it always lands before any role installs a package - the snapshot pin has to be in force first. Both apt.conf.d filenames are the ones cloud-init wrote, kept verbatim: vm-fingerprint.sh records apt config by filename, and renaming them would make every fingerprint captured so far incomparable. The timers are masked as well as disabled, which a stock image is not - a package postinst re-running `systemctl preset` can undo a disable, and a test environment must not change itself. Expect that to show in the first post-cutover fingerprint diff. qemu-guest-agent is deliberately absent: apply blocks on the agent reporting an address and the inventory reads that address from the agent, so ansible cannot be what installs it. It comes from the golden image. The timezone task is the repo's first non-builtin FQCN, which CI could not have resolved - it installs bare ansible-core while requirements.txt pins the batteries-included ansible package, so community.* passes locally and fails there. ansible/requirements.yml pins what is used and the workflow installs from it, which closes that asymmetry rather than working around it. Measured at 8s and 27 MB, against 12s of apt work Step 6 removes from the same job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
Nothing reads cloud-init/ any more, so the templates and the check script that guarded them go. With check-cloud-init.sh gone, CI loses the `cloud-init` and `python3-yaml` apt install, its own step, and tofu_wrapper: false - the wrapper was disabled solely because it buffered stdio and hung that script's `tofu console` pipe. The docs carried the old three-layer story throughout, so this rewrites rather than patches: a golden image before first boot, PVE's native cloud-init for identity, ansible for everything else. Specifically - Prerequisites and Setup lose the snippets datastore, the Datastore.Allocate paragraph, the OpenTofuSnippetStore role and the whole provisioning-SSH-key step; Credentials drops to two rows and says why the third is gone; the repeatability notes name the base role instead of bootcmd, and gain the distinction that now matters - packages, archive_snapshot and the timezone reach a running VM, package_upgrade does not. A new "Building the golden image" section sits between Setup and startup, which is where a reader following the setup needs it. CONTRIBUTING.md's "run what CI runs" block matched CI, so it had to move with it, and it gains the rule that any collection beyond ansible.builtin must be pinned in the same change that uses it. Opportunistic fixes found while reading: "subdiredtory", optel-lgtm -> otel-lgtm (the image really is grafana/otel-lgtm), inventory/example.yaml -> inventory-example.yaml, and the Fingerprints section, which described a tracked file and a -build1/-build2 naming the repo does not use - fingerprints/ is gitignored and the workflow captures to one path and diffs it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
Reprovisioning found the premise incomplete: the snippet was not the only thing the provider did over SSH. A disk whose source is `file_id` goes through the provider's "custom disk" path, which shells out to `qm` on the node as root - so the first apply after removing the ssh block failed with "unable to authenticate user over SSH", not on cloud-init but on disk creation. import_from is the API-native equivalent: it passes PVE's import-from parameter, which does the same convert-and-import server-side. It only accepts `images` or `import` content, never `iso`, so the golden image is now uploaded as `import` content - and named .qcow2, because PVE restricts import uploads to .ova/.qcow2/.raw/.vmdk and the image genuinely is qcow2. The extension is load-bearing, not cosmetic. vm-fingerprint.sh needed a fix to run at all: `cloud-init status` exits 2 when the boot finished with recoverable errors, and the set -eu in its remote script turned that into a silent, message-free abort. Degraded is now the steady state here, because the user-data PVE generates uses the top-level `user:` key that cloud-init deprecated in 22.2 - nothing in this repo emits it and nothing can suppress it. The capture records the long form so the degradation is visible and a future PVE fixing it shows up as a diff, minus last_update, which is a timestamp. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
Both found by the Step 7 fingerprint diff rather than by reading the code. Key-only SSH. The old snippet set ssh_pwauth: false, which made cloud-init write 50-cloud-init.conf. PVE's generated user-data has no equivalent, so the guarantee had come to rest entirely on the cloud image happening to ship 60-cloudimg-settings.conf - still true, still off, but no longer ours and not something this repo would notice losing. The base role now writes its own drop-in, numbered 10- because sshd takes the FIRST value it obtains for a keyword and reads the include glob in lexical order, which is the opposite of apt.conf.d. KbdInteractiveAuthentication comes with it: disabling PasswordAuthentication alone still leaves a PAM keyboard-interactive path to password login. A separate task runs `sshd -t` over the whole assembled config before the reload handler flushes, so a bad fragment fails the play with the running sshd untouched. Guest address selection. The agent reports one address list per interface, and flattening them was correct only until the docker role brought up a docker0 at 172.17.0.1 - the inventory takes element 0 as ansible_host, and it stayed right by luck rather than by rule. ipv4_addresses / mac_addresses / network_interface_names share an index, so the module now selects the entry matching network_device[0].mac_address. That beats both alternatives on offer: docker's bridge address is configurable via daemon.json bip and podman, libvirt and CNI each bring their own ranges, while interface names vary by image (eth0, ens18, enp0s18, br-*, veth*). Neither can present the MAC ProxMox assigned to net0. Verified on VM 500: playbook applied both, second run changed=0, `sshd -T` reports passwordauthentication and kbdinteractiveauthentication no, and the vms output no longer carries 172.17.0.1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
Not everyone using this repo drives it with Claude Code, and AGENTS.md is the name most other coding agents look for. A symlink rather than a copy so the two can never drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
A review pass turned up claims that were true when written and stopped being true during Step 7, plus two instructions that never worked here. Stale after the import_from change: the golden image needs a datastore with the `import` content type, not `iso`, and the disk names it as `import_from`, not `file_id` - which also means `disk[0].import_from`, not `disk[0].file_id`, is what `ignore_changes` covers. Never worked: `git mv inventory/<name>.yaml inventory/destroy/`, in both the usage list and the fingerprint walkthrough. This template gitignores `inventory/*.yaml` and `inventory/destroy/*.yaml`, so git mv fails with "not under version control". Plain mv, with a note that a fork tracking its own specs can use the git form. Same class of error in the fingerprint section, which reached for `git add -f` and `git diff` against a gitignored `fingerprints/**` - now cp and diff. Also: CLAUDE.md still described fingerprint captures as landing in a tracked file, and omitted key-only SSH from the base role's inventory of what it owns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
vpzed
previously approved these changes
Aug 28, 2026
Bring in CI validate change so this code can pass CI
ansible-lint failed on PR #22 with "couldn't resolve module/action 'community.general.timezone'" even though the galaxy install step ran. The step was a no-op: the runner already has both collections somewhere its own ansible-galaxy can see, so it reported "nothing to do" - and ansible-lint, which runs from its own pipx venv, reads none of those paths. ansible-lint's fallback of installing requirements.yml itself cannot help either: ansible_compat looks for requirements.yml relative to the directory holding .git, so ansible/requirements.yml is invisible to it no matter what working-directory the step uses. Export ANSIBLE_COLLECTIONS_PATH for the job and install into it with --force, which ansible_compat preserves and prepends its cache to. Reproduced the failure in a venv holding only ansible-core 2.21.3 and ansible-lint 26.8.0, and confirmed this makes the same run pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
The job-level env: block cannot see the runner context, so
${{ runner.temp }} there made the whole workflow invalid: the run failed in
0s with "this run likely failed because of a workflow file issue", and no
pull_request run was created for the commit at all. Set the variable inside
the step via $RUNNER_TEMP and GITHUB_ENV, which every later step inherits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55
vpzed
approved these changes
Aug 28, 2026
vpzed-dev-lux
added a commit
that referenced
this pull request
Aug 28, 2026
* make the local venv match CI CI and a workstation ran different software, and the difference decided lint results: requirements.txt pinned the batteries-included `ansible` package, so locally every collection in it resolved and ansible/requirements.yml was decorative, while CI installed only the two pins into a pipx-based environment. A task using an unlisted community.* module passed here and failed there - which is exactly how PR #22 went red. Both sides now build the same environment from the same two files: - requirements.txt pins ansible-core instead of ansible (regenerated with uv pip freeze; platformdirs moved 4.11.4 -> 4.11.5 in the fresh resolve). - CI drops pipx for `uv venv --python 3.14 --managed-python` plus `uv pip install -r requirements.txt`, the same commands README.md gives an operator, and puts .venv/bin on $GITHUB_PATH. - scripts/install-collections.sh is the one way collections are installed, used by both, into the ANSIBLE_COLLECTIONS_PATH its caller names: tofu.env sets .venv/share/ansible/collections locally, the workflow's job env: block the same path under the workspace. Recreating the venv now takes the collections with it. - check-ansible.sh verifies the installed collections against the pins before anything else, so a missing one says so instead of surfacing as an opaque ansible-lint "couldn't resolve module/action". Verified: check-ansible.sh reports ok on all four checks, tofu validate and fmt -check pass, tofu plan is unchanged, and pointing a task at an unpinned collection now fails ansible-lint locally the way it always did in CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55 * pin setup-uv to an exact tag astral-sh/setup-uv publishes no moving major alias past v8: v9 and v10 exist only as full version tags, so @v10 failed the run at "Prepare all required actions" before any step ran. Pinning the exact version is what the rest of this repo does anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzjHsAQ3HkQjLdgv5PVi55 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.