From 62d3348388413603a23b1a5a030855e5de8a6e75 Mon Sep 17 00:00:00 2001 From: Mathieu Tarral Date: Tue, 25 Aug 2026 03:00:15 +0200 Subject: [PATCH] docs: make the capture path followable, and add LICENSE/NOTICE The tutorial and the system-deps guide both sent readers to "the docker-compose stack shipped with neogit". No such stack exists, in the tree or anywhere in neogit's history. Both now use a single `docker run` for Neo4j, which is all a first capture needs: neogit defaults to local-filesystem object storage, so MinIO is optional and the tutorial's MINIO_* secrets were never required. oswatcher-deploy is named as the full-stack option rather than the starting point. `poetry install` also could not succeed on a machine prepared by our own apt line. `guestfs` and `libvirt-python` are C extensions built from source, so the install died at `fatal error: guestfs.h: No such file or directory`. Adds the build headers and a note explaining they are headers, not runtime tools. Verified in a clean ubuntu:24.04 container: guestfs and libvirt-python both compile and import. Also adds the Apache 2.0 LICENSE and NOTICE the README badge already pointed at, expands the README and docs index with the pipeline diagram and a tutorial-first entry point, and marks the finished items in OPEN_SOURCE_TASKS.md. Docs build clean with -W. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01FNgjQ6o613ULuMRGhLnyUU --- LICENSE | 202 ++++++++++++++++++++++ NOTICE | 6 + OPEN_SOURCE_TASKS.md | 20 +-- README.md | 56 +++++- docs/source/how-to/install-system-deps.md | 13 +- docs/source/index.md | 12 +- docs/source/tutorials/first-capture.md | 26 ++- 7 files changed, 308 insertions(+), 27 deletions(-) create mode 100644 LICENSE create mode 100644 NOTICE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..ea076a5 --- /dev/null +++ b/NOTICE @@ -0,0 +1,6 @@ +osw-builder +Copyright 2016-2026 Mathieu Tarral + +This product is licensed under the Apache License, Version 2.0. +You may obtain a copy of the License in the LICENSE file or at +http://www.apache.org/licenses/LICENSE-2.0 diff --git a/OPEN_SOURCE_TASKS.md b/OPEN_SOURCE_TASKS.md index f39fa19..3bc125d 100644 --- a/OPEN_SOURCE_TASKS.md +++ b/OPEN_SOURCE_TASKS.md @@ -33,17 +33,15 @@ Submodule : `github.com/OSWatcher/packer-templates` --- -## #4 Cleanup : API keys, branches mortes, refs privées - -- `build.py:158` : `username="oswatcher"` hardcodé → rendre configurable -- Vérifier les workflows CI (`.github/workflows/`) pour refs à infra privée -- Supprimer les branches locales et remote obsolètes (50+ branches stale sur origin) - - Branches locales à évaluer : `feature/branch-cascade-inheritance`, `feature/os-entry-inheritance`, `feature/vagrant-realtime-logging` -- Fichiers parasites à la racine à supprimer : - - `capture.log`, `packer-build.log`, `vagrant.log`, `domain.xml` - - `REPORT.md`, `debug_guestfs.py` - - `rewrite_commits_nonbusiness.py`, `extract_business_hours_commits.py` - - `update_windows_release_dates.cypher` +## 🔄 #4 Cleanup : API keys, branches mortes, refs privées + +- ✅ `debug_guestfs.py`, `domain.xml` supprimés du repo +- ✅ `PAT_TOKEN` retiré de tous les workflows CI (packer-templates public, deps sur PyPI) +- ✅ `GIT_AUTH_TOKEN` secret retiré du Dockerfile +- ✅ `ARG GIT_USERNAME=wenzel` retiré du Dockerfile +- ✅ `GHCR_TOKEN` auth retiré (image packer-templates publique sur GHCR) +- ✅ Branches stale supprimées (déjà nettoyées) +- ⬜ Merger `feat/use-oswatcher-plugins-from-pypi` → master et pousser sur GitHub --- diff --git a/README.md b/README.md index dafe23f..e4461ee 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,29 @@ # osw-builder -A pipeline tool that builds VM images from ISOs and captures their filesystems into a Neo4j graph database — a queryable graph of operating system evolution covering Windows 95 → 11 and Ubuntu 6.10 → 25.04. +[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE) +[![Python versions](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml) +[![CI](https://github.com/OSWatcher/osw-builder/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/OSWatcher/osw-builder/actions/workflows/ci.yml) +[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue.svg)](https://oswatcher.github.io/osw-builder/) + +> Turn an OS ISO into a queryable graph snapshot. osw-builder builds a VM image from an installer ISO, then captures its filesystem and registry into a Neo4j graph — fully unattended — so you can diff and query operating systems the way you query a git history. + +It feeds a queryable graph of operating system evolution covering Windows 95 → 11 and Ubuntu 6.10 → 25.04. **What it does, end to end:** 1. Build a VM image from an ISO using Packer (runs inside Docker, no local Packer install needed) 2. Register the image with Vagrant/libvirt -3. Capture the filesystem and registry into Neo4j via [neogit](https://github.com/OSWatcher/neogit) +3. Capture the filesystem and registry into Neo4j via [neogit](https://github.com/OSWatcher/neogit) as content-addressed Merkle trees 4. Optionally search for and install OS updates, snapshotting and capturing after each one +``` +ISO ─▶ image_builder ─▶ vagrant ─▶ capture ─▶ updates ─▶ Neo4j graph + (Packer/Docker) (libvirt) (libguestfs) (apt / + Windows Update) +``` + +📖 **Full documentation:** — tutorials, how-to guides, configuration reference, and architecture explanations. + --- ## Prerequisites @@ -25,10 +40,18 @@ A pipeline tool that builds VM images from ISOs and captures their filesystems i | Python 3.11+ | Runtime | | Poetry | Dependency management | -**Infrastructure** (see [neogit](https://github.com/OSWatcher/neogit) for setup) +**Infrastructure** (only needed for capture — see [neogit](https://github.com/OSWatcher/neogit) for setup) - Neo4j 5.x — graph database where OS snapshots are stored -- MinIO (or any S3-compatible store) — object storage for blob data +- Object storage for file contents — neogit defaults to the **local filesystem**, so MinIO (or any S3-compatible store) is optional and only needed for a distributed/production setup like [oswatcher-deploy](https://github.com/OSWatcher/oswatcher-deploy) + +--- + +## Two ways in + +**🎓 I just want to try it.** Follow the [first-capture tutorial](https://oswatcher.github.io/osw-builder/tutorials/first-capture.html): it walks you from zero to a captured Ubuntu image with a single Neo4j container and no product keys. Budget one to two hours, mostly unattended. + +**🏗️ I want to run this for real.** Read the rest of this README, then the [how-to guides](https://oswatcher.github.io/osw-builder/how-to/index.html) for providing ISOs, adding images, and building without capture. For the full OSWatcher infrastructure (Neo4j + MinIO + API + frontend), see [oswatcher-deploy](https://github.com/OSWatcher/oswatcher-deploy). --- @@ -40,6 +63,10 @@ cd osw-builder poetry install ``` +`--recurse-submodules` matters: the Packer templates live in a git submodule. If you forgot it, run `git submodule update --init`. + +System dependencies (QEMU, libvirt, Vagrant, Docker, libguestfs) are not Python packages — install them first. See [Install system dependencies](https://oswatcher.github.io/osw-builder/how-to/install-system-deps.html). + --- ## Configuration @@ -74,13 +101,15 @@ Defines all supported images with their build configuration (Packer template, an ### neogit credentials -osw-builder uses neogit to write to Neo4j and MinIO. Create `~/.secrets.toml` with: +osw-builder uses neogit to write the graph to Neo4j and the file contents to object storage. Create `~/.secrets.toml` with at least the Neo4j connection: ```toml [default] NEO4J_URI = "bolt://localhost:7687" NEO4J_USER = "neo4j" NEO4J_PASSWORD = "your-password" +# Object storage. neogit defaults to local filesystem storage, so MinIO is +# optional. Add these only if you point neogit at a MinIO/S3 backend. MINIO_URL = "http://localhost:9000" MINIO_ACCESS_KEY = "minioadmin" MINIO_SECRET_KEY = "minioadmin" @@ -170,7 +199,24 @@ Legacy Windows 95/98/ME/2000 entries require pre-built Vagrant boxes (build your --- +## Documentation + +The full documentation is organised with the [Divio system](https://docs.divio.com/documentation-system/) and published at : + +| Section | What it covers | +|---------|----------------| +| [Tutorials](https://oswatcher.github.io/osw-builder/tutorials/index.html) | Hands-on: your first Ubuntu capture, end to end | +| [How-to guides](https://oswatcher.github.io/osw-builder/how-to/index.html) | Install system deps, provide ISOs, add a new image, build without capture | +| [Reference](https://oswatcher.github.io/osw-builder/reference/index.html) | CLI options, configuration schema, module API | +| [Explanation](https://oswatcher.github.io/osw-builder/explanation/index.html) | Pipeline design, image inheritance, response files | + ## Related projects - [neogit](https://github.com/OSWatcher/neogit) — the content-addressed Merkle-tree library that backs capture +- [packer-templates](https://github.com/OSWatcher/packer-templates) — the Packer build templates (a submodule of this repo) +- [oswatcher-deploy](https://github.com/OSWatcher/oswatcher-deploy) — full production stack (Neo4j, MinIO, API, frontend) - [pywinupdate](https://github.com/OSWatcher/pywinupdate) — standalone WinRM/Ansible Windows Update CLI; independent from the OS-agnostic update orchestration in `osw_builder/updates/`, but scratches a similar itch + +## License + +Licensed under the Apache License 2.0. diff --git a/docs/source/how-to/install-system-deps.md b/docs/source/how-to/install-system-deps.md index 42fc7a9..b5e4d76 100644 --- a/docs/source/how-to/install-system-deps.md +++ b/docs/source/how-to/install-system-deps.md @@ -20,12 +20,21 @@ sudo apt update sudo apt install -y \ qemu-kvm libvirt-daemon-system libvirt-clients \ vagrant libguestfs-tools sshpass \ - docker.io python3 python3-pip + docker.io python3 python3-pip \ + libguestfs-dev libvirt-dev python3-dev pkg-config gcc # Vagrant libvirt provider vagrant plugin install vagrant-libvirt ``` +```{note} +The last line installs build headers, not runtime tools. `guestfs` and +`libvirt-python` are C extensions that Poetry compiles from source, so without +`libguestfs-dev` and `libvirt-dev` the install fails at +`fatal error: guestfs.h: No such file or directory` long before you reach a +capture. +``` + ## Post-install checks Add yourself to the `libvirt`, `kvm`, and `docker` groups so you do not need `sudo`: @@ -51,4 +60,4 @@ sudo libguestfs-test-tool # ends with "libguestfs-test-tool: PASS" ## Infrastructure (separate from system packages) -Capturing to a graph requires Neo4j and MinIO running somewhere reachable. These are not installed here — use the docker-compose stack from [neogit](https://github.com/OSWatcher/neogit). If you only want to *build* images and not capture, you can skip this; see {doc}`build-without-capture`. +Capturing to a graph requires a reachable Neo4j instance; object storage defaults to the local filesystem, so MinIO is optional. The quickest way to get Neo4j is a single container — `docker run -p 7474:7474 -p 7687:7687 -e NEO4J_AUTH=neo4j/your-password neo4j:5.26` (see [neogit](https://github.com/OSWatcher/neogit) for details, or [oswatcher-deploy](https://github.com/OSWatcher/oswatcher-deploy) for the full stack with MinIO). If you only want to *build* images and not capture, you can skip this entirely; see {doc}`build-without-capture`. diff --git a/docs/source/index.md b/docs/source/index.md index 5ae2f55..fa79762 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -1,6 +1,16 @@ # osw-builder -**osw-builder** builds VM images from ISOs and captures their filesystems into a Neo4j graph database — a queryable graph of operating system evolution covering Windows 95 → 11 and Ubuntu 6.10 → 25.04. +**osw-builder** turns an OS installer ISO into a queryable graph snapshot. It builds a VM image with Packer, captures the resulting filesystem and registry into a Neo4j graph as content-addressed Merkle trees, and — optionally — installs OS updates one by one, capturing the state after each. The whole pipeline runs unattended. + +It feeds a queryable graph of operating system evolution covering Windows 95 → 11 and Ubuntu 6.10 → 25.04. Because every snapshot lives in the same graph, you can ask questions an ordinary disk image never answers — *which files changed between two Windows builds, which registry keys a given update touched, on which OS versions a symbol first appeared.* + +``` +ISO ─▶ image_builder ─▶ vagrant ─▶ capture ─▶ updates ─▶ Neo4j graph + (Packer/Docker) (libvirt) (libguestfs) (apt / + Windows Update) +``` + +**New here?** Jump straight to the {doc}`first-capture tutorial ` — it takes you from an empty machine to a captured Ubuntu image with one Neo4j container and no product keys. This documentation is organised according to the [Divio documentation system](https://docs.divio.com/documentation-system/) — four distinct kinds of documentation that serve different needs: diff --git a/docs/source/tutorials/first-capture.md b/docs/source/tutorials/first-capture.md index 9da7c56..ddd3b09 100644 --- a/docs/source/tutorials/first-capture.md +++ b/docs/source/tutorials/first-capture.md @@ -20,22 +20,32 @@ poetry install The `--recurse-submodules` flag matters: the Packer templates live in a git submodule. If you forgot it, run `git submodule update --init` now. -## Step 2 — Start Neo4j and MinIO +## Step 2 — Start Neo4j -osw-builder writes captured filesystems into Neo4j (the graph) and MinIO (the file contents). The quickest way to get both is the docker-compose stack shipped with [neogit](https://github.com/OSWatcher/neogit). Start it, then confirm you can reach the Neo4j browser at . +osw-builder writes the captured filesystem into Neo4j (the graph) and the file contents into object storage. [neogit](https://github.com/OSWatcher/neogit) — the library that does the writing — defaults to **local filesystem** object storage, so for this tutorial you only need a Neo4j container. One `docker run` is enough: -## Step 3 — Tell neogit where the databases are +```bash +docker run --rm --name osw-neo4j \ + -p 7474:7474 -p 7687:7687 \ + -e NEO4J_AUTH=neo4j/your-password \ + neo4j:5.26 +``` + +Leave it running and confirm you can reach the Neo4j browser at (log in with `neo4j` / `your-password`). + +```{note} +For a full OSWatcher stack with MinIO object storage, an API, and a frontend, use [oswatcher-deploy](https://github.com/OSWatcher/oswatcher-deploy) instead. You do not need it for a first capture. +``` + +## Step 3 — Tell neogit where Neo4j is -Create `~/.secrets.toml` so osw-builder can authenticate against Neo4j and MinIO: +Create `~/.secrets.toml` so osw-builder can authenticate against Neo4j. Because neogit uses local object storage by default, the MinIO keys are optional here: ```toml [default] NEO4J_URI = "bolt://localhost:7687" NEO4J_USER = "neo4j" NEO4J_PASSWORD = "your-password" -MINIO_URL = "http://localhost:9000" -MINIO_ACCESS_KEY = "minioadmin" -MINIO_SECRET_KEY = "minioadmin" ``` ## Step 4 — Point an image at an ISO @@ -85,7 +95,7 @@ LIMIT 25 ## What you have learned -You ran the four phases of osw-builder — build, register, capture — and produced a real OS snapshot in a graph database. You also met the two configuration files you will use constantly: `config.yaml` for your local overrides and `default_settings.yaml` for the shared catalogue. +You ran the first three phases of osw-builder — build, register, capture — and produced a real OS snapshot in a graph database. (The fourth phase, update search, you deliberately skipped with `--search-updates=false`.) You also met the two configuration files you will use constantly: `config.yaml` for your local overrides and `default_settings.yaml` for the shared catalogue. ## Where to go next