Range42 Backend API

FastAPI application that orchestrates Proxmox infrastructure deployments by executing Ansible playbooks via ansible-runner. Part of the Range42 cyber range platform. Designed to sit behind a Kong API gateway that handles authentication and ACLs.

---

Table of Contents

• Quick Start
• Docker: Build & Publish
• Configuration
• API Documentation
• WebSocket API
• Project Structure
• Architecture
• Development
• Test Structure
• License

---

Quick Start

Option 1 -- Docker

# Build and start (uses the built image — no source bind-mounts)
VAULT_PASSWORD=my-secret docker compose up

Builds the image from the local source and starts the API on port 8000. The application code, Ansible playbooks, and inventory are baked into the image at build time.

Environment variables: Configured via the host environment or a .env file. Required: at least one of VAULT_PASSWORD_FILE or VAULT_PASSWORD for vault-encrypted operations.

Volumes:

• ${SSH_KEY_PATH:-~/.ssh} -- SSH private keys (read-only) — needed for Ansible over SSH

Health check: The container pings /docs/openapi.json every 30s (5s timeout, 10s start period, 3 retries).

Option 2 -- start.sh

./start.sh

The script resolves PROJECT_ROOT_DIR, sources .env if present, installs Ansible collections on first run, and launches uvicorn with --reload.

Option 3 -- Manual (development)

# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install Python and Ansible dependencies
pip install -r requirements.txt
ansible-galaxy collection install -r requirements.yml -p ~/.ansible/collections

# Set required environment variables (or create a .env file)
export PROJECT_ROOT_DIR="$(pwd)"
export VAULT_PASSWORD_FILE="/path/to/vault-pass.txt"

# Start the dev server
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

---

Docker: Build & Publish

The Dockerfile is a two-stage build (builder → runtime) based on Debian Bookworm (python:3.13-bookworm / python:3.13-slim-bookworm):

• Stage 1 builder — installs Python dependencies into /opt/venv and Ansible collections into /usr/share/ansible/collections.
• Stage 2 runtime — copies the virtualenv and collections from the builder; bakes in application code; runs uvicorn.

Non-root user

The runtime image runs as a non-root user (range42, UID/GID 1000 by default). If your SSH keys are owned by a different UID, pass matching build args so the container user can read the mounted keys:

docker compose build   # uses UID/GID 1000
# or match your host user:
UID=$(id -u) GID=$(id -g) docker compose build

SSH host key checking is enabled (ANSIBLE_HOST_KEY_CHECKING=True). Pre-populate ~/.ssh/known_hosts on the host before mounting, or the first Ansible connection to an unknown host will fail.

Build locally

docker compose build
# or directly:
docker build -t ghcr.io/range42/range42-backend-api:latest .

Run locally (validate the image)

VAULT_PASSWORD=my-secret docker compose up
# API reachable at http://localhost:8000/docs/swagger

Publish to GHCR

# 1. Authenticate (one-time per machine)
echo $GITHUB_TOKEN | docker login ghcr.io -u <github-username> --password-stdin

# 2. Build and tag
IMAGE=ghcr.io/range42/range42-backend-api
VERSION=v0.1   # or $(git describe --tags --always)

docker build -t ${IMAGE}:${VERSION} -t ${IMAGE}:latest .

# 3. Push
docker push ${IMAGE}:${VERSION}
docker push ${IMAGE}:latest

Or with Compose (sets IMAGE_NAME for the service):

IMAGE_NAME=ghcr.io/range42/range42-backend-api:v0.1 docker compose build
IMAGE_NAME=ghcr.io/range42/range42-backend-api:v0.1 docker compose push

Vault password in production

VAULT_PASSWORD passed as an environment variable is visible via docker inspect and in /proc/<pid>/environ inside the container. For production use VAULT_PASSWORD_FILE pointed at a mounted secret file:

# Create a secret file (outside the repo)
echo "my-vault-password" > /run/secrets/vault_pass
chmod 600 /run/secrets/vault_pass

# Pass the file path, not the password itself
VAULT_PASSWORD_FILE=/run/secrets/vault_pass docker compose up

OpenAPI spec

The committed openapi.json at the repository root reflects the current API surface. It is used to bootstrap the Kong API gateway configuration, and CI fails if it is out of date — regenerate and commit it whenever you add or modify a route:

RANGE42_AUTH_MODE=development \
PROJECT_ROOT_DIR=$PWD \
API_BACKEND_WWWAPP_PLAYBOOKS_DIR=$PWD \
API_BACKEND_PUBLIC_PLAYBOOKS_DIR=$PWD \
API_BACKEND_INVENTORY_DIR=$PWD/inventory \
PYTHONPATH=. python -c "
import json
from app.main import create_app
print(json.dumps(create_app().openapi(), indent=2))
" > openapi.json

The path variables satisfy import-time reads. The command uses development authentication mode only for schema generation; it does not start an HTTP service or load operator credentials. The generated document does not depend on these values. Production authentication remains required by default.

---

Configuration

All settings are read from environment variables in app/core/config.py. Nothing is hard-coded.

Variable	Required	Description	Default
PROJECT_ROOT_DIR	Yes	Absolute path to the project root	. (cwd)
VAULT_PASSWORD_FILE	Yes*	Path to the Ansible Vault password file	--
VAULT_PASSWORD	Yes*	Ansible Vault password as a string	--
API_BACKEND_WWWAPP_PLAYBOOKS_DIR	No	Local playbooks directory	PROJECT_ROOT_DIR/
API_BACKEND_PUBLIC_PLAYBOOKS_DIR	No	External playbooks repository path	--
API_BACKEND_INVENTORY_DIR	No	Ansible inventory directory	PROJECT_ROOT_DIR/inventory/
API_BACKEND_VAULT_FILE	No	Path to vault-encrypted variables file	--
CORS_ORIGIN_REGEX	No	Regex for allowed CORS origins	localhost / 127.0.0.1 / [::1] only
HOST	No	Server bind address	0.0.0.0
PORT	No	Server listen port	8000
DEBUG	No	Enable debug mode (true, 1, or yes)	false

*One of VAULT_PASSWORD_FILE or VAULT_PASSWORD must be set for vault-encrypted operations.

Logging

The API uses Python's logging module with structured output. Log level is controlled by uvicorn:

uvicorn app.main:app --log-level debug   # verbose
uvicorn app.main:app --log-level info    # default
uvicorn app.main:app --log-level warning # quiet

When DEBUG=true, the app registers a custom 422 handler that logs full validation error details at ERROR level -- useful for debugging malformed requests during development.

---

API Documentation

Once the server is running, interactive docs are available at:

Format	URL
Swagger UI	/docs/swagger
ReDoc	/docs/redoc
OpenAPI JSON	/docs/openapi.json

WebSocket API

VM Status Stream

Real-time VM status updates via WebSocket. Polls the Proxmox API directly (not via Ansible) for low latency.

URL: ws://host:8000/ws/vm-status

Query Parameters:

Parameter	Required	Description	Default
node	No	Proxmox node name to monitor	Read from inventory

Authentication: Proxmox API credentials are read from the backend's inventory/hosts.yml file. The frontend does not need to handle tokens.

Message Format

Initial connection -- full state:

{
  "type": "full",
  "vms": [
    {
      "vmid": 100,
      "name": "my-vm",
      "status": "running",
      "cpu": 12.5,
      "mem": 2147483648,
      "maxmem": 4294967296,
      "uptime": 86400,
      "template": 0,
      "tags": "web;production"
    }
  ]
}

Subsequent updates -- diff only:

{
  "type": "diff",
  "changes": {
    "100": { "type": "changed", "vmid": 100, "status": "stopped", "cpu": 0.0 },
    "102": { "type": "added", "vmid": 102, "name": "new-vm", "status": "running" },
    "101": { "type": "removed", "vmid": 101 }
  }
}

Error:

{ "error": "Proxmox credentials not found in backend inventory" }

Behavior:

• Polls every 5 seconds
• Template VMs are excluded
• Status changes and CPU changes > 2% trigger a diff
• Connection closes on credential errors

---

Project Structure

range42-backend-api/
|-- app/
|   |-- main.py                  # FastAPI app factory, CORS, vault lifespan
|   |-- core/
|   |   |-- config.py            # Centralized settings from env vars
|   |   |-- runner.py            # Ansible playbook execution engine
|   |   |-- extractor.py         # Structured result extraction from events
|   |   |-- vault.py             # Vault password file management
|   |   |-- exceptions.py        # Custom exception handlers
|   |-- routes/
|   |   |-- __init__.py          # Router assembly and prefix mapping
|   |   |-- vms.py               # VM lifecycle (list, start, stop, create, delete, clone)
|   |   |-- vm_config.py         # VM configuration (get config, set tags)
|   |   |-- snapshots.py         # VM snapshots (list, create, delete, revert)
|   |   |-- firewall.py          # Firewall (aliases, rules, enable/disable)
|   |   |-- network.py           # Network interfaces (VM and node level)
|   |   |-- storage.py           # Storage (list, download ISO, templates)
|   |   |-- runner.py            # Generic bundle/scenario runner
|   |   |-- debug.py             # Debug endpoints (ping, test functions)
|   |   |-- ws_status.py         # WebSocket real-time VM status
|   |-- schemas/
|   |   |-- base.py              # Shared Pydantic base models
|   |   |-- vms.py               # VM request/response schemas
|   |   |-- vm_config.py         # VM config schemas
|   |   |-- snapshots.py         # Snapshot schemas
|   |   |-- firewall.py          # Firewall schemas
|   |   |-- network.py           # Network schemas
|   |   |-- storage.py           # Storage schemas
|   |   |-- debug/               # Debug endpoint schemas
|   |-- utils/
|   |   |-- checks_playbooks.py  # Playbook path validation and resolution
|   |   |-- checks_inventory.py  # Inventory path validation and resolution
|   |   |-- text_cleaner.py      # ANSI escape code stripper
|   |   |-- vm_id_name_resolver.py  # VM ID to name resolution via Ansible
|-- tests/                       # Pytest test suite
|-- curl_utils/                  # Manual testing curl scripts
|-- playbooks/                   # Local Ansible playbooks (generic, ping)
|-- inventory/                   # Ansible inventory files
|-- Dockerfile                   # Multi-stage Docker build
|-- docker-compose.yml           # Compose service definition
|-- start.sh                     # Development startup script
|-- requirements.txt             # Python dependencies
|-- requirements.yml             # Ansible Galaxy requirements

---

Architecture

Request Flow

HTTP Request
  --> FastAPI route handler
    --> Pydantic schema validation
      --> Path / inventory checks (checks_playbooks.py, checks_inventory.py)
        --> runner.py (ansible-runner in temp directory)
          --> Extract structured results (extractor.py)
            --> JSONResponse (rc + result or log lines)

Key Design Decisions

• Temp directory per run -- Each playbook execution creates an isolated temp directory containing a copy of the playbook tree, inventory, and environment variables. The directory is cleaned up in a finally block to prevent leaks.

• Vault lifecycle -- On app startup, the lifespan context manager either reads VAULT_PASSWORD_FILE directly or writes VAULT_PASSWORD to a secure temp file. Both are cleaned up on shutdown.

• Two response modes -- Endpoints that accept as_json can return either structured data extracted from Ansible events ("result" key) or raw log lines ("log_multiline" array).

• Path traversal protection -- All playbook and inventory names are validated against ^[A-Za-z0-9_-]+(?:/[A-Za-z0-9_-]+)*$ and resolved paths are checked with is_relative_to() to prevent directory traversal attacks.

• No auth in this layer -- Authentication, ACLs, and rate limiting are handled by the Kong API gateway in front of this API. CORS is restricted to localhost origins only.

Route Prefixes

Prefix	Module	Purpose
/v0/admin/proxmox/vms/	vms.py	VM list and lifecycle
/v0/admin/proxmox/vms/vm_id/	vms.py	Single VM operations
/v0/admin/proxmox/vms/vm_id/config/	vm_config.py	VM configuration
/v0/admin/proxmox/vms/vm_id/snapshot/	snapshots.py	VM snapshots
/v0/admin/proxmox/firewall/	firewall.py	Firewall management
/v0/admin/proxmox/network/	network.py	Network interfaces
/v0/admin/proxmox/storage/	storage.py	Storage and ISOs
/v0/admin/run/bundles/	runner.py	Bundle execution
/v0/admin/run/scenarios/	runner.py	Scenario execution
/v0/admin/debug/	debug.py	Debug/test endpoints
/ws/vm-status	ws_status.py	WebSocket VM status

---

Development

Running Tests

# All tests
python3 -m pytest tests/ -v

# Specific test file
python3 -m pytest tests/test_checks_playbooks.py -v

# Specific test
python3 -m pytest tests/test_ws_helpers.py::TestComputeDiff::test_detects_status_change -v

Test Structure

File	Covers
test_api_smoke.py	App startup, OpenAPI schema, docs endpoints
test_routes_registered.py	Golden route reference (verifies all 69 routes are registered)
test_config.py	app/core/config.py settings and defaults
test_vault.py	app/core/vault.py VaultManager lifecycle
test_runner.py	app/core/runner.py log building
test_runner_internals.py	Runner helpers: envvars, cmdline, temp dir setup
test_extractor.py	app/core/extractor.py event parsing
test_exceptions.py	Custom validation error formatting
test_schemas.py	Pydantic request schema validation + backward-compat aliases
test_schemas_replies.py	Pydantic response schema validation
test_checks_inventory.py	Inventory name validation and path traversal detection
test_checks_playbooks.py	Playbook name validation and path traversal detection
test_ws_helpers.py	WebSocket helpers: diff computation, credential loading
test_route_debug.py	Debug endpoint integration tests (mocked runner)
test_route_vms.py	VM endpoint integration tests (mocked runner)

Route handler tests mock run_playbook_core() so no Ansible or Proxmox connection is needed.

The golden route reference (tests/fixtures/routes_golden.json) is a safety net that ensures refactoring never accidentally drops an endpoint. If you add or remove a route, update this file.

Manual Testing

Curl scripts for every endpoint are available in curl_utils/.

Code Conventions

• Imports: Absolute from app. (no relative imports except in __init__)
• Naming: req for request objects, rc for return codes, extravars for Ansible extra variables
• HTTP codes: 200 for Ansible success (rc=0), 500 for failure, 400 for validation errors
• Commit style: Conventional commits -- feat(scope):, fix(scope):, docs:, refactor:
• Branch naming: feature/description, fix/description, release/x.y.z

---

v1 API surface

See docs/v1-routes.md for the full endpoint map. Highlights:

• /v1/catalog/sources -- git source registration.
• /v1/catalog/entries -- cross-source browse + manifest detail.
• /v1/projects -- project CRUD + compose + validate.
• /v1/deployments -- create/list/get + attempts + preflight + events (SSE) + timings + cancel.
• /v1/proxmox/hosts -- host CRUD + health.
• /v1/health, /v1/health/ready, /v1/admin/stats.

Canonical error envelope per spec section 18.1 -- every /v1 error includes {error, message, code, details[], trace_id, timestamp}.

Git catalog onboarding

POST /v1/catalog/sources/default registers the public https://github.com/range42/range42-catalog repository on main without a token. It returns 200 with the existing or newly created source, and repeated requests reuse that registration. Registration does not clone the repository; refresh or browse it when ready.

For another repository, send POST /v1/catalog/sources with the Git server URL and a repository binding:

{
  "provider": "github",
  "base_url": "https://github.com",
  "auth_kind": "none",
  "repos": [{"owner": "range42", "repo": "range42-catalog", "branch": "main"}]
}

Register one repository per source so catalog entries have an unambiguous source and path. branch defaults to main; owner supports nested GitLab groups. Use the repository name without .git. Legacy requests without repos remain accepted, but refreshing an empty source returns 400 with SOURCE_REPOS_REQUIRED and registration instructions.

GET /v1/catalog/sources returns a page with items, total, offset, and limit. Every source response includes repos with repository IDs, names, branches, and last_refreshed_at. Refresh a registered source with POST /v1/catalog/sources/{id}/refresh, then browse GET /v1/catalog/entries?source_id={id}. Refresh reports how many repositories and recognized manifests it found; browsing currently clones again and is not a persistent catalog cache.

Catalog entry summaries and details include the checked-out Git commit in sha, so project attachments can pin the version that was browsed.

Private HTTPS repositories use auth_kind: "pat" with token_ref on creation. Rotate credentials through PATCH /v1/catalog/sources/{id} with {"auth_kind":"pat","token_ref":"..."}, or clear them with {"auth_kind":"none"}. Responses expose only has_token, never the token. Put credentials in token_ref, not in the server URL. Catalog registration is separate from deployment inventory credentials and host configuration.

Detached runner

See the runner lifecycle. The v1 runtime spawns ansible-runner run in an independent process session with per-attempt artifacts. After a hard API crash, recovery verifies process identity before adopting a live runner, restores recorded exit results, or reports an unknown outcome. Events pass through the redaction pipeline with replay deduplication. Full provisioning shares a lock inherited by the runner, including across API crashes. Graceful API shutdown detaches local observation while preserving the runner and its workspace-owned credentials; explicit cancellation signals it. Legacy scoped actions require their own operation playbook and use the same attempt reservation; unsupported actions return 409 OPERATION_UNSUPPORTED. Teardown preserves workspace inventory, credentials, and logs. See the operation contract.

Events + SSE

Each deployment owns ~/range42.config/<CODENAME>-<SCENARIO>/events.jsonl with event_seq as the cursor. SSE endpoint GET /v1/deployments/:id/events replays up to 5000 events then tails live writes. Deployments behind Kong/Nginx must set proxy_buffering off -- see docs/sse-proxy.md.

Local-FS invariant

Workspace root and SQLite DB must be on a local filesystem (ext4/xfs/ btrfs/zfs/tmpfs). NFS/CIFS/FUSE is refused at deployment create with HTTP 409 and code WORKSPACE_NON_LOCAL_FS.

---

License

GPL-3.0

Named API token roles and durable mutation audit are documented in named access and audit. Existing single-token deployments remain compatible.