diff --git a/docs/backup-and-restore.md b/docs/backup-and-restore.md index 6d936d762..b0c5a31b9 100644 --- a/docs/backup-and-restore.md +++ b/docs/backup-and-restore.md @@ -51,11 +51,22 @@ disk. Keeping a copy somewhere else is the owner's step: point `--output-dir` at another disk, or copy the file off the machine. Keep at least one older known-good copy. +In `v0.142.1-beta.1`, Ctrl-C can stop the CLI while Core keeps making the +live backup. Check `/api/backups` for its phase and wait for completion +before starting an update. [The cancellation fix](https://github.com/srcfl/ftw/pull/1615) +is not included in that beta. + The archive leaves out other backups and old rollback points. It carries a consistent copy of `cache.db`: savings history is costed from its past prices, and a price provider sends only today and tomorrow. Managed-driver links that point inside the persistent directory become relative links in the archive, so restore can move the data to another directory or machine. + +The archive also leaves out `cloud.key` and `cloud.key.tmp`. Pairing and +WebPush keys remain. After a full restore, FTW Cloud stays off until the +owner grants it again under a new home ID. Local control and stored goals +remain available. + Backup never follows these links to copy a host file; verification rejects link chains that escape the data directory or form a cycle. @@ -79,8 +90,9 @@ in the CLI environment if the site requires LAN auth. Set `state.backup_dir` to a mounted external directory to keep Core's own copies off the data disk; FTW must be able to write it. -In Docker 0.x, stop FTW and archive the whole data directory, as in -[Docker](native-beta.md#docker). +In Docker 0.x, use the release image's offline `ftw-backup` helper to create +the verified archive on another disk. See +[Back up and restore new Docker](native-beta.md#back-up-and-restore-new-docker). On older Docker installs (1.x–3.x), open **FTW Update Center → Full backups**, choose **Create full backup**, then **Download** and save the `.ftwbak` file @@ -168,6 +180,15 @@ Pass `-config` to `create` when the seed has a name other than `/config.yaml`. The config seed must be inside the data directory. + +From `v0.141.0-beta.1`, the helper records and checks a complete previous file +list before revert. An interrupted move or an older safety point without that +record stops before another import or revert can certify a partial directory. +Keep Core stopped and retain all directories. Restore a verified full backup +to a separate target; do not remove the check or discard newer readings. +A returned `safety_complete: false` refers to the retained rejected target, +which must not be reactivated. See [the migration guide](migration.md#if-import-or-return-stops). + ## Svenska – kortversion att skicka till en användare 1. Sätt i ett USB-minne eller anslut en annan disk till datorn där FTW körs. diff --git a/docs/migration.md b/docs/migration.md new file mode 100644 index 000000000..c0468deca --- /dev/null +++ b/docs/migration.md @@ -0,0 +1,462 @@ +# Move FTW with your data + +Keep settings, history, identity, credentials, drivers and stored goals when +moving to the new line. Use a full backup and a separate target copy. Keep +the source install and an off-host backup until the target and the return +path have passed their checks. [Svenska](#svenska). + +This draft targets `v0.142.1`, which has not shipped as stable. The published +beta is `v0.142.1-beta.1`. Data-copy tests do not establish that every old +installation can switch safely. The remaining paths below need work before +the guide can ship with stable. + +## Start from the installation you have + +| Source | Export and import | +|---|---| +| New 0.x native or Docker | Keep the existing data. Follow the [normal update](native-beta.md), including a verified full backup. | +| Old 0.x, 1.x, 2.x or SQLite-based 3.x | Export a full `.ftwbak`, or make a stopped copy of the entire data directory. Restore that backup on the separate target before its first configured start. | +| A DuckDB beta | Preserve the DuckDB file, hot SQLite, cold archives and settings together. Run [the exact target's offline converter](history-conversion.md) on the copied data before Core opens them. | +| Home Assistant app | Download a backup that includes FTW and keep its key. Use the [offline HA exporter](https://github.com/srcfl/ftw/releases/tag/ha-export-v1-beta.1) to check and copy its FTW component, then make a full FTW backup at the old `/data` root. The complete Supervisor route is still being qualified. | +| Only a backup remains | Identify its format and source version from its metadata. Use the checked full-backup route, or the matching old image on a separate recovery copy. | +| Custom paths, external files or another host platform | Inventory the actual mounts, config seed, databases, archives, keys, drivers and start rules first. Resolve external paths before activation. | + +Record the actual source version, data path, config path and startup owner. +The old ready-made Pi image uses Docker. A source running a 32-bit OS can +export a stopped copy to a 64-bit Linux target; it cannot run the new binary. +These source/platform combinations still need their own installation tests. + +Active SQLite settings take precedence over an old YAML seed. The config API +masks secrets and is not a full export. Include environment files and service +or Compose definitions needed to explain the source's paths and credentials. +Preserve shared MQTT and other services used by the equipment. + +FTW Cloud needs a new owner grant after a full-backup restore. The backup +keeps pairing and WebPush keys but leaves out `cloud.key` and its temporary +file. A restored Cloud grant reads as off until the owner turns Cloud on +again under a new home ID. Local operation and stored goals remain. + +## Export a complete source copy + +Use the source version's full-backup UI or helper when it has one. Verify the +finished archive and copy it off the source host. A verified server copy does +not establish that the downloaded file has the same bytes. + +For a source without a helper, stop Core and preserve its whole data tree, +including SQLite WAL files, keys, drivers and required archives. On a +separate 64-bit Linux copy, the target's `ftw-backup create` can open legacy +SQLite read-only without upgrading it. Pass `-core-version` with the verified +**source** version and `-config` if the config seed has another name. Keep +external files and their path mapping explicit; the helper requires its +config seed inside the data tree. + +For an external layout, keep the stopped source and its config seed intact. +Use a second writable copy to collect the database, required archives, keys +and driver files inside one data tree. Map each known file path in that +copy's active SQLite config and YAML seed; keep all other settings and +credentials. If the source has no stored config, keep YAML as its source +until Core imports it. Do not add a `config_database` reference to a database +that has no stored config. Verify the resulting full backup and restore it +on a separate target. An unresolved external path must stop the move. + +Do not start an old Core merely to inspect the migration copy. Preserve the +original bytes and use the full-backup inspector or converter. When a format +or source is unknown, keep the copy and resolve that case before starting it. + +For a Home Assistant backup, follow the exporter's included README. It needs +Linux and Python 3.11 or later, checks plain and protected v2/v3 components, +and keeps checked internal driver links. It refuses outside links and unknown +layouts. The extracted copy must be writable when the FTW helper opens old +SQLite; WAL/SHM sidecars may be needed. Keep the original backup unchanged. +Use the printed source version in `ftw-backup create -core-version`, and mount +the extracted data at `/data` when making the full backup. That preserves the +source root for verified relocation; a host path is not a replacement for it. + +## Restore a full backup on a new native target + +Prepare a new Linux host or Pi card with [the exact release package](native-beta.md#install). +The installer starts an empty setup service. Stop and disable it before +import, so it cannot start configured Core during an unfinished move. + +Copy the verified full backup to `/var/tmp/ftw-move.ftwbak`. Use the exact tag +installed on this target in place of `v0.X.Y`. The following block verifies, +restores and fixes ownership. It leaves Core stopped: + +```bash +( + set -eu + tag=v0.X.Y + archive=/var/tmp/ftw-move.ftwbak + helper=/opt/ftw/releases/$tag/ftw-backup + sudo test -x "$helper" + sudo systemctl disable --now ftw + sudo "$helper" verify -archive "$archive" + sudo "$helper" restore -archive "$archive" -data /var/lib/ftw -yes + sudo chown -R ftw:ftw /var/lib/ftw +) +``` + +Restore checks the archive before replacing data and prints the retained +previous-data directory. Keep that path. It maps archive-internal config +paths to `/var/lib/ftw`, while keeping other stored settings. A directory move +with unresolved external paths stops before activation. Do not remove the +path check to make a restore pass. + +For DuckDB, complete the offline conversion now, using the matching converter +package and all copied sources. Keep Core stopped until it succeeds. A file +that changed during conversion stops resume; keep the original and follow +the converter's diagnosis. + +For a new Docker target, use its separate Compose project and the +[verified restore recipe](native-beta.md#update-new-docker). Keep Core +stopped through any conversion. Do not attach new Core to the source's live +data volume. + +## If import or return stops + +Keep both Core processes and their autostart off. Retain the full backup and +all active, staging and safety directories. Do not delete a hidden directory +or bypass a check to make the command succeed. + +The new helper checks a restore point's complete file list before reverting. +It refuses a partial point, an older point without that record, or a retry +that would treat a partly moved active directory as complete. Recover the +verified full backup into a separate directory instead; retain the current +files too, including readings written after that backup. + +If recovery returns `safety_complete: false`, the previous complete data have +returned, but the retained rejected target is partial. Keep it for inspection; +do not start Core from it or try to switch back to it. Check the recovered +installation and its start rules before enabling control. + +## Switch control and verify + +1. Stop old Core and its updater. Disable the confirmed old systemd, Compose, + timer or custom startup path. In Home Assistant, also disable **Start on + boot** and **Watchdog**. Keep needed MQTT running. +2. Confirm that the old process has stopped. A port error or one visible + dashboard does not establish that only one Core runs. +3. Start the prepared target. On native, run + `sudo systemctl enable --now ftw`. Use the saved API port; migration keeps + a custom port too. +4. Check the actual running version and binary, saved settings and credentials, + key identity, goals, representative historical periods, archive exports, + fresh device readings and new history writes. Check the plan and actual + equipment feedback before accepting control. +5. At a safe time, reboot the target and check again that old Core stays off, + exactly one Core runs, the imported data remain and new writes continue. + +If a check fails, stop new Core, preserve its data and diagnose. Return with +the matching old version and its saved data, with new Core stopped. Restore +only the source start rules recorded before the move. Keep data collected on +the target; returning does not merge those readings into old Core. + +## What the copy tests cover + +The published `v0.142.1-beta.1` Core passed healthy, offline and invalid +balance cases over two starts each on separate synthetic Linux copies. +Known meter values remain in history while missing flows and totals stay +unknown. Energy today shows measured subtotals with a gap mark; the full +range remains unknown when measurements are incomplete. It also records the time spent in forecast connection, query, scan +and decode phases when a read fails. The timeout cause remains unproven. +These checks alone do not qualify the older migration routes against this beta. + +The published `v0.142.1-beta.1` helper also passed full backup, verification, +portable restore and revert on separate copies of published HA 3.8 split +SQLite and current partial/quality history. Historical rows, settings and +identity remained after explicit data-root and restore-generation mappings. +Revert kept new target writes in its safety directory. These are data-copy +checks; the HAOS, MQTT and installation tests below retain their own versions. + +The same published Core and helper passed a separate Cloud restore check +without network access. Core created a local grant on the source copy. A +verified full backup omitted the Cloud key, and two restored starts kept +Cloud off. A new explicit grant used a different home ID. Source keys and +modes, pairing identity and WebPush identity remained. This tests local +restore behavior, not a connection to the Cloud service. + +The published `v0.142.1-beta.1` Core, helper and converter also passed 30 +checks from published `v3.2.0-beta.1` with retained Parquet and 500,000 extra +synthetic DuckDB rows. A process kill interrupted the copy before publication; +resume checked the source, and a changed source stopped it. Two target +starts and two portable-restore starts kept history, goals, settings and +identity. Return and revert retained both copies. This does not qualify +an installation change, Pi/SD performance or physical power loss. + +A separate protected full backup and full return passed on a separate HAOS 18.3 +ARM64 VM, with Supervisor 2026.09.3 reporting supported and healthy. Its +published HA 3.8 FTW source used synthetic data. Export and full FTW backup +with the exact published `v0.141.4-beta.1` Core/helper passed two target +starts, new writes and revert on a separate Linux ARM64 data copy. Old +chart rows, two archive hours, settings, credentials, goals and identity +remained. The checks verify root relocation and the older planner and SoC +field mappings rather than requiring those fields to keep their old form. +Native installation and physical cutover need separate proof. + +The protected HAOS return restored a synthetic template sensor's entity ID, +unique ID and state, and both HA and FTW test files. Old FTW still needed +the owner recovery command below before it could start. That command kept +all 61 checked files' bytes and modes and both internal links. The matching +old Core then returned with its chart rows and two archive hours. The VM +and FTW test targets are stopped; their backups and data remain. +This template check does not test FTW MQTT entities, every community app, +real equipment or power-loss recovery. + +A separate MQTT move on that own HAOS VM kept the same broker running while +only the selected FTW Core ran. The published HA 3.8 source and exact +`v0.141.4-beta.1` target kept all 44 recorded MQTT entity IDs, unique IDs +and device links through two target starts and matching old return. The same +grid sensor reported 700 W, then the target's new synthetic 800 W, then the +old source's 700 W. Old chart rows and both stored hours remained; new ticks +had no failed or rejected writes. Both full backups remain off the VM. + +The checks compare credentials and identity without exporting their values. +They verify the old Balanced forecast margin's move to the current Balanced +style, alongside the root, planner and SoC mappings. A guest OS reboot with +the old app's Start on boot, Watchdog and auto-update off started neither +old Core nor the removed target. This covers one synthetic MQTT setup and a +Docker data target on HAOS. It does not qualify every entity or command, +community layout, native installer, real device or power-loss case. + +The published `v0.141.2-beta.1` Core and helper preserve partial history in +a full backup. Small synthetic meter/battery cases passed two starts, +portable restore and return: a working meter keeps recording while the +battery is offline, and missing flows and totals stay unknown. A balance +warning also leaves measured flows in the record. The browser shows the +gap and warning, then new values after recovery. These checks do not qualify +the older installation routes below against this newer beta; their actual +tested target remains `v0.141.0-beta.1`. + +The published `v0.141.3-beta.1` helper also exports the older split SQLite +history, whose schema has none of the six newer partial/quality tables. +It refuses an incomplete set. Actual plain and protected Supervisor full +backups from a synthetic published HA 3.8 source passed export, portable +restore, two starts per route and revert with the published 0.141.3 packages. +Old charts, stored hours, config, credentials, goals, identity and managed +driver choices remained. The trial used the official unsupported Supervisor +devcontainer, not a supported HAOS host or real equipment. + +The full Supervisor return job restored files, but the old app failed to +start because its private files had the wrong owner. The separate +[owner recovery command](https://github.com/srcfl/home-assistant-addons/blob/main/docs/restore-data-owner.md) +now provides a checked route for that failure. Run it in the stopped app's +matching image with only its verified data mounted. It preserves bytes, +private file modes and internal links, and stops on external or broken links, +hard links or extra mounts. Keep Start on boot and Watchdog off; record the +image and data path before Supervisor removes the stopped container. + +A complete protected return job on that separate test instance passed the +command, old Core start, retained chart rows and two archive hours, then a +final stop. Its source was the same published HA 3.8 image. This command +does not publish or update the retired app line. +Automatic Supervisor return remains unqualified. This older devcontainer +check did not test FTW MQTT entities. +Keep the original app and its data; do not treat a completed backup job as +proof that the app can return without help. + +The published `v0.141.0-beta.1` packages have passed representative +SQLite/settings copies from 0.130.4, 1.15.0, 2.3.2 and 3.8.0-beta.1. +The matching Core and converter pair also passed DuckDB copies from +3.2.0-beta.1 and 3.4.2-beta.4. Each test uses the published old binary or +image and synthetic data. The checks retain exact old chart rows, settings, +credentials, goals and identity over two target starts, new writes and +return to the matching old Core. + +Those four SQLite sources also passed full-backup restore to both the same +data path and a new path with the published target helper. Two synthetic +Parquet rows in the old 2.x long format retain their bytes and JSON/CSV +exports over two target starts. Stored settings, credentials, goals, +identity and new writes remain, and revert returns the previous target. +These small cases do not cover every older cold format or external file. + +A separate 2.3.2 archive test passed 37 checks with 1,036,800 synthetic +Parquet values in ten daily files and twelve series. Restore to the same +and a new root retained exact hourly counts, sums, minimums, maximums and last timestamps +over two starts. JSON/CSV exports, settings, goals, identity, new writes, +revert and raw exports from the matching old Core passed. The compressed +files total about 9 MiB; this tests row count on Linux ARM64 in Docker, +not a large disk footprint, Pi/SD performance or physical power loss. + +DuckDB tests include hot SQLite, imported or retained long-format Parquet, +500,000 extra raw values, interrupted conversion and resume, generation +checks and portable backup/restore/revert. These copies do not qualify every +old schema, archive format, platform or complete installation change. + +Supervisor-format tests cover plain, compressed and encrypted version 2/3 +components using the published 3.8 HA image. The offline exporter and `v0.141.0-beta.1` +copy tests also cover synthetic managed-driver choices and internal links. +They do not cover a complete Supervisor job, every community app, a signed +driver-channel move or HA entity continuity. An isolated Linux systemd test +does not establish a physical Pi cutover or a power-loss recovery. + +Four separate managed-driver copy tests also passed with the published +`v0.141.0-beta.1` helper and Core. Current and older default manifests, +same and new data roots, two starts and revert keep the selected driver +active and preserve its previous-version path. These checks use synthetic +records and valid file hashes; they do not prove a signed-channel install. + +A later two-host test passed 39 checks from the published 3.8 Core and +helper to the published `v0.142.1-beta.1` native package. The unchanged +installer refused the source's existing `ftw` account and installed on a +fresh Debian 13 ARM64 target. Full backup transfer, two target starts and +matching old return retained 24 exact chart rows, two stored hours, settings, +credentials, goals and keys. The check compared every selected old config +field after explicit path, planner, SoC and forecast-margin mappings. +The source data stayed intact and new target writes had no errors. + +Real systemd rules kept only the selected Core running after both test +machines restarted, including after return. Both stopped full backups were +verified off the guest, and both machines ended stopped with data retained. +These OrbStack machine restarts test Linux service startup, not a physical +Pi power cycle. The source has no Parquet files; this test does not add +Parquet coverage, every custom layout, in-place installation or equipment +control. + +A separate two-host Linux ARM64 test moved a synthetic 3.8 custom systemd +source to a fresh `v0.141.0-beta.1` native target. The exact published old +helper created the full backup. Both hosts restarted with only the selected +Core enabled. The checks retained the source's exact chart rows and stored +hours, settings, credentials, goals and keys, verified new writes, and +returned to the matching old binary while keeping the target data. The +installer ran on the fresh target; this does not qualify an in-place +install, every custom service, physical equipment or a Pi power cycle. + +A separate same-host test used the [native pilot](self-update.md#pilot-an-older-native-systemd-site) +with a synthetic 3.8 systemd source: `/opt/ftw/ftw`, the `ftw` user and a +data bind. The corrected operator CLI and published `v0.141.0-beta.1` +package passed 24 checks. Charts, stored hours, settings, credentials, +goals, identity and new writes survived the switch and a machine restart +with one Core. Return to the matching old Core passed another restart, +and both off-host backups remain. This covers that Linux pilot layout; +it does not qualify every native install or physical power loss. The +normal fresh-host installer still refuses an existing installation. + +The published 2.3.2 Docker image also passed a move to `v0.141.0-beta.1` +using two Compose projects and separate named data volumes. A synthetic +MQTT meter and an independent subscriber used the same broker throughout +backup, two target starts and return to old Core. The checks retained exact +old chart rows, settings, credentials, goals, identity and Lua files, and +verified new writes. The broker and subscriber kept their process start +times. The test checked saved Compose and container restart policies; it +did not reboot Docker or the host, run the old updater, or control equipment. + +A later ARM64 Docker test passed 62 checks from the published 2.3.2 Core +and updater to the exact published `v0.141.4-beta.1` Core/helper. The old +updater answered on its Unix socket. Separate data trees and saved Compose +and container policies kept old Core and updater off while the beta target +continued writing after a verified Docker daemon restart and guest OS +reboot. Shared MQTT recovered with fresh synthetic readings. Both target +starts and old return retained the exact 25 source chart rows, two Parquet +values, settings, credentials, goals and identity. Full backups are verified +off the VM, including the final stopped target data; revert keeps new data. + +The test used an own QEMU/HVF HAOS guest as a Docker host. Supervisor marked +standalone images as `unsupported/software`; its normal check returned to +supported and healthy after cleanup. This does not qualify a standalone FTW +Docker installation as a supported HAOS app, every host startup unit, +old updater jobs, the optional old optimizer, physical equipment or power +loss. The test VM is shut down and all proof containers are removed. + +A separate test moved the published 2.3.2 AMD64 Docker image to the published +`v0.141.0-beta.1` ARM64 Core and helper. All 29 checks passed, including two +target starts, new writes and return to old Core with data, settings, +credentials, goals, identity and shared MQTT retained. Both binary architectures +were checked. The AMD64 source ran through emulation on Apple Silicon. +This covers the tested Linux Docker copy across CPU architectures; it does +not qualify 32-bit sources or native macOS/Windows paths. + +A separate test passed 42 checks from one custom Linux 386 build of the +unchanged 2.3.2 source to the published `v0.142.1-beta.1` ARM64 Core and +helper. The ELF32 source ran under QEMU user emulation and wrote current +history with millisecond timestamps. A verified full backup from a stopped, +writable export copy kept the source unchanged. Two ARM64 starts retained +25 exact old chart rows, two Parquet values, stored goals and credentials, +identity and all old YAML config fields after checked path, planner, SoC +and PV-unit mappings. New writes had no errors or rejected ticks. Return +to the matching 32-bit build and revert passed; the old source, backups +and new target database remain. All test containers are stopped. +This is a data-copy check for that custom build. It is not a published +32-bit package, an ARMv7/Pi test, a 32-bit OS, an installer change, reboot +or physical power-loss proof. + +A native macOS ARM64 copy test passed 45 checks from an own Mach-O build +of the unchanged 2.3.2 source to the published `v0.142.1-beta.1` Linux +ARM64 Core and helper. The macOS process wrote new SQLite rows with all +network access denied; its source HTTP API was blocked too. A separate, +stopped Linux export copy kept the original macOS data unchanged. Two Linux +starts retained 25 exact old chart rows, two Parquet values, stored goals +and credentials, identity and all old YAML config fields after checked +path, planner, SoC and PV-unit mappings. New writes, native macOS return +and revert passed. Backup bytes were checked outside the export container, +on the same physical Mac. All source and target processes are stopped. +This covers that custom data-copy layout. It is not a published macOS +package, source API or installer test, Windows migration, a backup on a +separate physical host, equipment or power-loss proof. + +One external Linux layout from published 2.3.2 also passed 36 checks with +the published `v0.141.4-beta.1` ARM64 Core and helper. Its database used a +subdirectory, while the config seed, driver files and Parquet archive used +other paths. The helper refused an external config seed and stopped a move +with unresolved paths before replacing target data. A separate copy with +explicit path mappings retained 25 old chart rows, two Parquet values, +settings, credentials, goals, keys and driver files over two target starts +and return to old Core. The check compared every old config field, with +explicit mappings for paths, old planner choice and units. Revert kept new +target data. This covers that stopped-copy layout, not every external mount, +an installer change, physical equipment or power loss. + +The published `v0.141.0-beta.1` Linux ARM64 backup helper also passed process +crash tests during preservation, activation, return and a later extraction. +The tests check that incomplete copies cannot replace active data, all old +and new files remain, and a complete backup can restore on a separate target. +These tests use synthetic files and do not prove recovery from physical +power loss or an interrupted installation change. + +Large archives on Pi/SD, other external layouts, cross-platform and other 32-bit sources, +complete Supervisor control, interrupted installation changes and physical cutover +remain release requirements. Unknown formats keep their source data and +require a checked import path. + +## Svenska + +Ta med dina inställningar och data. Börja med en full backup av gamla FTW +och behåll den gamla installationen. Kontrollera filen efter kopiering till +en annan disk eller dator. Använd en separat värd eller ett nytt Pi-kort som mål. + +1. Identifiera gamla versionen, dataformatet och de faktiska filvägarna. + Aktiva SQLite-inställningar gäller före gammal YAML. Maskerad API-config + saknar hemligheter och räcker inte som backup. +2. Exportera allt som behövs: data, config, lösenord, nycklar, mål, + drivrutiner och arkiv. Home Assistant-backup behöver sin nyckel. Använd + den länkade exportören för FTW-komponenten och följ dess README för full + backup med den gamla roten `/data`. + FTW Cloud kräver ett nytt godkännande efter full återställning. Backupen + utelämnar Cloud-nyckeln; återställd Cloud är avstängd tills ägaren slår på + den igen med ett nytt home-ID. Pairing, WebPush och sparade mål består. + Ligger filer på externa sökvägar: behåll den stoppade källan och samla + filerna i en extra skrivbar kopia. Mappa bara kända sökvägar i kopians + aktiva SQLite-config och YAML-seed. Behåll övriga inställningar. Lägg + inte till `config_database` om databasen saknar lagrad config. Olösta + externa sökvägar ska stoppa flytten före aktivering. +3. Installera exakt målversion på den separata Linuxmaskinen. Stoppa och + stäng av nya tjänstens autostart före import. Använd blocket ovan för en + verifierad `.ftwbak`; det lämnar Core stoppad. +4. Har källan DuckDB, kör målversionens konverterare på kopian innan Core + startar. Vid ett fel: behåll båda kopiorna och lös felet. +5. Stoppa gamla Core och updater samt deras autostart. Stäng också av Home + Assistants **Start on boot** och **Watchdog**. Behåll nödvändig MQTT. +6. Starta nya Core. Kontrollera config, mål, identitet, gamla tidsperioder, + arkiv, färska enhetsvärden, plan och nya skrivningar. Upprepa efter en + planerad omstart och kontrollera att gamla Core fortfarande är avstängd. + +Vid ett avbrott eller fel: håll båda Core och deras autostart avstängda. +Behåll backupen och alla data-, staging- och återställningskataloger. En +ofullständig eller äldre återställningspunkt utan kvitto ska inte tvingas +igång. Återställ fullbackupen till en separat katalog och behåll också data +som skrevs efter backupen. `safety_complete: false` betyder att den sparade +avvisade målkopian är ofullständig; använd den inte för en ny start. + +Vid återgång: stoppa nya Core först och använd rätt gammal version med dess +sparade data. Behåll också nya data. Guidens provade datafall är avgränsade; +hela flytten från Supervisor, egna layoutar och olika värdplattformar behöver +fortfarande egna prov före stable. diff --git a/docs/native-beta.md b/docs/native-beta.md index 6fb3c9995..95dac4228 100644 --- a/docs/native-beta.md +++ b/docs/native-beta.md @@ -1,21 +1,41 @@ # Install and update FTW -**FTW 2.x and 3.x will receive no further updates. All new development and -releases use the new 0.x line. Move to it now to follow the latest fixes and -features. Do not install 3.x beta or use it as an intermediate upgrade.** +**This draft targets `v0.142.1`; stable has not been published yet.** FTW +2.x and 3.x get no more updates. [Svenska](setup-guide/update-sv.md). -The new line starts at `v0.131.0-beta.1`. Older 0.x releases, up to 0.130.x, -belong to the retired line too. The lower version number is deliberate. -Choosing `beta` in an old installation does not move it to the new line. +## Quick answer -You can switch now with a new setup and separate data. The guided migration -that preserves old settings, history, identity and goals is not ready yet. -If you need those data moved before switching, get help for your exact -installation; do not copy old databases into a new install. - -Use this guide for both people and agents. [Svenska](setup-guide/update-sv.md). -The owner runs the host; FTW supplies the commands. Native systemd and Docker -on 64-bit Linux use the same release package. +| You run | Do this | +|---|---| +| New FTW 0.x, native | Open **More → Version → Update FTW** in the web UI, or run `ftw update` on the box. Nothing else. | +| New FTW 0.x in Docker | Set `FTW_VERSION=v0.142.1` in `.env` and run `docker compose up -d --build`. [Details](#update-new-docker) | +| No FTW yet, on 64-bit Linux or a new Pi card | [Install](#install) with the stable tag. | +| Old FTW 2.x, 3.x or the old Pi image | Move data and config through a full backup to a second SD card or host. [Steps](#coming-from-an-older-ftw) | +| Home Assistant app | Keep the app backup and move to another Linux host; keep Home Assistant on MQTT. [Steps](#coming-from-an-older-ftw) | +| Windows or macOS | Not supported. Use a Raspberry Pi or another 64-bit Linux machine. | + +While an update runs, keep the box powered and do not press Restart. Wait for +the update to finish. On native, `ftw rollback` returns to the previous +compatible release. For Docker, follow [Update new Docker](#update-new-docker). + +**Keep your settings, history, identity and goals when you move.** Follow +[Move with your data](migration.md) to choose the source format, export a full +backup, import a separate copy and check the result before switching control. +A new site uses setup; an existing site uses its saved settings. Keep the old +installation and an off-host backup for return. + +The migration guide names the data cases tested with the published beta and +the work still needed for other layouts. An import error is a stop point: +keep both copies and resolve it before starting Core. The masked config API +is not a full settings export. DuckDB uses the exact target's +[history converter](history-conversion.md); it needs no intermediate release. + +The rest of this page has the details, for people and agents. The owner runs +the host; FTW supplies the commands. Native systemd and Docker on 64-bit Linux +use the same release package. The new line starts at `v0.131.0-beta.1`; older +0.x releases, up to 0.130.x, belong to the retired line. Choosing `beta` in an +old installation does not move it to the new line, and 3.x beta is not a step +on the way. ## Choose your path @@ -97,10 +117,10 @@ running Core; inspect containers and their restart rules. A service called ## Coming from an older FTW Move directly to the new line using one of the paths below. Keep the old -installation and its data for recovery; the new setup does not import them. -Before switching, save device addresses, goals and schedules, and make a -verified full backup using the tools your installed version supports. Keep -that backup off the host. Not every old version has the same backup UI. +installation and its data for recovery. Use [Move with your data](migration.md) +for a verified full backup and an import into the separate target. Keep the +backup off the host. Check saved settings, credentials, goals and historical +periods on that target before it takes control. Not every old version has the same backup UI. Calendar support is gone: use loadpoint targets and ready-by schedules. **Only one Core may control the equipment.** An old Core can keep controlling @@ -125,7 +145,8 @@ new Core, and stop new Core before returning to the old install. control the same equipment. If the old card supplied MQTT, arrange a broker for the new setup. The old card's broker does not move with FTW. 6. **In SSH on the new card:** follow [Install](#install), then open - `http://:8080/setup` and set up the site again. + [import the saved data](migration.md#restore-a-full-backup-on-a-new-native-target) + before the new Core takes control. The saved API port may differ from 8080. 7. Follow [Verify the switch](#verify-the-switch). To return, shut down and put the old card back. Also stop any new FTW you ran @@ -143,17 +164,26 @@ because the native installer refused the old card. `~/forty-two-watts`; use the path found on this host. Save its Compose files and `.env` with the backup so recovery uses the same old version. 2. Check whether that project also supplies Mosquitto or another needed - service. Arrange continued MQTT before stopping the whole project. -3. Stop the old project with its actual files and project name. For a standard - project, run `docker compose down` from its directory (with `sudo` if - required). Do not add `-v`, delete data, or stop Docker as a whole. + service. Keep those services running. Use `docker compose config --services` + with the source's actual files and project name to identify them. +3. Stop only the confirmed Core and updater services. For a Core service named + `ftw`, run `docker compose stop ftw`, then + `docker update --restart=no "$(docker compose ps -aq ftw)"`. Repeat for an + updater if one exists. Set `restart: "no"` for those services in the saved + source Compose files too. Check that their containers are stopped and their + restart policies are `no`. Do not run `docker compose down` while that + project still supplies needed MQTT. Do not add `-v` or delete data. 4. Check systemd, timers and custom scripts that could recreate or start old Core or its updater at boot. Disable only the confirmed old start path. Docker's container restart policy is only one possible start path. -5. Follow [Docker](#docker) in a new directory with separate empty data, - normally `~/ftw-local`. If that directory already exists, inspect it first; - do not overwrite an earlier test or attach the old data directory. -6. Set up the site and follow [Verify the switch](#verify-the-switch). + When keeping the broker, name its service in later Compose commands. + `docker compose up -d` for the whole old project starts old Core again, + even with `restart: "no"`. +5. Prepare [Docker](#docker) in a new directory, normally `~/ftw-local`. Keep + new Core stopped until [the copied data are imported](migration.md). Inspect + an existing target directory before using it. +6. Check the imported settings and data, then follow + [Verify the switch](#verify-the-switch). To return, stop the new project first, then start the saved old project with its original files and version pin. Restore only the start rules you disabled. @@ -172,11 +202,12 @@ with a separate FTW host still works; see [Home Assistant](ha-integration.md). return to [the second-card steps](#raspberry-pi-use-a-second-sd-card) first. This section does not update or migrate an old installation. -Choose an exact published new 0.x beta from [Releases](https://github.com/srcfl/ftw/releases). -Check that it includes the Linux package for your architecture and its SHA-256 -file. Do not use `releases/latest`: it still points to the retired 2.x line. -Copy the tag from the release title; do not retype it. Replace only -`v0.X.Y-beta.N` in the block below. Paste the **whole block, including `(` +Use the newest stable tag from [Releases](https://github.com/srcfl/ftw/releases), +`v0.142.1` or later with no `-beta` in it. Choose a beta only if you test new +builds. Check that the release includes the Linux package for your +architecture and its SHA-256 file. Do not use `releases/latest`: it still +points to the retired 2.x line. Copy the tag from the release title; do not +retype it. Replace only `v0.X.Y` in the block below. Paste the **whole block, including `(` and `)`**, into SSH on the new host. It stops at the first error and downloads that release's installer to a temporary file. `--fresh-host` confirms that this host has no existing FTW site, including @@ -185,7 +216,7 @@ stopped containers or data in a custom directory: ```bash ( set -eu - tag=v0.X.Y-beta.N + tag=v0.X.Y if [[ ! "$tag" =~ ^v0\.([0-9]+)\.[0-9]+(-beta\.[0-9]+)?$ ]] || (( 10#${BASH_REMATCH[1]} < 131 )); then echo "STOP: copy an exact published new 0.x tag from Releases." >&2 exit 1 @@ -228,7 +259,7 @@ closing or reloading the page does not cancel or repeat the update. ftw status # version, releases, last update, disk, health ftw update # install the next release on the saved channel ftw rollback # return to the previous release -ftw backup --output-dir /media/usb/ftw # make a verified backup and copy it off the disk +ftw backup # make a verified full backup; prints its path ftw support # write the redacted support file for a report journalctl -u ftw -n 100 # logs sudo systemctl restart ftw # restart @@ -239,9 +270,8 @@ when the box is current or the update succeeded, and 1 when a step failed. ## Update a native box -Run the commands on the box. From another computer, connect with -`ssh @` first. The native web UI shows the version and -update notice; it has no Update button. +Use **More → Version → Update FTW** in the web UI, or run the commands on the +box. From another computer, connect with `ssh @` first. ```bash ftw status # check the running version, saved channel and health @@ -250,9 +280,13 @@ ftw status # confirm the running version and health afterward journalctl -u ftw --since "10 minutes ago" --no-pager ``` -`ftw update` follows the saved channel. Use `ftw update --channel beta` if -you mean to switch this site to beta. Publishing a beta does not update a -box by itself. Do not rerun the installer for a routine Core update. +`ftw update` follows the channel you chose with `--channel`. If you never +chose one, it follows what the box runs: a stable release follows stable, a +beta follows beta. Beta also offers a newer stable, so a beta box moves to +stable when one comes out. To stay on beta after that, run +`ftw update --channel beta`; to leave beta, run `ftw update --channel stable`. +Publishing a release does not update a box by itself. Do not rerun the +installer for a routine Core update. The command shows each step and waits until Core is ready. Check that its `Now running` line and the final `ftw status` name the expected version, @@ -335,7 +369,7 @@ the installer from that release. Replace the placeholder with its exact tag: ```bash ( set -eu - tag=v0.X.Y-beta.N + tag=v0.X.Y if [[ ! "$tag" =~ ^v0\.([0-9]+)\.[0-9]+(-beta\.[0-9]+)?$ ]] || (( 10#${BASH_REMATCH[1]} < 131 )); then echo "STOP: copy an exact published new 0.x tag from Releases." >&2 exit 1 @@ -360,8 +394,8 @@ include both plugins. Do not reinstall Docker on a running site without checking its existing services. Docker Desktop has a different network setup; this Linux host-network recipe does not establish LAN device access there. -Copy an exact published new 0.x tag from [Releases](https://github.com/srcfl/ftw/releases). -Replace only `v0.X.Y-beta.N` and paste the **whole block, including `(` and `)`**. +Copy the newest stable tag from [Releases](https://github.com/srcfl/ftw/releases). +Replace only `v0.X.Y` and paste the **whole block, including `(` and `)`**. It downloads both files to a temporary directory before creating the project, so a failed download leaves no project behind. Correct the tag or connection and paste the whole block again. It stops at the first error and refuses an @@ -371,7 +405,7 @@ else; do not delete it to make the command pass. ```bash ( set -eu - tag=v0.X.Y-beta.N + tag=v0.X.Y if [[ ! "$tag" =~ ^v0\.([0-9]+)\.[0-9]+(-beta\.[0-9]+)?$ ]] || (( 10#${BASH_REMATCH[1]} < 131 )); then echo "STOP: copy an exact published new 0.x tag from Releases." >&2 exit 1 @@ -449,20 +483,76 @@ image: `docker image ls ftw-local` lists them, and `docker image rm ftw-local:` frees the space of one you no longer need. -Back up and restore the whole data directory while FTW is stopped: +### Back up and restore new Docker + +Use the `ftw-backup` tool in the same release image. It verifies a portable +`.ftwbak` archive and retains replaced data during restore. These commands +apply to the new Docker project above; they do not migrate an older install. +Run them from `~/ftw-local`. Only the `ftw` service stops. + +Mount another disk first and replace `/media/usb/ftw-backups` with a dedicated +directory on that disk. The container runs as UID 100, GID 101. The backup +block creates that directory with those owners and private permissions: + +```bash +( + set -eu + backup_dir=/media/usb/ftw-backups + sudo install -d -o 100 -g 101 -m 0700 "$backup_dir" + docker compose stop ftw + docker compose run --rm --no-deps --entrypoint /app/ftw-backup \ + --volume "$backup_dir:/backup:z" ftw \ + create -state /app/data/state.db -data /app/data -output /backup -progress + docker compose start ftw +) +``` + +The final JSON names the verified archive under `/backup`; the file is in +`backup_dir` on the host. Keep the printed name. A backup error stops the +block and leaves FTW stopped. Resolve it before restarting; do not treat a +partial file as a backup. + +To restore, use the exact `.ftwbak` name and a compatible release image: ```bash -docker compose stop -sudo tar -czf /media/usb/ftw/ftw-data-$(date +%F).tar.gz data -docker compose start - -# restore -docker compose stop -sudo mv data data.before-restore-$(date +%F-%H%M) -sudo tar -xzf /media/usb/ftw/ftw-data-.tar.gz -docker compose start +( + set -eu + backup_dir=/media/usb/ftw-backups + archive=ftw-full-backup-YYYYMMDDTHHMMSS.ftwbak + test -f "$backup_dir/$archive" + docker compose stop ftw + docker compose run --rm --no-deps --entrypoint /app/ftw-backup \ + --volume "$backup_dir:/backup:ro,z" ftw \ + restore -archive "/backup/$archive" -data /app/data -yes + docker compose start ftw +) +``` + +Restore verifies and extracts before replacing data. Any helper error stops +the block before Core starts. Earlier data stays under the printed +`/app/data/.ftw-pre-restore-…` path. Keep it until you have checked config, +identity, old and new history, goals, fresh device readings and planning. +Wait for healthy status in `docker compose ps`, then run +`docker compose exec ftw ftw status`. + +To return to the retained data, replace `safety` with that printed container +path. Use the compatible image for those data: + +```bash +( + set -eu + safety=/app/data/.ftw-pre-restore-YYYYMMDDTHHMMSS + docker compose stop ftw + docker compose run --rm --no-deps --entrypoint /app/ftw-backup ftw \ + revert -data /app/data -safety "$safety" -yes + docker compose start ftw +) ``` +These helpers accept `.ftwbak`, not an old raw `.tar.gz`. Keep raw archives +and the matching old installation for recovery; do not rename them to +`.ftwbak`. See [full backup and restore](backup-and-restore.md). + To remove it, run `docker compose down`, remove the `ftw-local` images and delete `~/ftw-local`. diff --git a/docs/operations.md b/docs/operations.md index 338969ceb..d7cbca493 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -10,7 +10,7 @@ safely when it is unavailable. ## Install [Install and update FTW](native-beta.md) is the full guide and explains how -to choose a published new 0.x beta. Use the installer from the same tag you +to choose a published stable or beta tag. Use the installer from the same tag you install, on a fresh 64-bit Raspberry Pi OS, Debian or Ubuntu host. Follow the complete [native install block](native-beta.md#install). It stops @@ -430,8 +430,8 @@ the `ftw` command and the unit move with `install.sh --refresh` ## Release recovery -No 0.x stable has shipped; run a beta only on an agreed test site. A beta and -its promoted stable build identify the same commit. +Stable is the default for every site; run a beta only on an agreed test site. +A beta and its promoted stable build identify the same commit. - Native: `ftw rollback` returns to the previous release when it reads the same data. If Core does not start and `ftw` cannot reach it, roll back diff --git a/docs/self-update.md b/docs/self-update.md index a698bfcea..c086418a8 100644 --- a/docs/self-update.md +++ b/docs/self-update.md @@ -1,8 +1,9 @@ # Updates and release channels [ADR 0007](adr/0007-self-updating-binary.md) defines the move to a native -Core. `v0.131.0-beta.1` is the first published native beta. No native stable -release or guided migration for an existing box has shipped. +Core. `v0.131.0-beta.1` is the first published native beta and `v0.140.5` +the first native stable. Guided migration for an existing old box has not +shipped. There are two channels: @@ -43,11 +44,13 @@ exact 0.x tags. Publication never installs a release on a site by itself. ### Pilot: an older native systemd site This is not the user migration path. `migrate-native` in the -[operator CLI](../scripts/ftwctl.py) moves an older **direct native systemd** +[operator CLI](https://github.com/srcfl/ftw/blob/af22b488cbda7945d78d30a915fa81b71fe8f1ce/scripts/ftwctl.py) moves an older **direct native systemd** site, one whose unit runs `/opt/ftw/ftw` as the `ftw` user, onto 0.x release slots. The owner's home box was moved with it. It stages the release under a separate root, `/opt/ftw-native` by default, and switches the unit with a -drop-in, so such a box keeps a different layout from a fresh install. Run it +drop-in, so such a box keeps a different layout from a fresh install. Use +the linked script revision on the operator computer; older release checkouts +can reject a valid backup timestamp on older Python. Run it on another computer, through an SSH tunnel to the box's API: ```bash @@ -99,7 +102,8 @@ local terminal and check the running version, health and live readings. Publishing the release and installing it on a box are separate steps. A native beta must run for a week on the home box and at least one other real -site with no open `release-blocker`. Only then can the owner dispatch the same +site with no open `release-blocker`; the owner may shorten that and must say +so in the stable release notes. Only then can the owner dispatch the same workflow for `v0.X.Y` stable, naming the tested `source_beta`. Stable checks the published beta receipt and package hashes and uses the same source commit. Beta and stable contain different embedded version strings, so each @@ -108,8 +112,8 @@ alone is not field validation. On a native site the owner chooses when to update, from More → Version, on the machine, or from their own timer or agent. -[Try the 0.x beta](native-beta.md) is the tester's guide. The installer puts -the `ftw` command on `PATH`: +[Install and update FTW](native-beta.md) is the guide for everyone. The +installer puts the `ftw` command on `PATH`: ```bash ftw status # version, published release, last update, health diff --git a/docs/setup-guide/update-sv.md b/docs/setup-guide/update-sv.md index 4e1c7a46e..0c28f0a51 100644 --- a/docs/setup-guide/update-sv.md +++ b/docs/setup-guide/update-sv.md @@ -1,22 +1,38 @@ # Byt till nya FTW och håll det uppdaterat -[English](../native-beta.md). +**Detta utkast gäller `v0.142.1`; stable är ännu inte publicerad.** FTW +2.x och 3.x får inga fler uppdateringar. [English](../native-beta.md). -**FTW 2.x och 3.x får inga fler uppdateringar. All fortsatt utveckling går -till den nya 0.x-serien. Vill du ha de senaste funktionerna och rättningarna -är det dags att byta nu. Installera inte 3.x-beta, och uppdatera inte en äldre -installation till 3.x som ett steg på vägen.** +## Kort svar -Den nya serien börjar med `v0.131.0-beta.1`. Det lägre versionsnumret är -avsiktligt. Äldre 0.x-versioner, till och med 0.130.x, hör till den gamla -serien. `beta` är kanalen inom en serie; ett kanalbyte flyttar inte en gammal -installation till den nya serien. - -Du kan börja använda nya FTW nu, med nya inställningar och egen datalagring. -Den guidade flytten som tar med gamla inställningar och historik är ännu inte -klar. Behöver du föra över dessa data innan du byter, ta hjälp med just din -installation. Kopiera inte gamla databaser till den nya installationen på egen -hand. +| Du kör | Gör så här | +|---|---| +| Nya FTW 0.x, native | Öppna **More → Version → Update FTW** i webbsidan, eller kör `ftw update` på boxen. Inget mer. | +| Nya FTW 0.x i Docker | Sätt `FTW_VERSION=v0.142.1` i `.env` och kör `docker compose up -d --build`. [Detaljer](#docker-från-det-nya-releasepaketet) | +| Inget FTW än, på 64-bitars Linux eller ett nytt Pi-kort | [Installera](#installera-på-det-nya-kortet-eller-en-tom-linux-maskin) med den stabila taggen. | +| Gamla FTW 2.x, 3.x eller den gamla Pi-imagen | Flytta data och config med en full backup till ett andra SD-kort eller en annan maskin. [Steg](#byt-på-raspberry-pi-med-ett-nytt-sd-kort) | +| Home Assistant-appen | Ingen ny app än. Kör nya FTW på en annan Linux-maskin och koppla Home Assistant via MQTT. [Steg](#välj-en-väg) | +| Windows eller macOS | Stöds inte. Använd en Raspberry Pi eller en annan 64-bitars Linux-maskin. | + +Låt boxen ha ström och tryck inte Restart medan uppdateringen pågår. Vänta tills +uppdateringen är klar. På native återgår `ftw rollback` till föregående +kompatibla release. För Docker, följ [Docker-steget](#docker-från-det-nya-releasepaketet). + +**Ta med inställningar, historik, identitet och mål.** Börja med +[Flytta med din data](../migration.md#svenska). Ta en full backup, importera +på en separat kopia och kontrollera resultatet innan nya FTW tar över. +Behåll gamla installationen och en backup utanför maskinen för återgång. +En befintlig anläggning använder sina sparade inställningar. + +Guiden anger provade datafall och kvarvarande arbete för andra layoutar. +Vid ett importfel: behåll båda kopiorna och lös felet före start. Maskerad +API-config är ingen full export. DuckDB använder +[målversionens konverterare](../history-conversion.md). + +Den nya serien börjar med `v0.131.0-beta.1`. +Äldre 0.x-versioner, till och med 0.130.x, hör till den gamla serien. `beta` i +en gammal installation flyttar den inte till nya serien, och 3.x-beta är inget +steg på vägen. ## Välj en väg @@ -108,10 +124,10 @@ den igen vid omboot. som kan styra samma utrustning. Ordna MQTT om utrustningen behöver det. En broker på gamla kortet följer inte med. 6. **I SSH på nya kortet:** följ [Installera på det nya kortet](#installera-på-det-nya-kortet-eller-en-tom-linux-maskin) - nedan. Öppna sedan `http://:8080/setup` och ställ in anläggningen. - Historik, inlärning och gamla inställningar ligger kvar på gamla kortet; - de följer inte automatiskt med. Använd mål och ready-by-scheman i stället - för gamla kalenderhändelser. + nedan. [Importera din backup](../migration.md#restore-a-full-backup-on-a-new-native-target) + innan nya Core tar över. Kontrollera sparade mål och inställningar. Den + sparade API-porten kan skilja sig från 8080. Använd mål och ready-by-scheman + för tidigare kalenderhändelser. 7. Kontrollera rätt version, friska enheter, färska mätvärden och aktuell plan. Starta sedan om Pi:n när anläggningen kan tåla avbrottet och kontrollera samma saker igen. @@ -127,8 +143,8 @@ du kan identifiera och stoppa gamla Core och updater, hantera deras startregler och bevara MQTT. Välj annars nytt kort eller be om hjälp. En nekad native-installation på gamla kortet är inte ett skäl att fortsätta med Docker. -Den nya Docker-installationen använder en egen katalog och egen data. -Den flyttar inte gamla data. +Den nya Docker-installationen använder en egen katalog. +[Importera en verifierad kopia](../migration.md#svenska) av dina data där. 1. Identifiera det gamla Compose-projektet, dess Core, updater, datakataloger och startregler. Vanliga kataloger är `/opt/ftw`, `~/ftw` och @@ -136,17 +152,25 @@ Den flyttar inte gamla data. 2. Ta en full backup och spara den utanför maskinen. Behåll också Compose-filer, overrides och `.env`, så att du kan återställa rätt gamla version. 3. Kontrollera om projektet även kör Mosquitto eller andra tjänster som - utrustningen behöver. Ordna fortsatt MQTT innan du stoppar hela projektet. -4. Stoppa den gamla installationen med dess riktiga Compose-filer och - projektnamn. Ett vanligt projekt kan stoppas med `docker compose down` - från rätt katalog. Använd inte `-v` och radera inga datakataloger. + utrustningen behöver. Behåll dem igång. Identifiera tjänsterna med + `docker compose config --services`, projektets riktiga filer och projektnamn. +4. Stoppa bara gamla Core och updater. Heter Core-tjänsten `ftw`, kör + `docker compose stop ftw`, sedan + `docker update --restart=no "$(docker compose ps -aq ftw)"`. Gör samma sak + för updater om den finns. Sätt också `restart: "no"` för dessa tjänster i + gamla Compose-filerna. Kontrollera att containrarna är stoppade och har + restart policy `no`. Kör inte `docker compose down` om projektet fortfarande + ger utrustningen MQTT. Använd inte `-v` och radera inga datakataloger. Kontrollera även systemd, timers och egna skript som kan skapa eller starta - den igen. Stäng inte av Docker som helhet. + den igen. Ange brokerns tjänstenamn i senare Compose-kommandon: + `docker compose up -d` för hela gamla projektet startar gamla Core igen, + även med `restart: "no"`. Stäng inte av Docker som helhet. 5. Följ [Docker-guiden för nya FTW](https://github.com/srcfl/ftw/blob/master/docs/native-beta.md#docker). Använd en ny katalog, normalt `~/ftw-local`, och en tom datakatalog. Återanvänd inte den gamla datakatalogen eller en redan befintlig testkatalog utan att först kontrollera vad den innehåller. -6. Ställ in anläggningen och kontrollera version, enheter, mätvärden och plan. +6. Importera din backup med nya Core stoppad. Kontrollera sparade inställningar, + historik, version, enheter, mätvärden och plan innan den tar över. Kontrollera att bara en Core körs och styr utrustningen. Upprepa kontrollen efter en planerad omboot. @@ -166,17 +190,19 @@ kortet, gå tillbaka till kortbytet ovan. `--fresh-host` betyder att ingen FTW-installation eller dess data finns här, även om den är stoppad. 1. Öppna [Releases](https://github.com/srcfl/ftw/releases) på din vanliga dator. - Välj en publicerad **ny 0.x-beta** med Linux-paket och SHA-256-fil för din - maskin. Använd inte `releases/latest`; den pekar på gamla 2.x. + Välj den senaste **stabila** taggen, `v0.142.1` eller senare utan `-beta`. + Välj en beta bara om du provar nya byggen. Releasen ska ha Linux-paket och + SHA-256-fil för din maskin. Använd inte `releases/latest`; den pekar på + gamla 2.x. 2. Kopiera taggen från releasens rubrik. Skriv inte av den för hand. Byt bara - ut `v0.X.Y-beta.N` i blocket nedan mot taggen du kopierade. + ut `v0.X.Y` i blocket nedan mot taggen du kopierade. 3. Klistra in **hela blocket, inklusive `(` och `)`**, i SSH på nya kortet. Det stannar vid första felet och hämtar skriptet till en tillfällig fil. ```bash ( set -eu - tag=v0.X.Y-beta.N + tag=v0.X.Y if [[ ! "$tag" =~ ^v0\.([0-9]+)\.[0-9]+(-beta\.[0-9]+)?$ ]] || (( 10#${BASH_REMATCH[1]} < 131 )); then echo "STOP: copy an exact published new 0.x tag from Releases." >&2 exit 1 @@ -200,13 +226,16 @@ Kör på FTW-maskinen: ```bash ftw status -ftw update --channel beta +ftw update ftw status journalctl -u ftw --since "10 minutes ago" --no-pager ``` -`--channel beta` väljer beta och försöker uppdatera. När rätt kanal redan är -sparad räcker `ftw update` nästa gång. Du kan också välja +`ftw update` följer kanalen du valt med `--channel`. Har du aldrig valt följer +den det boxen kör: en stabil version följer stable, en beta följer beta. Beta +erbjuder också en nyare stable, så en betabox går över till stable när en +sådan kommer. Vill du stanna på beta efter det, kör `ftw update --channel beta`; +vill du lämna beta, kör `ftw update --channel stable`. Du kan också välja **More → Version → Update FTW** i native-webbsidan. Knappen använder samma Core-API och launcher som kommandot. Den visar hämtade byte, klara steg och resultatet efter omstart. Behåll strömmen på. Att stänga eller ladda om sidan @@ -256,6 +285,75 @@ inte den här Docker-installationen. Återgång sker genom att välja föregåen kompatibla `FTW_VERSION` och köra Compose igen. Docker har ingen automatisk återgång till föregående version. +### Backup och restore för nya Docker + +Använd `ftw-backup` från samma release-image. Verktyget verifierar ett +portabelt `.ftwbak`-arkiv och behåller tidigare data vid restore. Stegen gäller +det nya Compose-projektet, inte flytt av en äldre installation. Kör från +`~/ftw-local`. Bara tjänsten `ftw` stoppas. + +Montera en annan disk först. Byt `/media/usb/ftw-backups` till en särskild +backupkatalog där. Containern använder UID 100 och GID 101; blocket skapar +katalogen med dessa ägare och privata rättigheter: + +```bash +( + set -eu + backup_dir=/media/usb/ftw-backups + sudo install -d -o 100 -g 101 -m 0700 "$backup_dir" + docker compose stop ftw + docker compose run --rm --no-deps --entrypoint /app/ftw-backup \ + --volume "$backup_dir:/backup:z" ftw \ + create -state /app/data/state.db -data /app/data -output /backup -progress + docker compose start ftw +) +``` + +Slutlig JSON anger det verifierade arkivets namn under `/backup`. Filen finns +i `backup_dir` på värden. Spara namnet. Vid fel stannar blocket och FTW förblir +stoppad. Lös felet innan du startar igen; en ofärdig fil är ingen backup. + +För restore: ange exakt `.ftwbak`-namn och använd en kompatibel release-image: + +```bash +( + set -eu + backup_dir=/media/usb/ftw-backups + archive=ftw-full-backup-YYYYMMDDTHHMMSS.ftwbak + test -f "$backup_dir/$archive" + docker compose stop ftw + docker compose run --rm --no-deps --entrypoint /app/ftw-backup \ + --volume "$backup_dir:/backup:ro,z" ftw \ + restore -archive "/backup/$archive" -data /app/data -yes + docker compose start ftw +) +``` + +Verktyget verifierar och packar upp innan data ersätts. Ett fel stoppar blocket +innan Core startar. Tidigare data finns kvar i den utskrivna sökvägen +`/app/data/.ftw-pre-restore-…`. Behåll dem tills config, identitet, gammal och ny +historik, mål, färska mätvärden och plan har kontrollerats. Vänta på frisk +status i `docker compose ps` och kör sedan +`docker compose exec ftw ftw status`. + +För att återgå till sparade data: sätt `safety` till den utskrivna sökvägen i +containern. Använd en image som kan läsa dessa data: + +```bash +( + set -eu + safety=/app/data/.ftw-pre-restore-YYYYMMDDTHHMMSS + docker compose stop ftw + docker compose run --rm --no-deps --entrypoint /app/ftw-backup ftw \ + revert -data /app/data -safety "$safety" -yes + docker compose start ftw +) +``` + +Verktyget läser `.ftwbak`, inte äldre råa `.tar.gz`-arkiv. Behåll sådana arkiv +och den matchande gamla installationen för återställning. Att döpa om filen +gör den inte kompatibel. Läs [full backup och restore](../backup-and-restore.md). + ## För agenter som hjälper till - Välj en väg. Native och Docker är alternativ, inte steg efter varandra. diff --git a/scripts/test_install_docs.py b/scripts/test_install_docs.py index 7481266a8..eca064588 100644 --- a/scripts/test_install_docs.py +++ b/scripts/test_install_docs.py @@ -14,7 +14,14 @@ def recipe(path, marker): blocks = re.findall(r'```bash\n(.*?)\n```', path.read_text(), re.S) - matches = [block for block in blocks if marker in block and 'tag=v0.X.Y-beta.N' in block] + matches = [block for block in blocks if marker in block and 'tag=v0.X.Y\n' in block] + assert len(matches) == 1, (path, marker, len(matches)) + return matches[0] + + +def recovery_recipe(path, marker): + blocks = re.findall(r'```bash\n(.*?)\n```', path.read_text(), re.S) + matches = [block for block in blocks if marker in block and 'docker compose run' in block] assert len(matches) == 1, (path, marker, len(matches)) return matches[0] @@ -49,7 +56,7 @@ def stub(self, name, body): path.chmod(0o755) def run_recipe(self, marker, failure='', tag='v0.138.2-beta.1'): - block = recipe(EN, marker).replace('v0.X.Y-beta.N', tag) + block = recipe(EN, marker).replace('tag=v0.X.Y\n', f'tag={tag}\n') # Change only the destination, never the user's HOME or real Docker state. if marker == 'compose up': self.assertEqual(block.count('"$HOME/ftw-local"'), 1) @@ -60,12 +67,48 @@ def run_recipe(self, marker, failure='', tag='v0.138.2-beta.1'): calls = self.log.read_text() if self.log.exists() else '' return result, calls + def test_migration_restore_stops_on_failure_and_leaves_core_off(self): + blocks = re.findall(r'```bash\n(.*?)\n```', + (ROOT / 'docs/migration.md').read_text(), re.S) + block = next(b for b in blocks if 'helper=/opt/ftw/releases/' in b) + release = self.work / 'releases' / 'v0.140.11' + release.mkdir(parents=True) + helper = release / 'ftw-backup' + helper.write_text('#!/bin/bash\nset -eu\necho "helper $*" >> "$FTW_TEST_LOG"\n' + '[ "$FTW_TEST_FAIL" != "$1" ]\n') + helper.chmod(0o755) + self.stub('systemctl', 'echo "systemctl $*" >> "$FTW_TEST_LOG"') + self.stub('chown', 'echo "chown $*" >> "$FTW_TEST_LOG"') + block = block.replace('tag=v0.X.Y', 'tag=v0.140.11') + block = block.replace('/opt/ftw/releases', str(self.work / 'releases')) + block = block.replace('/var/lib/ftw', str(self.work / 'target')) + for failure in ('verify', 'restore', ''): + with self.subTest(failure=failure): + self.log.write_text('') + self.env['FTW_TEST_FAIL'] = failure + result = subprocess.run(['bash', '-c', block], env=self.env, + cwd=self.work, capture_output=True, text=True) + calls = self.log.read_text() + self.assertIn('systemctl disable --now ftw', calls) + self.assertNotIn('systemctl start', calls) + self.assertNotIn('systemctl enable', calls) + if failure: + self.assertNotEqual(result.returncode, 0) + self.assertNotIn('chown', calls) + if failure == 'verify': + self.assertNotIn('helper restore', calls) + else: + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn('helper verify', calls) + self.assertIn('helper restore', calls) + self.assertIn('chown -R ftw:ftw', calls) + def test_translations_use_the_same_native_commands(self): self.assertEqual(recipe(EN, '--fresh-host'), recipe(SV, '--fresh-host')) def test_rejects_placeholder_old_line_and_mistyped_tag_before_work(self): for marker in ('--fresh-host', 'compose up'): - for tag in ('v0.X.Y-beta.N', 'v3.8.0-beta.1', 'v0.130.0', 'v0.0.138-beta.1'): + for tag in ('v0.X.Y', 'v0.X.Y-beta.N', 'v3.8.0-beta.1', 'v0.130.0', 'v0.0.138-beta.1'): with self.subTest(marker=marker, tag=tag): result, calls = self.run_recipe(marker, tag=tag) self.assertNotEqual(result.returncode, 0) @@ -82,11 +125,15 @@ def test_native_download_failure_does_not_run_stale_installer(self): self.assertFalse(Path(download).exists()) def test_native_success_runs_download_and_removes_temp_file(self): - result, calls = self.run_recipe('--fresh-host') - self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn('INSTALL\n', calls) - download = shlex.split(calls.splitlines()[0])[-1] - self.assertFalse(Path(download).exists()) + for tag in ('v0.140.11', 'v0.138.2-beta.1'): + with self.subTest(tag=tag): + self.log.write_text('') + result, calls = self.run_recipe('--fresh-host', tag=tag) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn('INSTALL\n', calls) + self.assertIn(f'/srcfl/ftw/{tag}/scripts/install.sh', calls) + download = shlex.split(calls.splitlines()[0])[-1] + self.assertFalse(Path(download).exists()) def test_failed_docker_download_leaves_no_project_and_allows_retry(self): for failure in ('download', 'dockerfile'): @@ -133,6 +180,70 @@ def test_docker_success_writes_ignore_before_build(self): self.assertEqual(result.returncode, 0, result.stderr) self.assertIn('docker compose up -d --build', calls) + def run_recovery(self, marker, failure='', archive_exists=True): + backup_dir = self.work / 'usb-backups' + backup_dir.mkdir(exist_ok=True) + archive = backup_dir / 'ftw-full-backup-YYYYMMDDTHHMMSS.ftwbak' + if archive_exists: + archive.write_text('test archive; helper is stubbed\n') + self.stub('install', 'echo "install $*" >> "$FTW_TEST_LOG"') + self.stub('docker', '''echo "docker $*" >> "$FTW_TEST_LOG" +if [ "$FTW_TEST_FAIL" = stop ] && [ "$1 ${2:-}" = 'compose stop' ]; then exit 1; fi +case " $* " in + *' create '*) [ "$FTW_TEST_FAIL" != create ] || exit 1 ;; + *' restore '*) [ "$FTW_TEST_FAIL" != restore ] || exit 1 ;; + *' revert '*) [ "$FTW_TEST_FAIL" != revert ] || exit 1 ;; +esac''') + block = recovery_recipe(EN, marker).replace( + 'backup_dir=/media/usb/ftw-backups', 'backup_dir=' + shlex.quote(str(backup_dir))) + self.env['FTW_TEST_FAIL'] = failure + result = subprocess.run(['bash', '-c', block], cwd=self.work, env=self.env, + capture_output=True, text=True) + calls = self.log.read_text() if self.log.exists() else '' + return result, calls + + def test_recovery_translations_use_the_same_commands(self): + for marker in ('create -state /app/data/state.db', 'restore -archive', 'revert -data'): + self.assertEqual(recovery_recipe(EN, marker), recovery_recipe(SV, marker)) + + def test_recovery_helper_failure_never_restarts_core(self): + for action, marker in (('create', 'create -state /app/data/state.db'), + ('restore', 'restore -archive'), ('revert', 'revert -data')): + with self.subTest(action=action): + self.log.write_text('') + result, calls = self.run_recovery(marker, action) + self.assertNotEqual(result.returncode, 0) + self.assertIn('docker compose stop ftw\n', calls) + self.assertIn('--entrypoint /app/ftw-backup', calls) + self.assertNotIn('docker compose start', calls) + + def test_failed_stop_does_not_run_recovery_or_start_core(self): + for marker in ('create -state /app/data/state.db', 'restore -archive', 'revert -data'): + with self.subTest(marker=marker): + self.log.write_text('') + result, calls = self.run_recovery(marker, 'stop') + self.assertNotEqual(result.returncode, 0) + self.assertNotIn('docker compose run', calls) + self.assertNotIn('docker compose start', calls) + + def test_missing_restore_archive_stops_before_touching_core(self): + result, calls = self.run_recovery('restore -archive', archive_exists=False) + self.assertNotEqual(result.returncode, 0) + self.assertEqual(calls, '') + + def test_recovery_success_starts_only_ftw_after_helper(self): + for marker in ('create -state /app/data/state.db', 'restore -archive', 'revert -data'): + with self.subTest(marker=marker): + self.log.write_text('') + result, calls = self.run_recovery(marker) + self.assertEqual(result.returncode, 0, result.stderr) + docker_calls = [call for call in calls.splitlines() if call.startswith('docker ')] + self.assertEqual(docker_calls[0], 'docker compose stop ftw') + self.assertIn('compose run --rm --no-deps --entrypoint /app/ftw-backup', docker_calls[1]) + self.assertEqual(docker_calls[2], 'docker compose start ftw') + if marker == 'restore -archive': + self.assertIn(':/backup:ro,z', docker_calls[1]) + if __name__ == '__main__': unittest.main()