RC-19 test report: upgrade from 1.5.10.2518 fails on the first pass (schema 290), and a rerun deletes the 1.5 web backup #2000

Follow-up to #1945, same KVM lab, rebuilt from scratch on 2026-10-28. Two parts: a fresh RC-19 install with the new FOS init, and an upgrade of a fresh 1.5 stable server.

Setup

• Host: KVM, --machine pc, isolated network per server, FOG DHCP on Kea. Server VMs: Debian 12 genericcloud, 4 vCPU, 4 GB.
• RC: rc-1.6.0 at 494b9ca (1.6.0-RC-19), installfog.sh -y --channel rc.
• 1.5: stable at c97e205 (1.5.10.2518), installfog.sh -y. Then rc-1.6.0 at 4983c25 (RC-19 plus the translation sync), installfog.sh -y --channel rc.
• FOS: Latest, EXP_20261027-181357 (initramfs).

Fresh RC-19 install: all green

One capture failed with e2fsck failed to check /dev/vda1 (shrinkPartition) … recovering journal, after I hard-reset the source. That is note 5 of #1945 (FOS side). A clean reboot of the same kind of source captured fine.

Upgrade 1.5.10.2518 → RC-19

Before the upgrade: one image captured on 1.5, mysqldump and .fogsettings copied by hand.

1. The first pass stops at Updating Database

 * Backing up database.........................................Done
 * Updating Database...........................................Failed!

fog_schema_update_error.log:

Update ID: 290
Database Error: ... SQLSTATE[42S02]: Base table or view not found: 1146 Table 'fog.fileDeleteQueue' doesn't exist ...
SQL: ALTER TABLE `fileDeleteQueue` ADD `fdqPathType` VARCHAR(255) NOT NULL

What I see:

• A fresh 1.5.10.2518 arrives with schemaVersion.vValue = 288 and no fileDeleteQueue table. The comments in commons/schema.php:4822 and src/Db/SchemaReconciler.php:32 expect a fully patched 1.5.10 at 277.
• So the updater starts at element 288. 1.6's steps // 287 (CREATE TABLE fileDeleteQueue) and // 288 are skipped. // 289 runs, then // 290 ALTERs a table that does not exist.
• 1146 is not in $skiperrs (src/Pages/SchemaUpdaterPage.php:393-399), so the update breaks there.
• The reconcile after the loop (SchemaUpdaterPage.php:491) still runs and creates fileDeleteQueue: create_time is 16:54:32, and the error log is written at 16:54:34. But the error is already recorded. The installer prints Failed! and vValue stays at 289.

A second installfog.sh -y --channel rc completes: Updating Database...OK, Setup complete, vValue 445. The image captured on 1.5 then deploys from 1.6 to a blank disk, and the clone carries the marker.

So the server recovers, but only after a rerun that the admin has no reason to try first, because the release notes ask them to treat the upgrade as one-way.

A fix for this part follows in a pull request that references this issue: on 1146, run the reconcile at the point of failure and retry the statement once. With it, the same fresh 1.5.10.2518 upgrades in one pass (Setup complete, vValue 445, no fog_schema_update_error.log), and the 1.5 image deploys from the upgraded server.

One side observation from comparing the upgraded database with a fresh RC-19 one: the only difference is users.uName and users.uCreateBy, which stay varchar(40) NOT NULL from 1.5 where a fresh install has varchar(255) NULL. The two-pass upgrade without the fix ends up the same way. That fits the reconciler never retyping an existing column, so I am only noting it.

2. The rerun deletes the only 1.5 web-tree backup

The web tree is backed up to fog_web_${version}.BACKUP, and ${version} is the version being installed. lib/common/functions.sh:13005-13009 deletes that directory first, then copies the current tree into it. On the rerun above, the current tree is already 1.6, so the 1.5 copy is replaced.

After the second pass:

$ ./revertfog.sh --list
 * Database dumps under /home/fogDBbackups, newest first:
   rejected fog_sql_1.6.0-RC-19_20261028_165900.sql  -- it contains the 1.6-only table `userAuths` -- this is a 1.6 dump
   USABLE   fog_sql_1.6.0-RC-19_20261028_165426.sql  (schema count 288)

 * Web tree backups under /home:
   rejected fog_web_1.6.0-RC-19.BACKUP  -- not a 1.5 tree (no lib/fog/system.class.php, or it carries src/)

The database dumps carry a timestamp, so the 1.5 dump survives. The web tree has no 1.5 copy left. After the first pass alone, --list and --dry-run accepted both backups and described the revert correctly.

revertfog.sh already knows this case. Its dry run says "a second install run replaces the 1.5 copy with the 1.6 one", skips the web tree, and asks for a 1.5 reinstall. A real revert then worked: schema back to 288 with the image and host, and the 1.5 installer rebuilt the tree. So nothing is unrecoverable. The gap is that issue 1 makes that second run the normal path, and the release notes promise names that would have kept the copy.

The release notes say both backups are "named after the version being upgraded from, not the one you're upgrading to" (docs/release/1.6.0-release-notes.md:120). Both are actually named after RC-19.

3. revertfog.sh is not in a 1.5 checkout

The release notes say to run bin/revertfog.sh "from the bin/ directory of a 1.5 checkout" (line 128). It is in working-1.6 and rc-1.6.0, but not in stable (c97e205) or dev-branch (17973cd). Run from the 1.6 checkout, --list and --dry-run work as documented.

Note: a 1.5 install today downloads the experimental FOS

downloadfiles() in 1.5 picks a pinned FOS release only when the version matches ^[0-9]\.[0-9]\.[0-9]+$ (lib/common/functions.sh:5613 on stable). 1.5.10.2518 has four parts, so it takes releases/latest, which is EXP_20261027-181357. That release is not marked pre-release. The 1.5 server got the same init.xz as RC-19 (cpio, same size), and capture and deploy on 1.5 worked with it. I am mentioning it only because the next FOS "Latest" reaches every 1.5 install as well.

Evidence kept

The failed-pass logs (upgrade.log, fog_error_1.6.0-RC-19.log, fog_schema_update_error.log) and a dump taken right after the failure are on the lab server. I can attach any of them, or rerun the upgrade from a fresh 1.5.10.2518 to test another change.

🤖 Generated with Claude Code