Problem
create freezes tables without checking for in-progress mutations. AddTableToLocalBackup calls ch.FreezeTable directly (pkg/backup/create.go:986) — there is no look at system.mutations.
If a heavy mutation (ALTER TABLE ... DELETE/UPDATE, MATERIALIZE ...) is running while the table is frozen, the frozen shadow can capture a part set that is mid-replacement. The resulting backup can reference parts whose data files were replaced/removed between FREEZE and upload, producing backups that upload "successfully" but fail on restore with errors like:
marks file '...' doesn't exist
... object doesn't exist ...
This is a silent correctness issue: the backup looks fine until you need it.
Proposal
- Add
WaitMutationsToComplete(ctx, database, table, timeoutSeconds) to pkg/clickhouse that polls system.mutations WHERE is_done = 0 for the table and returns an error if mutations are still in progress after the timeout (fail the backup loudly rather than produce a corrupt one).
- Add a
wait_mutations_before_freeze_timeout option to the clickhouse config section, default 0 = disabled, preserving current behavior exactly (opt-in).
- Call it immediately before
FreezeTable in AddTableToLocalBackup.
We run this with a 1800s timeout in a production fork backing up multi-TiB clusters, where it eliminated a class of restore failures traced to freeze-during-mutation.
I have a PR ready to submit.
Problem
createfreezes tables without checking for in-progress mutations.AddTableToLocalBackupcallsch.FreezeTabledirectly (pkg/backup/create.go:986) — there is no look atsystem.mutations.If a heavy mutation (
ALTER TABLE ... DELETE/UPDATE,MATERIALIZE ...) is running while the table is frozen, the frozen shadow can capture a part set that is mid-replacement. The resulting backup can reference parts whose data files were replaced/removed between FREEZE and upload, producing backups that upload "successfully" but fail on restore with errors like:This is a silent correctness issue: the backup looks fine until you need it.
Proposal
WaitMutationsToComplete(ctx, database, table, timeoutSeconds)topkg/clickhousethat pollssystem.mutations WHERE is_done = 0for the table and returns an error if mutations are still in progress after the timeout (fail the backup loudly rather than produce a corrupt one).wait_mutations_before_freeze_timeoutoption to theclickhouseconfig section, default0= disabled, preserving current behavior exactly (opt-in).FreezeTableinAddTableToLocalBackup.We run this with a 1800s timeout in a production fork backing up multi-TiB clusters, where it eliminated a class of restore failures traced to freeze-during-mutation.
I have a PR ready to submit.