Skip to content

create can FREEZE a table while mutations are in progress, producing backups that fail restore #1455

Description

@minguyen9988

Problem

create freezes tables without checking for in-progress mutations. AddTableToLocalBackup calls ch.FreezeTable directly (pkg/backup/create.go:986) — there is no look at system.mutations.

If a heavy mutation (ALTER TABLE ... DELETE/UPDATE, MATERIALIZE ...) is running while the table is frozen, the frozen shadow can capture a part set that is mid-replacement. The resulting backup can reference parts whose data files were replaced/removed between FREEZE and upload, producing backups that upload "successfully" but fail on restore with errors like:

marks file '...' doesn't exist
... object doesn't exist ...

This is a silent correctness issue: the backup looks fine until you need it.

Proposal

  • Add WaitMutationsToComplete(ctx, database, table, timeoutSeconds) to pkg/clickhouse that polls system.mutations WHERE is_done = 0 for the table and returns an error if mutations are still in progress after the timeout (fail the backup loudly rather than produce a corrupt one).
  • Add a wait_mutations_before_freeze_timeout option to the clickhouse config section, default 0 = disabled, preserving current behavior exactly (opt-in).
  • Call it immediately before FreezeTable in AddTableToLocalBackup.

We run this with a 1800s timeout in a production fork backing up multi-TiB clusters, where it eliminated a class of restore failures traced to freeze-during-mutation.

I have a PR ready to submit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions