What does Snapshots, Backups, and Incremental Snapshots mean in CloudStack

[daviftorres]

Dear CloudStack fellows,

Recently, Lucian mentioned the new 4.22 feature that "enables incremental volume snapshots" (https://www.youtube.com/watch?v=ZZi3CpX6u2I&t=680s).

I would like to open a discussion so we can collaboratively clarify the different types of snapshots and backups that CloudStack currently supports, and how they actually work.

First, there is an important conceptual distinction between snapshots and backups:

• Snapshots are primarily point-in-time rollback mechanisms. They rely on layers of changes, and usually do not involve copying the full dataset.
• Backups are copies of data intended for restore, retention, and disaster recovery.

From there, things become more nuanced:

• A full backup is a complete bit-for-bit copy of the original data.
• An incremental backup contains only the changes since the last backup, which helps save space and time.
  • Incremental backups often rely on snapshot mechanisms to track changes.
  • A synthetic full backup can later be built by applying all incremental layers on top of the initial full backup.

Where I am still unclear is how these concepts map into the Apache CloudStack ecosystem:

• Snapshots in CloudStack are stored on Secondary Storage alongside templates and images, which makes them feel closer to “copies” than pure rollback layers.
• NAS Backup appears similar to snapshots, but sends a volume copy to a separate repository.

For the sake of simplicity, I am focusing first on KVM + NFS but it is not limited to.

These distinctions matter, especially when SLAs, compliance requirements, and operational policies treat snapshots and backups very differently.

I encourage others to share their understanding and explanations. My goal is to consolidate this discussion and translate it into clearer documentation afterward.

Regards,

Davi Torres

[daviftorres]

We upgraded our Dev environment to 4.22, and we are testing instance and volume snapshots to visualize the incremental portion.

I can confirm that instance snapshots sometimes use QCOW2's internal snapshots.

Table vm_snapshots

Note: I had to include memory because the instance snapshots could (in that occasion) only be taken when the instance is in the Running state. Please correct me if I am wrong. This instance was deployed from an ISO.

Then, for no apparent reason on another instance that was deployed from a template to a unlinked sparse provisioning, it "decided" to take external snapshots.

It can also be observed on the XML Dump of the instance.

If you know what logic is used to define if the snapshot will be used for snapshots (internal or external), please comment below.

[daviftorres]

You can most certainly create instance snapshots while the instance is in a shutdown state. You can also create instance snapshot without memory.

Dear @chunkyen , than you for your reply.

What you said is generally correct, but in practice it is not always true. There are cases where creating a snapshot without memory is not allowed, and the UI error message does not clearly explain why.

For example, in the instance below, snapshot creation only works if Snapshot memory is enabled:

However, on other instances, snapshot creation works fine without memory, for example:

Some additional context:

The instance that only allows snapshots with memory has qemu-guest-agent installed and was deployed from an ISO.

The instance that allows snapshots without memory was deployed from an out-of-the-box template and does not have qemu-guest-agent installed.

So while snapshots without memory are supported in general, there seems to be some internal logic or constraint that blocks it in certain cases, and the current error message does not make this clear.

Do you know what specifically causes this behavior?

[daviftorres]

Note: I had to include memory because the instance snapshots could (in that occasion) only be taken when the instance is in the Running state. Please correct me if I am wrong. This instance was deployed from an ISO.

Answering my own question, the fact that limited the Instance Snapshot without adding the memory was the missing setting rootDiskController = osdefault:

I managed to "fix" the ISO template by adding those settings in the table vm_template_details.

Besides the fix mentioned above, there is a limitation around mixing snapshot types. If a snapshot that includes memory already exists, the system does not allow creating a snapshot without memory. The reverse is also true.

My understanding is that disk-only snapshots use QCOW2 metadata, while disk-plus-memory snapshots use a chain-based approach. Because these rely on different underlying mechanisms, mixing them is not allowed, likely to prevent data corruption or loss. Like described in PR #11039

[JoaoJandre]

@daviftorres It is actually the other way around. Disk-only VM snapshots are chain-based, that is, it only uses external snapshots. While Disk-and-memory snapshots are always internal. But they indeed cannot be mixed because of the different approaches.

[daviftorres]

Thanks @JoaoJandre, I’m not sure why I worded it that way, but I had the same understanding.

I also thought it might be possible (outside of CloudStack) to create internal snapshots without memory, with libvirt using QCOW2 metadata. Based on what you say, it is not possible. Correct?

[JoaoJandre]

As far as I know, up until Libvirt version 9 it is not possible:

❯ virsh snapshot-create-as --domain 2 --name test --disk-only --live --diskspec vda,snapshot=internal
error: Operation not supported: live snapshot creation is supported only during full system snapshots

❯ virsh snapshot-create-as --domain 2 --name test --disk-only --diskspec vda,snapshot=internal
error: unsupported configuration: active qemu domains require external disk snapshots; disk vda requested internal

❯ virsh --version
9.0.0

But from my brief research, it could be available in the future (see https://lists.libvirt.org/archives/list/users@lists.libvirt.org/thread/3RUFKPWNGOQNYY6QWLH34LA4GEXSC7M6/)

[daviftorres]

When working with volume snapshots, the behavior is very different from what one might expect.

Table snapshot_store_ref

Each consecutive snapshot creates a new file in Secondary Storage, and every one of these files has exactly the same size and the same hash.

Therefore, I conclude that volume snapshots are effectively standalone full backups that are sent to Secondary Storage each time.

Even the status uses the work Backup.

[gp-santos]

Hi! Yes, volume snapshots are backups with an unfortunate name.

Just pointing out you can also enable the configuration kvm.incremental.snapshot, which creates incremental chains of snapshot.delta.max size. (This would be the "incremental volume snapshots" mentioned in the first message)

[daviftorres]

Thank you @gpordeus . I do have it enabled and I am trying to understand how they work because labels and descriptions are not always clear. For example, in some occasions it does internal (leveraging QCOW2's metadata) and others it does external (chaining multiple QCOW2 files).

[gp-santos]

When using volume snapshots, there are internal pointers in the original volume (in the primary storage) that are used only to create the deltas, which are then backed up to the secondary storage. Would this be what you mean? There's more info about the workflow in #8907.

[daviftorres]

Moreover, in the snapshots table (see below), I found some references to different snapshot types (from 0 to 7, I guess).

And, I came across this documentation page that mentions snapshot types: https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots

However, the information there is not detailed enough for me to fully understand the differences, so I would appreciate a more thorough explanation.

[slavkap]

@daviftorres,
MANUAL means that those are Volume snapshots initiated by the user
GROUP - those are instance snapshots without the memory. But there were some changes in the latest CS versions for NFS storage and such DB records could be missing.

[daviftorres]

Volume Snapshots (aka Instance Volume Snapshot)

When attached to a running instance, the button Take Snapshot of the volume is not available by default.

If the global setting Kvm snapshot enabled (kvm.snapshot.enabled) is set to true...

…the Take Snapshot button becomes available even while the VM is running.

I understand that taking a snapshot of a volume attached to a running VM is potentially can compromise filesystem and application-level data consistency. This issue is solved by capturing the memory with the disk into the snapshot.

There is a confusion caused but the name Instance Volume Snapshot because it is in fact a Volume Snapshot technically is a Backup. In summary, I recommend changing the name to just Backup or Volume Backup.

[JoaoJandre]

As others explained, when using KVM, volume snapshots are, by default, volume backups. I'm +1 on changing the nomenclature but sadly I'm not sure everyone would agree.

[daviftorres]

I agree with this naming change because it causes a lot of confusion for people who use and manage ACS.

Half the time, I feel like I need to give a whole lecture just to explain that the names don’t actually mean what they say… and people end up falling asleep 🥱. Then, I will try again next time 😄.

[daviftorres]

Instance Snapshot

Volumes in CloudStack can be linked or unlinked, and it’s important to understand the difference.

Linked Volumes

• These volumes are externally linked to the template image.
• Essentially, the volume acts as an overlay on top of the template, similar to an external snapshot backed by the template.

Example: Linked Volume deployed from Thin Provisioning offer.

Here, the instance’s volume file is layered on the template image.

Example: Linked Volume deployed from Thin Provisioning offer and 2 Instance Snapshots.

Unlinked Volumes

These volumes are standalone files, independent of any template.

Example: Unlinked Volume deployed from ISO

Example: Unlinked Volume deployed from Sparse Provisioning.

Example: Unlinked Volumes with Internal Snapshot

Note: Mixing internal and external snapshots is not recommended.

This is where the ROOT disk provisioning type is typically configured, at the Compute Offering:

And also at the Add Disk Offering creation:

Provisioning types explained

• Thin Provisioning - Linked Clone
  • Instances are created as linked clones.
  • The template cannot be removed while instances depend on it.
  • If the template becomes corrupted or inaccessible, ALL dependent instances will also fail.
• Sparse Provisioning - Full Copy (Lazy Thick)
  • A full copy of the original image is created (not linked).
  • Disk space is allocated, but zero blocks are written on demand.
  • Faster to create than Fat provisioning.
• Fat Provisioning - Full Copy (Eager Thick)
  • A full copy of the original image is created (not linked).
  • All disk blocks are fully zeroed at creation time.
  • Slower to create, but disk is fully initialized upfront.

[ingox]

@daviftorres there are pro and con for the different snapshot and backup options. Find them here:

Instance snapshot:

• is stored on the primary storage
• is no backup; just to freeze a certain state if you do changes
• is using qcow2 functionality
• does snapshot of the entire VM including all disks
• can include memory as well
• can have differential snapshots
  Volume snapshot:
• is copied to the secondary storage
• is individual to each volume
• can be differential
• you won't get a consistent snapshot across all volumes of a VM for the same point in time
• can be copied to other zones automatically
• can be restored in other zones without the need of template now
  NAS backup:
• does backup the entire VM to an external location
• can be restored to other zones with access to this backup storage
• does full backup only at the moment - differential backup is hopefully to come soon

I hopefully have covered most of the important criteria to be considered.

[davift]

Thank you @ingox , for your reply. This is pretty much in the lines that I want to consolidate knowledge.

[slavkap]

Just to add that for the block storage providers there could be some differences. From what I know
VM snapshots:
Linstor/StorPool/PowerFlex - don't include the memory and the consistency of the snapshots is handled by the storage plugins (there is no freezing of the VMs during instance snapshots).
Ceph - don't include the memory. The consistency is handled while the VM is frozen
Volume snapshots:
Linstor - from what I know it could keep the snapshot only on primary or only on secondary storage via global setting
StorPool - could be kept only on primary storage or on both primary/secondary storage. Are incremental
Ceph - can keep the snapshot on primary storage only or on secondary storage

For the rest of the storage plugins I don't have much experience

[bilalinamdar]

Can this be used for Linstor primary storage ?

[ingox]

@bilalinamdar What exactly to do you want to do on the Linstor storage?

[bilalinamdar]

@bilalinamdar What exactly to do you want to do on the Linstor storage?

I mean to ask this backup incremental feature works with NFS but will it work with linstor drbd?

[daviftorres]

I can't say much about Linstor/StorPool but these global settings appear on 4.22 when compared with 4.20.

storpool.snapshot.recovery.from.remote.check Storpool snapshot recovery from remote check
Minimal interval (in seconds) to check and recover StorPool snapshot from remote.
Default 300

use.storage.replication Use storage replication
For snapshot copy to another primary storage in a different zone. Supports only StorPool storage for now.
Default false

If you find answers, please share here.

[slavkap]

@bilalinamdar, @daviftorres, StorPool does not support backups through CloudStack for now (we have internal methods for backups). I think and Linstor does not support backups either (probably @rp- can confirm)
@daviftorres, both setting are used in the feature you mentioned and for now it is working only for StorPool

[rp-]

We have here work going on to support nas backup: #12218

[ingox]

@bilalinamdar see this article from @rajujith:

https://www.shapeblue.com/nas-backup-and-recovery-plugin/

The NAS Backup and Recovery Plugin currently supports NFS (Network File System) and operates as a straightforward B&R recovery provider, making it both storage and vendor-agnostic. This enables administrators to do efficient backups of running instances, regardless of their source storage. It also can be used to create backups of stopped instances stored on both NFS and local primary storage. However, please note that restore operations are currently supported only for NFS and local primary storage.

[chunkyen]

Hi, the global settings seems to imply that B&R framework will also work with Dell Networker. Do you know if this is the case?

[TadiosAbebe]

@ingox , actually starting from 4.22, restore operations for ceph is also supported. #11684

[DRDAVIDBANNER]

From an operator/MSP perspective, we treat snapshots and backups as separate things even where the underlying CloudStack terminology overlaps.

We generally think of a snapshot as a platform-level point-in-time mechanism, whereas a backup service needs to cover the wider operational requirements: independent retention, policy management, recovery workflows, monitoring, reporting and ideally recovery into another location for DR.

At Quadris, we use Sendense for backup and DRaaS across our CloudStack environment rather than relying solely on CloudStack snapshots. One of our requirements was support for block-storage environments as well as CloudStack-aware recovery.

Full disclosure: I am involved with Sendense, but we have also been running it operationally in the Quadris cloud for around six months. From our perspective, the important question is less about whether CloudStack calls an object a “snapshot” or “backup,” and more about whether the overall design meets the required RPO, RTO, retention and failure-domain requirements.

[davift]

Thank you for your collaboration on this post, @DRDAVIDBANNER.

This discussion started when I began testing Incremental Instance Snapshots and realized that many variables were interacting and affecting the results.

I tried to map out which features should be prioritized and which ones should be avoided because they conflict with others. For example, the rootdiskcontroller setting and the combination of internal and external snapshots are just a couple of cases.

I will share my recommendations soon based on the points we have discussed so far.

[RyanP-Testing]

Hi Davi,

This is a very useful discussion, especially because the terms “snapshot” and “backup” can easily get mixed together when looking at CloudStack from an operational or SLA perspective.

From a practical CloudStack perspective, I think it helps to separate the concepts into three areas:

1. Volume Snapshot

A CloudStack volume snapshot represents the state of a volume at a particular point in time. It is primarily useful for rollback, cloning, recovery, or creating a new volume from a previous state.

However, the actual storage behavior is dependent on the underlying storage provider. A CloudStack snapshot should therefore not automatically be interpreted as a traditional array-level snapshot or as a standalone disaster-recovery backup.

2. Backup

A backup is better viewed as a copy intended for longer-term retention and recovery, preferably on storage that is independent from the primary workload/storage environment.

This distinction becomes important operationally. If the underlying storage is lost or the storage system itself becomes unavailable, a snapshot located on that same storage infrastructure may not provide the same protection as an independent backup.

3. Incremental Volume Snapshots in CloudStack 4.22

The new incremental snapshot capability makes the distinction even more interesting.

Instead of repeatedly transferring the entire volume for every snapshot, subsequent snapshots can represent only the data that has changed since the previous snapshot. This can significantly reduce the amount of data transferred and stored.

The important point, however, is that “incremental” describes how the changed data is captured/stored; it does not automatically mean that the resulting snapshot should be treated as a complete independent backup.

For example, conceptually:

Full/Base Snapshot
       |
       +---- Changes 1
       |
       +---- Changes 2
       |
       +---- Changes 3

The complete point-in-time state may depend on the base data plus the required incremental changes.

For KVM + NFS, I would therefore avoid describing CloudStack snapshots simply as “full backups” or “incremental backups” without considering the underlying implementation and storage provider.

The other important distinction is retention and failure domain.

A snapshot can be extremely useful for:

• Quick rollback
• Recovering from an application or configuration change
• Creating a new volume
• Short-term recovery

A backup is generally intended for:

• Long-term retention
• Disaster recovery
• Protection against storage failure
• Compliance requirements
• Recovery when the primary storage is unavailable

So, from an operational perspective, I would use the following rule of thumb:

Snapshot = convenient point-in-time recovery

Backup = independent recovery copy

Incremental snapshot = optimized way of maintaining point-in-time states by tracking changed data

There is certainly some overlap between these concepts, but they should not be treated as interchangeable when designing an SLA or disaster-recovery strategy.

It would also be useful to document the behavior separately for each storage backend, because the implementation details can differ significantly between KVM/NFS, Ceph, local storage, and other storage providers.

That would make the documentation much clearer and prevent administrators from assuming that a CloudStack snapshot automatically provides the same protection as an independent backup.

Regards,

Ryan P.
**DevOps Engineer & *
AccuWeb.Cloud