Proxmox VM backup vs. wbadmin vs. [...]

[Toxik]

Hi,
until now I've used wbadmin to backup Windows machines (no matter if VM or on real hardware).
But now I've setup a Proxmox test machine and I've found the integrated VM backup feature which looks very interesting to me. Did a short test with a Linux guest, which worked flawlessly.

But I have some questions:

• If used together with the qemu-guest-agent on Windows, does the VM backup make use of VSS before creating a backup?
• How likeley is it to get a corrupted backup?
• Do you prefer the VM backups over e.g. wbadmin or other backup tools? Why?

Thanks!

 

[LnxBil]

If used together with the qemu-guest-agent on Windows, does the VM backup make use of VSS before creating a backup?

Yes, according to this https://pve.proxmox.com/wiki/Qemu-guest-agent

How likeley is it to get a corrupted backup?

Very unlikely, but you still make a backup of a live system. On Windows, you have VSS in which programs can tap in, on Linux you can just run any script just before the backup is done, so flushing file system caches, doing database logswitches ... everything is possible.

Do you prefer the VM backups over e.g. wbadmin or other backup tools? Why?

Time-to-Recovery is the best for the integrated solution and the backups can be restored in any PVE environment without any setup. Just copy the vzdump file over and restore.

 

[mbaldini]

1. Yes
2. I never got a corrupted vzdump backup
3. I usually make both vzdump backups and Veeam Agent backup, I use Veeam to restore individual files and vzdump to restore full VM

 

[Toxik]

Yes, according to this https://pve.proxmox.com/wiki/Qemu-guest-agent

Yes, I've already seen that it is possible. Just was not sure if this applies to backups.

Very unlikely, but you still make a backup of a live system. On Windows, you have VSS in which programs can tap in, on Linux you can just run any script just before the backup is done, so flushing file system caches, doing database logswitches ... everything is possible.

But is there at least any kind of snapshot of the VM taken before backup starts? Otherwise I cannot imagine how this should work.

Time-to-Recovery is the best for the integrated solution and the backups can be restored in any PVE environment without any setup. Just copy the vzdump file over and restore.

That's actually the reason why I think the integrated backup feature is so interesting. Even if my Proxmox server hardware goes up in smoke... Just import the file in another preinstalled Proxmox server and at least the machines are reachable again.

I've thought about the following:

• create a pool A for VM storage
• create a pool B on some other disks for local backup storage
• run backup from pool A to pool B for better backup performance.
  
• after backup finishes run a tool like borgbackup to do a deduplicated backup to some other machine X. With deduplication it should be possible to store much more backups on machine X.

 

[LnxBil]

But is there at least any kind of snapshot of the VM taken before backup starts? Otherwise I cannot imagine how this should work.

Yes, the default Backup mode is snapshot. RAM contents and file contents are frozen and copied off in the Hypervisor while the VM runs and new data is written "somewhere else" to not interfere with the backup. After the backup is finished, the state of the VM is synchronized and everything goes their normal way.

after backup finishes run a tool like borgbackup to do a deduplicated backup to some other machine X. With deduplication it should be possible to store much more backups on machine X.

Unfortunately, that's not going to work, because the vzdump is a) compressed and b) if uncompressed, the file is not 'deduplicable' (or deplucatable???) because blocks are not aligned in the file and it is not a "real" 1:1 block copy of the VMs data but PVE internal format VMA. The only option to get it to deduplicate everything is to unpack the VMA to a "real" 1:1 block copy of the VM and use tools for this. I did something similar with ZFS and the CoW nature of the filesystem to get a much smaller backup footprint while storing 1:1 block copies and be able to manually recreate vzdump files for disaster recovery while having the possibility to just restore files. I talked about it on the german ProxTalks conference in 2017. I cannot believe how often I've already talked about this in english an my slides are only in german (was a german conference) ... I really need to get down and translate them into english....

 

[Toxik]

Thanks for the info. Just for reference: I've also found those two links about borgbackup and VMs:
https://github.com/borgbackup/borg/blob/master/docs/misc/create_chunker-params.txt
https://mail.python.org/pipermail/borgbackup/2019q1/001280.html
Basically it translates to you can use smaller chunks for better deduplication but this will eat your RAM.

But anyway the Proxmox VM backups should be the way to go instead of wbadmin.

 

[Toxik]

But anyway the Proxmox VM backups should be the way to go instead of wbadmin.

Actually my plan is to store the vzdump backups via NFS on a backup NAS that is connected via 10G SFP+. But that way each backup will be a full backup: 6 VMs with a size of 1 TB each will need to transfer 6 TB for a backup. Even with a fast transfer of 300 MB/s this takes up to 6 hours.

 

[LnxBil]

Actually my plan is to store the vzdump backups via NFS on a backup NAS that is connected via 10G SFP+. But that way each backup will be a full backup: 6 VMs with a size of 1 TB each will need to transfer 6 TB for a backup. Even with a fast transfer of 300 MB/s this takes up to 6 hours.

Are these numbers for compressed or raw disk size backups? Compression is really good, if it increases the throughput.

 

[Toxik]

Actually ~6 TB is the storage size in the host, that is available for VMs. So the worst case scenario is that the whole 6 TB size is used by various VMs with full disks.
Also the 6 TB is on a ZFS volume, which is compressed by lz4. So compressing the backups may not drastically reduce the size.

 

[Toxik]

Unfortunately, that's not going to work, because the vzdump is a) compressed and b) if uncompressed, the file is not 'deduplicable' (or deplucatable???) because blocks are not aligned in the file and it is not a "real" 1:1 block copy of the VMs data but PVE internal format VMA. The only option to get it to deduplicate everything is to unpack the VMA to a "real" 1:1 block copy of the VM and use tools for this. I did something similar with ZFS and the CoW nature of the filesystem to get a much smaller backup footprint while storing 1:1 block copies and be able to manually recreate vzdump files for disaster recovery while having the possibility to just restore files. I talked about it on the german ProxTalks conference in 2017. I cannot believe how often I've already talked about this in english an my slides are only in german (was a german conference) ... I really need to get down and translate them into english....

Why not just do it this way:

• Install Proxmox with ZFS
• Install backup server with ZFS (Proxmox, FreeNAS, ...)

To do the backup:

• Freeze the guest file system and create a snapshot of the guest's zvol
• Use zfs send to sync the snapshot to the backup server
• Rsync additional config files (that are currently included in VMA) to the backup server

This way it is possible to have:

• Unlimited snapshots of the guest in Proxmox and also on backup server
• Snapshot sync to backup server will be as fast as possible
• No unnecessary extracting, copying, overwriting

 

[Toxik]

According to what I read here, this should be possible: https://wiki.qemu.org/Features/Snapshots#Guest_Agent

Note that guest agent collaboration is also needed for snapshots using other methods, such as snapshots performed on btrfs, LVM, enterprise storage, etc.

Communication to the guest agent should be done via serial interface: https://wiki.qemu.org/Features/GuestAgent
Maybe there is a simplier method?

 

[Toxik]

Why not just do it this way:

• Install Proxmox with ZFS
• Install backup server with ZFS (Proxmox, FreeNAS, ...)

To do the backup:

• Freeze the guest file system and create a snapshot of the guest's zvol
• Use zfs send to sync the snapshot to the backup server
• Rsync additional config files (that are currently included in VMA) to the backup server

This way it is possible to have:

• Unlimited snapshots of the guest in Proxmox and also on backup server
• Snapshot sync to backup server will be as fast as possible
• No unnecessary extracting, copying, overwriting

Basically this is, what pve-zsync already does
I just don't know, if guest file system is frozen before snapshot. But AFAIK this is only useful, if guest applications also take care of it.

 

[mmenaz]

Basically this is, what pve-zsync already does

Also have a look at storage replication https://pve.proxmox.com/wiki/Storage_Replication

 

[LnxBil]

But AFAIK this is only useful, if guest applications also take care of it.

Yes, that is a problem. You can hook into the freeze and run your own scripts, e.g. turn on backup mode on your database, etc.