Which Proxmox Backup Compression is Best for Rsync --fuzzy?

[Lonnie]

Proxmox provides 3 options for backup compression:

1. LZO (fast)
2. GZIP (good)
3. ZSTD (fast and good)

Which of these 3 would be most efficient for "Rsync --fuzzy Synchronization", which uses a previous backup at an offsite destination as the basis for minimizing data transfer of the most current backup?

If you don't know what I'm talking about, then read on below.

Although I backup my virtual machines daily onsite, once per week I also use rsync's --fuzzy argument to perform an offsite sync-backup.

Basically, the fuzzy option allows rsync to use a previous backup file as a means of efficiently creating a copy of your latest backup file. Instead of having to transfer the entire backup file over the wire, rsync compares the source file with the destination file. Any matching blocks of data do not have to be transferred; only the differences between the two files get transferred over the wire. For example, at one company they have a IBM DB2 database that is 50GB yet I'm able to back up that file daily and less than a 1GB actually gets transferred over the internet. After the backup completes, the source file and destination file have matching checksums.

For example, lets say "proxmox virtual machine 100" is being backed up using zst format, this command will use a previous backup (at the destination) as a fuzzy basis for transferring the current backup from source to destination:

rsync -vvv --compress --chmod=ug=rwX --delete-after --fuzzy --human-readable --itemize-changes --links --partial --progress --recursive --stats --timeout=7200 --times /home/companyA/virtualMachines/dump/vzdump-*-100-*.zst username@linux-remote-backup.com:/mnt/companyA/virtualMachines/100 ;

Notice "--fuzzy" in the command above and the wildcards "vzdump-*-100-*.zst" used to address the name differences between the current and previous backup files.

Which of Proxmox's 3 compression types would be most efficient for "Rsync --fuzzy Synchronization"? When it comes to this priority, would I be better off backing up the virtual machines's uncompressed folder instead of syncing the backup files instead?

 

[H4R0]

I did several tests with duplicity wich does incremental backups.

For incremental backups the only option is to go with no compression for the backup and let duplicity handle the compression.

Only with that, incremental backups work e.g a daily backup results in 100mb vs 1,5gb in case of zstd. (source vm size is 8gb)

Otherwise zstd is the second best, gzip takes longer with almost same compression result, LZO took longer then zstd and result in much bigger files.

About all compression methods resulted in incremental backups > 1GB, this might be fine for you but i wanted to reduce bandwidth as much as possible. Local backup storage didn't matter in my case i have enough space to make uncompressed backups.

 

[Lonnie]

It seems like there should be a compression file-type, in existence, that places priority on limiting the unnecessary variances that occur in succeeding backup files. If such a compression file-type exists, I'd like to see it get added to Proxmox.

In the meantime, I'll have to endure what is ironically the most efficient thing to remotely sync: an uncompressed backup.

 

[vmware]

The compression "file-type" already exists - the "--rsyncable" flag in GZIP.

https://superuser.com/questions/482758/rsync-friendly-gzip

 

[Lonnie]

For incremental backups the only option is to go with no compression for the backup and let duplicity handle the compression.

In your testing, were you backing up LXC containers exclusively?

In Proxmox 7, (by default) LXC containers backup to tar.zst, while KVMs backup to vma.zst. No-Compression backups of KVM produce a .vma file.

I read that VMA files create a large diff. In your testing, were KVM backups included (ones having the .vma file format)?

 

[H4R0]

Afaik I tested .vma from KVM and .tar.xxx from LXC.

I migrated from duplicity to pbs a long time ago.

Also I use zfs send for offsite sync now.

Takes less then 10 minutes to complete.