Allow different compression binaries and configuration for backups

[obrienmd]

We had great success adding pigz (multithreaded gzip), symlinking gzip to pigz, then turning off --rsyncable in your backup scripts (not compatible with pigz) and adding a cores limit (in our case 2) in your backup scripts gzip command line.

This sped up backups dramatically (almost linear per core). Perhaps you could add the option to use different compression binaries and options in Proxmox VE 2.0?

 

[dietmar]

This sped up backups dramatically (almost linear per core).

OK, added to the TODO list.

Perhaps you could add the option to use different compression binaries and options in Proxmox VE 2.0?

What other (besides pigz)?

 

[obrienmd]

What other (besides pigz)?

Not sure other than pigz... Not an expert on this stuff, I just know there are tons of people out there who use Proxmox w/ 8+ cores, and given backups are usually done at off-hours, why not use more than one (if only 2 or 3)?

 

[udo]

OK, added to the TODO list.



What other (besides pigz)?

Hi,
it's an old thread, but what's about pbzip2 for parallel bzip2? http://compression.ca/pbzip2/

bzip2 has the effort, if the archive is damage (e.g. one bit flip) only one block is corrupt! AFAIK with gzip the whole content after the damage is lost. Not very suitable for backup.

Udo

 

[dietmar]

Hi,
it's an old thread, but what's about pbzip2 for parallel bzip2? http://compression.ca/pbzip2/

Thanks for that hint - I will test that.

 

[dietmar]

bzip2 has the effort, if the archive is damage (e.g. one bit flip) only one block is corrupt! AFAIK with gzip the whole content after the damage is lost. Not very suitable for backup.

Seems that bzip2 needs much more CPU power that gzip (too much for a backup solution). I have some file where bzip2 is 3 times slower that gzip. So in the end parallel bzip is slower than single threaded gzip!

 

[escoreal]

Hi,

if you are planing to change something in the backup feature of Proxmox I recommend to take a look at storeBackup.
http://storebackup.org/

Especially the feature to use deduplication for block devices for the whole backup history looks promising.

At the moment I am testing this to use with an external script for KVM guests on logical volumes (taking backups with snapshots).

And I think such advanced backup features (version history, deduplication) would be great for Proxmox.

esco

 

[dietmar]

if you are planing to change something in the backup feature of Proxmox I recommend to take a look at storeBackup.
http://storebackup.org/

Thanks for that link.

 

[udo]

Seems that bzip2 needs much more CPU power that gzip (too much for a backup solution). I have some file where bzip2 is 3 times slower that gzip. So in the end parallel bzip is slower than single threaded gzip!

Hi Dietmar,
it's right that gzip is fast - but if you have enough cores pbzip2 is not too bad. It will be nice if we had the choice...

My test for the compression of a 20GB-disk-file:

time gzip xx
real    10m1.951s

time bzip2 xx
real    47m56.478s

time pbzip2 -p8 -k -f xx
real    6m55.389s

time pbzip2 -p16 -k -f xx
real    3m46.964s

time pbzip2 -p32 -k -f xx
real    2m5.743s

OK, it's depends on the server but for some people this will be a nice option.

Udo

 

[dietmar]

time pbzip2 -p32 -k -f xx
real    2m5.743s

OK, it's depends on the server but for some people this will be a nice option.

I guess that is even faster when using pigz ;-)

 

[udo]

I guess that is even faster when using pigz ;-)

Hi Dietmar,
of course - but with bzip the backup is safer agains bitfailure.

Udo

 

[escoreal]

Hi,

some results with storeBackup:

I made a very basic backup script with storeBackup and the performance is impressive. For example a 50 GB LV (NTFS, used space 17,8 GB) took 12m30s for the first and 6m36s for the second run. Used space with bzip2 is 6,5 GB on the backup server.

To access the backup system I use a local Samba share over an 1 GBit/s link.

The used script:

#!/bin/bash
volumes="$@"
snapshot_size="2G"
backup_dir="/mnt/backup"

for lv_path in $volumes
do
        lv_name=$(lvdisplay -c $lv_path|awk -F: '{ print $1 }' | awk -F/ '{ print $4 }')
        lvcreate -L $snapshot_size -s -n ${lv_name}_snapshot $lv_path
done

for lv_path in $volumes
do
        lv_name=$(lvdisplay -c $lv_path|awk -F: '{ print $1 }' | awk -F/ '{ print $4 }')
        storeBackup --checkDevicesCompr0 --linkToRecent ${lv_name}_last --logInBackupDir --checkDevices0 ${lv_path}_snapshot --checkDevicesDir0 volumes --checkDevicesBS0 10M --sourceDir /root/empty --backupDir $backup_dir --series ${lv_name}
done

for lv_path in $volumes
do
        lvremove -f ${lv_path}_snapshot
done

esco

 

[flow]

Hi Escoreal,

i must say I'm very intrigued by storeBackup (especially the deduplication feature)

I tried your script manually (for testing purposes just one VM)

Backing up und restoring a VM which was shut down during backup went fine.

But restoring a VM which was running didn't go so well.
The restored VM didn't find the bootsector and therefore didn't boot.

Have you had the same experience or do you backup your machines when they aren't running?

How do you restore your VMs?

I used something like this:

-cd into the directory containing the archive files

-bzcat -k *.bz2 | pv -cN bzcat > /dev/<VG-NAme>/vm-<ID>-disk-1

And thanks for pointing out storeBackup

 

[escoreal]

Hi flow,

I didn't any full recovery tests at the moment and made only some simple tests which worked. Made a diff of a "backup and recover" of storeBackup and and a copy with dd of the LV.
Example of recovery:

for i in $(ls);do bzcat $i >> /mnt/backup/recover.raw; done

But it shouldn't make any difference if the VM is running or not because storeBackup copies the snapshot of the LV. And the recovery should only be done in an inactive guest (LV or raw-file).

Another hint for backup (not only storeBackup) is to zero empty space. For windows you can use precompact.exe (for vhd-files) and for linux sfill:
http://serverfault.com/questions/16...-free-space-on-windows-for-better-compression

esco

 

[flow]

Thanks for the hint with precompact

Will try it later...

My restores are now working (it was just a typo )

 

[vcp_ai]

Can anybody please post results (improvements ?) by using precompact ??

Can it be used with windows XP KVM (ide raw) virtual machine ?

Also can not find a link to obtain this program (all what I've seen is that is a part of Windows 7 ...)

Thanks

 

[vcp_ai]

I've found a copy of precompact, and made tests.

Tests were done against a KVM Windows XP machine with a single 10 GB (IDE raw) disc.
There are 5,5 GB free, but may be sometime in the past disk was almost full, so it was supposed to be full of 'old-deleted' data. So mi interest for precompact....
Backup is against an external iscsi.
The results are:

Backup without compress:

 [FONT=Times New Roman][SIZE=2]
[/SIZE][SIZE=2]VMID   NAME                STATUS  TIME     SIZE      FILENAME  
  659  vpruautoma1-5.39   OK        00:12:02      8.79GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-19_28_01.tar  
                                        TOTAL     00:12:02      8.79GB [/SIZE][/FONT]

Backup with compress:

[FONT=Times New Roman][SIZE=2]VMID    NAME                STATUS      TIME      SIZE         FILENAME  
   659    vpruautoma1-5.39 OK        00:17:17     4.36GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-14_25_01.tgz 
                                       TOTAL       00:17:17     4.36GB [/SIZE][/FONT]

Same after precompact:

[FONT=Times New Roman][SIZE=2]VMID    NAME                STATUS      TIME      SIZE         FILENAME  
   659   vpruautoma1-5.39  OK        00:12:23     2.24GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-14_25_01.tgz 
                                        TOTAL      00:12:23     2.24GB [/SIZE][/FONT]

So with my configuration, I obtain very good results with precompact:
Same backup time than without compress, and half the space used than with compress ....

Regards

Vicente

 

[tafkaz]

Hi there,
i do have problems with the backup-speed in Proxmox too now and so i am very interested in better solutions.
Especially your approach with storebackup, Escoreal, seems to be very promising to me.
I have been using storebackup for other backups for nearly 7 years now, and it always worked like a charm.

But i am not very familiar with qemu, lvm, openvz and so i don't really know how to implement some vm-backup solution with storebackup.

Would it be possible for you to post a quick step-by-step "howto backup and restore proxmox-vms using storebackup"?
That would be just great!!!

And what do you think? Is integration of this outstanding backup-tool into Proxmox VE Web-GUI possible in the long run?
Could be just the solution you were looking for...

Thanks a lot for your help
Sascha

 

[escoreal]

Hi tafkaz,

I have been using storebackup for other backups for nearly 7 years now, and it always worked like a charm.

I use storeBackup since a month now to backup LVM snapshots without any problems.

Would it be possible for you to post a quick step-by-step "howto backup and restore proxmox-vms using storebackup"?

Running the above script for backup with a cron entry works for me.
Recovery with bzcat works, too.

And what do you think? Is integration of this outstanding backup-tool into Proxmox VE Web-GUI possible in the long run?

Yes, but then you wouldn't have one handy backup file and some requirements to the backup storage (hard links, handling a lot files). On the other hand the backup would be faster and could be versionized.
At the moment I am using ext3 and zfs over CIFS without any problems.

esco

 

[X1M]

I use BackupPC for my servers, it have a nice web interface. I run it inside each Virtual server, but not sure if that is optimal compared to the storebackup solution, I only take backup of the production data.

There is a LVM add-on for the BackupPC that focus on Virtual servers, it looks nice, maybe that could be a good solution to an Enterprise backup solution for Proxmox VE. I have not used it myself yet.

Have anyone experience with this?

Link:
http://www.biodec.com/tools/software/backuppc-lvm
http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=User_Scripts_-_Client_-_Backuppc-lvm