Allow different compression binaries and configuration for backups

obrienmd

Member
Oct 14, 2009
109
0
16
We had great success adding pigz (multithreaded gzip), symlinking gzip to pigz, then turning off --rsyncable in your backup scripts (not compatible with pigz) and adding a cores limit (in our case 2) in your backup scripts gzip command line.

This sped up backups dramatically (almost linear per core). Perhaps you could add the option to use different compression binaries and options in Proxmox VE 2.0?
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,707
373
103
Austria
www.proxmox.com

obrienmd

Member
Oct 14, 2009
109
0
16
What other (besides pigz)?
Not sure other than pigz... Not an expert on this stuff, I just know there are tons of people out there who use Proxmox w/ 8+ cores, and given backups are usually done at off-hours, why not use more than one (if only 2 or 3)?
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,707
373
103
Austria
www.proxmox.com
bzip2 has the effort, if the archive is damage (e.g. one bit flip) only one block is corrupt! AFAIK with gzip the whole content after the damage is lost. Not very suitable for backup.
Seems that bzip2 needs much more CPU power that gzip (too much for a backup solution). I have some file where bzip2 is 3 times slower that gzip. So in the end parallel bzip is slower than single threaded gzip!
 
Dec 22, 2010
78
0
6
Hi,

if you are planing to change something in the backup feature of Proxmox I recommend to take a look at storeBackup.
http://storebackup.org/

Especially the feature to use deduplication for block devices for the whole backup history looks promising.

At the moment I am testing this to use with an external script for KVM guests on logical volumes (taking backups with snapshots).

And I think such advanced backup features (version history, deduplication) would be great for Proxmox.

esco
 

udo

Famous Member
Apr 22, 2009
5,888
166
83
Ahrensburg; Germany
Seems that bzip2 needs much more CPU power that gzip (too much for a backup solution). I have some file where bzip2 is 3 times slower that gzip. So in the end parallel bzip is slower than single threaded gzip!
Hi Dietmar,
it's right that gzip is fast - but if you have enough cores pbzip2 is not too bad. It will be nice if we had the choice...

My test for the compression of a 20GB-disk-file:
Code:
time gzip xx
real    10m1.951s

time bzip2 xx
real    47m56.478s

time pbzip2 -p8 -k -f xx
real    6m55.389s

time pbzip2 -p16 -k -f xx
real    3m46.964s

time pbzip2 -p32 -k -f xx
real    2m5.743s
OK, it's depends on the server but for some people this will be a nice option.

Udo
 
Dec 22, 2010
78
0
6
Hi,

some results with storeBackup:

I made a very basic backup script with storeBackup and the performance is impressive. For example a 50 GB LV (NTFS, used space 17,8 GB) took 12m30s for the first and 6m36s for the second run. Used space with bzip2 is 6,5 GB on the backup server.

To access the backup system I use a local Samba share over an 1 GBit/s link.

The used script:
Code:
#!/bin/bash
volumes="$@"
snapshot_size="2G"
backup_dir="/mnt/backup"

for lv_path in $volumes
do
        lv_name=$(lvdisplay -c $lv_path|awk -F: '{ print $1 }' | awk -F/ '{ print $4 }')
        lvcreate -L $snapshot_size -s -n ${lv_name}_snapshot $lv_path
done

for lv_path in $volumes
do
        lv_name=$(lvdisplay -c $lv_path|awk -F: '{ print $1 }' | awk -F/ '{ print $4 }')
        storeBackup --checkDevicesCompr0 --linkToRecent ${lv_name}_last --logInBackupDir --checkDevices0 ${lv_path}_snapshot --checkDevicesDir0 volumes --checkDevicesBS0 10M --sourceDir /root/empty --backupDir $backup_dir --series ${lv_name}
done

for lv_path in $volumes
do
        lvremove -f ${lv_path}_snapshot
done
esco
 

flow

Member
Nov 10, 2010
28
1
23
Hi Escoreal,

i must say I'm very intrigued by storeBackup (especially the deduplication feature) :D

I tried your script manually (for testing purposes just one VM)

Backing up und restoring a VM which was shut down during backup went fine.

But restoring a VM which was running didn't go so well.
The restored VM didn't find the bootsector and therefore didn't boot.

Have you had the same experience or do you backup your machines when they aren't running?

How do you restore your VMs?

I used something like this:

-cd into the directory containing the archive files

-bzcat -k *.bz2 | pv -cN bzcat > /dev/<VG-NAme>/vm-<ID>-disk-1

And thanks for pointing out storeBackup :)
 
Dec 22, 2010
78
0
6
Hi flow,

I didn't any full recovery tests at the moment and made only some simple tests which worked. Made a diff of a "backup and recover" of storeBackup and and a copy with dd of the LV.
Example of recovery:
Code:
for i in $(ls);do bzcat $i >> /mnt/backup/recover.raw; done
But it shouldn't make any difference if the VM is running or not because storeBackup copies the snapshot of the LV. And the recovery should only be done in an inactive guest (LV or raw-file).

Another hint for backup (not only storeBackup) is to zero empty space. For windows you can use precompact.exe (for vhd-files) and for linux sfill:
http://serverfault.com/questions/16...-free-space-on-windows-for-better-compression

esco
 

flow

Member
Nov 10, 2010
28
1
23
Thanks for the hint with precompact :)

Will try it later...

My restores are now working (it was just a typo :) )
 

vcp_ai

Active Member
Jul 28, 2010
177
2
38
Valencia -Spain-
Can anybody please post results (improvements ?) by using precompact ??

Can it be used with windows XP KVM (ide raw) virtual machine ?

Also can not find a link to obtain this program (all what I've seen is that is a part of Windows 7 ...)

Thanks
 

vcp_ai

Active Member
Jul 28, 2010
177
2
38
Valencia -Spain-
I've found a copy of precompact, and made tests.

Tests were done against a KVM Windows XP machine with a single 10 GB (IDE raw) disc.
There are 5,5 GB free, but may be sometime in the past disk was almost full, so it was supposed to be full of 'old-deleted' data. So mi interest for precompact....
Backup is against an external iscsi.
The results are:

Backup without compress:
Code:
 [FONT=Times New Roman][SIZE=2]
[/SIZE][SIZE=2]VMID   NAME                STATUS  TIME     SIZE      FILENAME  
  659  vpruautoma1-5.39   OK        00:12:02      8.79GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-19_28_01.tar  
                                        TOTAL     00:12:02      8.79GB [/SIZE][/FONT]
Backup with compress:
Code:
[FONT=Times New Roman][SIZE=2]VMID    NAME                STATUS      TIME      SIZE         FILENAME  
   659    vpruautoma1-5.39 OK        00:17:17     4.36GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-14_25_01.tgz 
                                       TOTAL       00:17:17     4.36GB [/SIZE][/FONT]
Same after precompact:
Code:
[FONT=Times New Roman][SIZE=2]VMID    NAME                STATUS      TIME      SIZE         FILENAME  
   659   vpruautoma1-5.39  OK        00:12:23     2.24GB    /mnt/iscsi1/vzdump-qemu-659-2011_05_07-14_25_01.tgz 
                                        TOTAL      00:12:23     2.24GB [/SIZE][/FONT]
So with my configuration, I obtain very good results with precompact:
Same backup time than without compress, and half the space used than with compress ....

Regards

Vicente
 
Last edited:

tafkaz

Active Member
Dec 1, 2010
87
0
26
Geilenkirchen Germany
Hi there,
i do have problems with the backup-speed in Proxmox too now and so i am very interested in better solutions.
Especially your approach with storebackup, Escoreal, seems to be very promising to me.
I have been using storebackup for other backups for nearly 7 years now, and it always worked like a charm.

But i am not very familiar with qemu, lvm, openvz and so i don't really know how to implement some vm-backup solution with storebackup.

Would it be possible for you to post a quick step-by-step "howto backup and restore proxmox-vms using storebackup"?
That would be just great!!!

And what do you think? Is integration of this outstanding backup-tool into Proxmox VE Web-GUI possible in the long run?
Could be just the solution you were looking for...

Thanks a lot for your help
Sascha
 
Dec 22, 2010
78
0
6
Hi tafkaz,

I have been using storebackup for other backups for nearly 7 years now, and it always worked like a charm.
I use storeBackup since a month now to backup LVM snapshots without any problems.

Would it be possible for you to post a quick step-by-step "howto backup and restore proxmox-vms using storebackup"?
Running the above script for backup with a cron entry works for me.
Recovery with bzcat works, too.

And what do you think? Is integration of this outstanding backup-tool into Proxmox VE Web-GUI possible in the long run?
Yes, but then you wouldn't have one handy backup file and some requirements to the backup storage (hard links, handling a lot files). On the other hand the backup would be faster and could be versionized.
At the moment I am using ext3 and zfs over CIFS without any problems.

esco
 

X1M

Member
Apr 19, 2010
22
0
21
I use BackupPC for my servers, it have a nice web interface. I run it inside each Virtual server, but not sure if that is optimal compared to the storebackup solution, I only take backup of the production data.

There is a LVM add-on for the BackupPC that focus on Virtual servers, it looks nice, maybe that could be a good solution to an Enterprise backup solution for Proxmox VE. I have not used it myself yet.

Have anyone experience with this?

Link:
http://www.biodec.com/tools/software/backuppc-lvm
http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=User_Scripts_-_Client_-_Backuppc-lvm
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!