Allow different compression binaries and configuration for backups

tafkaz

Active Member
Dec 1, 2010
87
0
26
Geilenkirchen Germany
sfill could be very handy for my backup size problem.
(the backup size keeps growing)

On my KVM-VM with two VDs on an external iscsi storage:
How would i use it to free already empty space for good?
Shouldn't touch any real data of course...

thanks for your advice
Sascha

Forgot to mention, that the VM is a debian machine
 
Last edited:

THe_ZiPMaN

New Member
Aug 10, 2009
2
0
1
Italy
www.zipman.it
I guess that is even faster when using pigz ;-)
Yes, it's a bit faster. But it's not always the speed that you are interested in.
For example in Hetzner you have only 100GB of backup space included with the base contract, so the compression ratio is often more important than the CPU time spent (usually backups run in the night, when the load is low).

bzip2 have a dramatically better compression ratio compared with gzip.
and xz have an even better compression, but it's 10 times slower than gzip.

Here you are the times and size of the compression of an LVM of 10GB with a fresh debian squeeze 64 install on it

Using pigz
Code:
# time (cat /dev/vgs1/vm-100-disk-1 | pigz > vm-100-disk-1.gz)

real    2m24.517s
user    2m56.836s
sys    0m23.460s

-rw-r--r-- 1 root root 187480727 Apr 22 23:45 vm-100-disk-1.gz
Using pbzip2
Code:
# time (cat /dev/vgs1/vm-100-disk-1 | pbzip2 > vm-100-disk-1.bz2)

real    2m35.549s
user    4m4.963s
sys    0m21.649s

-rw-r--r-- 1 root root 161555772 Apr 22 23:36 vm-100-disk-1.bz2
Using xz
Code:
# time (cat /dev/vgs1/vm-100-disk-1 | xz > vm-100-disk-1.xz)

real    20m36.793s
user    19m16.798s
sys    0m27.434s

-rw-r--r-- 1 root root 132791004 Apr 23 00:08 vm-100-disk-1.xz
As you can see the difference between pigz and pbzip2 is not so big (the limit in this case is the disk speed).

I think it's important to offer the user the widest possible range of possibilities. He will choose the one that better fits his needs.

The best would be to check for the availability on the system of all the compressions utilities and if they are installed to let them being choose.
The more comprehensive list of stream compressors in debian that I can recall is: gzip, bzip2, pigz, pbzip2, lbzip2, xz, lzma.
The default, if available, should be pigz or pbzip2 and I'm more favorable with the second both for the higest compression ratio and, more important, for higher recovery success in case of corruption.
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,707
373
103
Austria
www.proxmox.com
The default, if available, should be pigz or pbzip2 and I'm more favorable with the second both for the higest compression ratio and, more important, for higher recovery success in case of corruption.
We use 'lzo' now as default, becaue we do not want to waste processor power. But we can add 'bzip/pbzip' as additional option. Please file a bug at http://bugzilla.proxmox.com if you are interested in that feature.
 

ccube

Active Member
Apr 5, 2011
49
0
26
Passau, Germany, Germany
I thing "wasting" cpu power can be useful in some situations. some of our customers, need backups of aroud 100-200GB every night. So with lzo it is a process of almost the whole day.
With pigz on all of the 16 cores i can reduce the data compression to a few hours...

I think it should be possible to let the user chose which compression he wants to use.

Maybe I can add them. :) Some hints, where to look at? Are there some git repos with pve source?
 

term

Member
Aug 29, 2013
68
1
6
I realize this is an ancient thread, but does anyone have storebackup working with proxmox 3.1, and if so, how?
 
Dec 22, 2010
78
0
6
I realize this is an ancient thread, but does anyone have storebackup working with proxmox 3.1, and if so, how?
Yes, with about 300 VMs (about 20 TB backup space). Works great but I had to made a few scripts to implement, monitor, checks backups and make possible to have it offsite and offline that don't work with any architecture.

esco
 

tafkaz

Active Member
Dec 1, 2010
87
0
26
Geilenkirchen Germany
Yes, with about 300 VMs (about 20 TB backup space). Works great but I had to made a few scripts to implement, monitor, checks backups and make possible to have it offsite and offline that don't work with any architecture.

esco
Wow, that sounds great!
Would you mind sharing your insights with us?
A little howto maybe?

I would defo appreciate that!

Thank you
Sascha
 
Dec 22, 2010
78
0
6
A little howto maybe?
A good start ;)
http://forum.proxmox.com/threads/44...d-configuration-for-backups?p=35171#post35171

The only thing I don't like is that this is a push backup and I use SMB mounts for that.

So I plan to change this to pull with rsync over SSH and with that to write the scripts more portable so they can be published..

But the recovery performance wasn't really that good. And we use this backups to generate testing environments. Thanks to the fact that the backups aren't in any proprietary format it was easy to make a own faster recovery script:
http://savannah.nongnu.org/support/?108391
http://savannah.nongnu.org/support/download.php?file_id=29161

esco
 

hevisko

New Member
Nov 15, 2012
21
1
1
Any progress or plans for including different compressors like lzma, pbzip2, pigz with options (like rsyncable)?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!