Speed up vzdump using pigz

e100

Renowned Member
Nov 6, 2010
1,268
45
88
Columbus, Ohio
ulbuilder.wordpress.com
Using pigz in place of gzip drastically increases the speed of my KVM backups(I do not use openvz)
I see supporting pigz is on the todo list, until then I used the method below.

vzdump passes the --rsyncable option to gzip, this is not supported by pigz.
Because of this you must edit vzdump to remove this option

I did not like having to edit vzdump because I wanted to avoid potential hassles with future updates so I choose to use a wrapper script to call pigz.

First you need pigz, I used pigz from Squeeze:
Code:
#wget http://ftp.us.debian.org/debian/pool/main/p/pigz/pigz_2.1.6-1_amd64.deb
#dpkg --install  pigz_2.1.6-1_amd64.deb
Next I created the wrapper script, edit the file:
vi /bin/pigzwrapper
Code:
#!/bin/sh
PATH=${GZIP_BINDIR-'/bin'}:$PATH
GZIP="-1"
exec /usr/bin/pigz -p 4  "$@"
I set GZIP="-1" to overwrite the --rsyncable option vzdump sets and to set the compression level lower so it compresses faster. You might choose a higher level of compression.
On the exec line change -p 4 to the number of processor cores you want pigz to use.

Save that edited file.

Make the file executable:
Code:
#chmod 755 /bin/pigzwrapper
Rename gzip:
Code:
#mv /bin/gzip /bin/gzip.original
Copy wrapper into place:
Code:
#cp /bin/pigzwrapper /bin/gzip
Now when gzip is executed you get pigz instead.
 
Nice. My proxmox box is just a dual-core, but that is better than nothing maybe. I will have to try and see...
 
WOOOHA! thanks for the great hint! changed from 15meg/second to 90 meg/second on a dual-quadcore 2.66 xeon (no hyperthreading) via NFS. regards from aachen, thomas
 
i use it too. very fast! i mean extreme fast. use it since one or two years. be smart - integrate it ;)
 
The plan is to integrate lzo (or lz4) in future versions.

I have not looked at lzo or lz4 but I do have some thoughts related to future features.

gzip is nice with the rsyncable option, great for people who want to transfer their backups across the Internet.

pigz helps the speed but breaks the rsyncable feature.

As far as I can see neither lzo or lz4 have a feature like gzip rsyncable option.

I would like to see vzdump support various methods and allow us to configure the method that best suits our needs.

Two ways to configure this would be great:
1. Per VM in the VM config file, this would override the default of you do not specify a compression option on the command line.
2. At the command line, this would override the VM config if you specify a specific method.
 
Yes, the plan is to make that configurable.

Resurrecting this thread.. pigz now supports --rsyncable flag in version 2.2.3 and on. Has anyone tried installing it (there is a backport version on debian.org) on either 1.x or 2.x ?

Is lzo so far superior that noone is using anymore? I think I'd rather use pigz with a couple free CPUs thrown at it than lzo.. anyone have an opinion?
 
Can I just +1 how much I want this in the current version - I've just tested this and its covered most of my backup woes - backups that were going 40-60mb/s with lzo are now full backup destination speed (110 mb/s) - this now makes the backups run faster than without compression, awesome.

All I'd like now is storing the VM state on backup so on restore the VM does not 'crash'!
 
yes, lzo uses much less resources.

Though LZO uses less resources, when I have the spare CPUs available, I'd rather use them than a single threaded LZO. Please consider adding pigz, or perhaps also pbzip2 as options??
 
Though LZO uses less resources, when I have the spare CPUs available, I'd rather use them than a single threaded LZO. Please consider adding pigz, or perhaps also pbzip2 as options??

Any decision whether to include parallel compression software like pigz and pbzip2?
 
  • Like
Reactions: V.Busch

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!