Proxmox and slow backup

Y

yarii

Renowned Member
Mar 24, 2014
150
8
83
Why does Proxmox use old gzip for backup? It's using only one-core when packing :( Does anyone tired to use pigz with proxmox? Would it be enough to do"ln -s /usr/bin/pigz gzip? :-) What are cons of doing that?
 
Last edited:
dietmar said:
We are using lzop by default. (pgzip is a waste of CPU resources).
Click to expand...

Using backup function with LZO packing is optymized for parrell but GZIP not.

Is lzo better than gzip?
Proxmox is offering both.
 
If you search the forums you can find my instructions on using pigz.

I *think* that pigz can be faster than lzop but like dietmar said pigz is a waste of CPU.
My backups are written to a LUKS encrypted disk and the encryption is usually the bottleneck so I have been quite happy with lzop.
 
e100 said:
If you search the forums you can find my instructions on using pigz.

I *think* that pigz can be faster than lzop but like dietmar said pigz is a waste of CPU. My backups are written to a LUKS encrypted disk and the encryption is usually the bottleneck so I have been quite happy with lzop.
Click to expand...

...OK

Just look on my top:
205788 root 20 0 6648 1128 376 S 81 0.0 72:34.43 lzop
CPU is going from 70% to 99%.

This machine also uses pbzip2 for packing big tcpdump logs. And the CPU usage for pbzip2 is going from 200% to 800%.

So where is the problem?

I got 4 VM's with 1TB on each. I need to backup that once per week. Right now it takes me too long.

I don't feel that my server is working 100% hard as possible. Why lzop is not using whole cpu power?
 
lzop is fast but is only single threaded.
lzop with less than 100% CPU means your system is unable to feed data to lzop faster than lzop can compress it or you are unable to write the resulting data to disk fast enough.

With fast IO backups can happen fast, especially if the data compresses well. if it does not compress well the write speed of the backup destination will have a huge impact on the speed.
Here are some numbers from my servers recent backups using lzop:
Code: 
100: Mar 25 00:21:49 INFO: transferred 80530 MB in 404 seconds (199 MB/s)

104: Mar 25 00:29:06 INFO: transferred 107374 MB in 431 seconds (249 MB/s)

177: Mar 25 01:03:59 INFO: transferred 302795 MB in 865 seconds (350 MB/s)

182: Mar 25 01:08:20 INFO: transferred 77309 MB in 253 seconds (305 MB/s)

197: Mar 23 22:08:34 INFO: transferred 6442 MB in 51 seconds (126 MB/s)

208: Mar 23 22:42:07 INFO: transferred 107374 MB in 953 seconds (112 MB/s)

130: Mar 23 22:06:08 INFO: transferred 633507 MB in 3962 seconds (159 MB/s)

I can write to my backup media at about 120MB/sec, the ones that were faster compressed better.
If I had faster backup media I am confident the numbers would be better.
 
e100 said:
lzop is fast but is only single threaded.
lzop with less than 100% CPU means your system is unable to feed data to lzop faster than lzop can compress it or you are unable to write the resulting data to disk fast enough.
Click to expand...

I changed destination media, now it's much better - LZO take about 100% 1-core CPU power.
LZO still uses only one core.

On LZO homepage I've found something like GNU parrel tool. Do You think that parallelism of LZO will help me to do my backups faster?

How can I check where is bottleneck? (for me it's LZO and no parraleism)
 
Last edited:
You must log in or register to reply here.
Top Bottom