Compression drastically affects Proxmox Backup Server performances

[_gabriel]

First PBS backup is slower than vzdump not only because compression but, imo, mainly because data at destination isn't saved "sequentially".
Data is written into half a million files of 2MB-4MB into 65536 subfolders of .chunks.
Outside the .chunk folder, there is only the mandatory index to define used chunks by each backup.

 

[mauro70]

Here the results of the new three tests:

(t1): backup via PBS: vzdump 105 --notes-template '{{guestname}}' --node pvesrv1 --remove 0 --storage pbs-var-lib-vz --mode stop
(t2): backup via local storage, ZSTD compression: vzdump 105 --storage local --compress zstd --node pvesrv1 --notes-template '{{guestname}}' --remove 0 --mode stop
(t3): backup via local storage, no compression: vzdump 105 --mode stop --storage local --notes-template '{{guestname}}' --compress 0 --node pvesrv1 --remove 0

(t1) average speed 195MB/sec (see backup-pbs-full-stopmode.log)
(t2) average speed 230MB/sec (see backup-local-zst-compression-full-stopmode.log)
(t3) average speed 860MB/sec (see backup-local-no-compression-full-stopmode.log)

I want you to know in advance that before each test I explicitly deleted the datastore and issued the command "rm -rf /var/lib/vz". So I really do not know why the statement "backup was done incrementally" is in the backup log of PBS.

First PBS backup is slower than vzdump not only because compression but, imo, mainly because data at destination isn't saved "sequentially".
Data is written into half a million files of 2MB-4MB into 65536 subfolders of .chunks.
Outside the .chunk folder, there is only the mandatory index to define used chunks by each backup.

Yes, as mentioned before, surely PBS introduces additional features (on top of compression) than vzdump: that should justify the 15% difference between (t1) and (t2), but in any case (t1) and (t2) are of the same order of magnitude.
However, I think it is evident that the difference between (t1 or t2) with (t3) can be only due to the compression feature, which hurts my old CPU, quite surely negligeable with more modern and powerful CPUs. That's why the option to disable compression could be appreciated by who, like me, has older CPUs which could achieve in any case higher speed without the compression.

Cheers
Mauro

 

[_gabriel]

This is the same test as you already did ...
you havn't tested a next backup !
+ a next backup without shutdown vm ( so mode suspend or snapshot) to even more speedup process.
this is why PBS has been.
vzdump is only good for one shot backup, but not really great for regular backups.

 

[mauro70]

Sorry _gabriel, what is the purpose honestly to test the next backup?
I mean, of course it will be quicker, however it will be quicker because it is an incremental backup, meaning there is less data to be backed up (so less data to be uncompressed/compressed). But that's not under discussion, incremental backups are in place to speedup the backup process, nobody says the contrary.
However the purpose of my post is to show that compression, at any stage (first backup, subsequent backups), introduces an overhead more or less negligeable, this is cristally clear.

Said that, the outcome of these tests can be accepted (and it can be decided to do something) or not, really, friends again.

Cheers
Mauro

 

[_gabriel]

all backups with PBS are full one ...
edit: (sorry for my english wording ...) there isn't diff betweeen backups in PBS.
even in snapshot/suspend mode which do not reread 100% source (only read changed blocks in VM) : on target , all are full backups.
This is the logic to understand.

 

[Dunuin]

(t1) average speed 195MB/sec (see backup-pbs-full-stopmode.log)
(t2) average speed 230MB/sec (see backup-local-zst-compression-full-stopmode.log)
(t3) average speed 860MB/sec (see backup-local-no-compression-full-stopmode.log)

Wouldn't it be better to compare the backup time instead of the MB/s?
Lets say you got a virtual disk with mostly zero blocks that are well compressible. Lets say without compression it copies with 800MB/s and with compression with 200MB/s. In case it would save 75% of space the backup, with 200MB/s could finish as fast as the uncompressed backup with 800MB/s.
But again, benchmarks aren't easy. If you work with compression muktiple runs with more and less compressible data would be needed.
Zstd won't help much backing up a virtual disk that is filled to 95% with already compressed video files. With a virtual disk that is only filled 10% with a database that got lots of the same DB entries it would save so much space and time.

At least here the average size of my chunks of my 3 PBSs is between 1-2MB and nearly all raw chunks are 4MB. So 50-75% less data to transfer/store and 2-4 times speed improvement which can compensate the slower transfer speeds.

 

[mauro70]

Wouldn't it be better to compare the backup time instead of the MB/s?
Lets say you got a virtual disk with mostly zero blocks that are well compressible. Lets say without compression it copies with 800MB/s and with compression with 200MB/s. In case it would save 75% of space the backup, with 200MB/s could finish as fast as the uncompressed backup with 800MB/s.
But again, benchmarks aren't easy. If you work with compression muktiple runs with more and less compressible data would be needed.
Zstd won't help much backing up a virtual disk that is filled to 95% with already compressed video files. With a virtual disk that is only filled 10% with a database that got lots of the same DB entries it would save so much space and time.

At least here the average size of my chunks of my 3 PBSs is between 1-2MB and nearly all raw chunks are 4MB. So 50-75% less data to transfer/store and 2-4 times speed improvement which can compensate the slower transfer speeds.

Everything right, no objection.
However, this does not change the matter: benchmarking isn't easy, we all agree on that. The optimal strategy (in any context), should be decided on the field. But the prerequisite to select the best strategy is that the tools (whatever are) have the necessary capabilities to allow the client to perform tests with the real architecture (and possibly with the real workload, thing which is much complex in production environments) by varying the configurations.
PBS is a good tool, but lacks the capability to deselect the compression, that's a fact. In some architectures, unselecting compression would not bring any added value (the benchmarks mentioned by Tom above shows exactly that), in others quite surely yes, it would (see my case, and there are many like me). It is now a matter to see if PROXMOX Development Team wants to seize the opportunity to add such capability to the tool or not, as simple as that.
It goes without saying that other backup tools (Veeam, Nakivo just to mention two examples) have such option. I do not see that drastically revolutionizing the spirit and the fundamentals of PBS...
To be honest, I do not know if it is possible to submit a request for enhancement, maybe there is a place, I'll give a look and in case I'll submit it.
Cheers
Mauro

 

[Dunuin]

To be honest, I do not know if it is possible to submit a request for enhancement, maybe there is a place, I'll give a look and in case I'll submit it.

Bugtracker is for bugs as well as feature requests: https://bugzilla.proxmox.com/enter_bug.cgi

 

[mauro70]

Thanks for pointing it, I did that. Hopefully will be taken into account.
Cheers
M

 

[JensF]

In this line: (t2): backup via local storage, ZSTD compression: vzdump 105 --storage local --compress zstd --node pvesrv1 --notes-template '{{guestname}}' --remove 0 --mode stop you can add the option --zstd 0 to use half of the available cores. Without that it always use only one of the cores. It's an 8 cores Xeon right? With the use of 4 cores you should have as almost the same speed as without compression...

 

[mauro70]

interesting hint, and indeed the result is about 780MB/sec, with the "side effect" that CPU usage raised at almost 60% during the execution of vzdump (more or less 4 cores used).
This of course is a good trick at the price of hurting the CPU.
However, it could be interesting to test it also with PBS: so how to configure that parameter into the backup job using PBS? In the interface there is no such possibility.
Thanks

 

[RolandK]

@Dunuin
> But yes, if you store 10 backups of each VM with only 1% changes between the backups, the first backup will be slow
> and read + transfer 100%. The next ones will only consume additional ~1% space and and ~99% of that VM can
> be skipped, so massive speed and space improvement.

i see typically see MUCH larger incremental diff with my VMs, even on those which are really idle...


btw, what pve version did you test with @mauro70 ? (because of https://bugzilla.proxmox.com/show_bug.cgi?id=4605 )

 

[mauro70]

@Dunuin
> But yes, if you store 10 backups of each VM with only 1% changes between the backups, the first backup will be slow
> and read + transfer 100%. The next ones will only consume additional ~1% space and and ~99% of that VM can
> be skipped, so massive speed and space improvement.

i see typically see MUCH larger incremental diff with my VMs, even on those which are really idle...


btw, what pve version did you test with @mauro70 ? (because of https://bugzilla.proxmox.com/show_bug.cgi?id=4605 )

PVE 8 and PBS 3

 

[RolandK]

>However, it could be interesting to test it also with PBS: so how to configure that
>parameter into the backup job using PBS? In the interface there is no such possibility.

there is no way, as pbs does not use commandline binary ( zstd multihread-capable binary)

not sure how many threads pbs is using with compression

 

[Dunuin]

A few months ago fiona added support for defining the number of workers for a backup job. Not sure how this effects compression, but my guess would be that fewer workers working in parallel also would mean less data that needs to be compressed in parallel and vice versa.

 

[mauro70]

>However, it could be interesting to test it also with PBS: so how to configure that
>parameter into the backup job using PBS? In the interface there is no such possibility.

there is no way, as pbs does not use commandline binary ( zstd multihread-capable binary)

not sure how many threads pbs is using with compression

this simply corroborates my view point: tuning options should be added to PBS, either to be configured via the PVE interface or in a config file (like vzdump.conf for vzdump). Without these, clients do not have opportunity to set the optimal backup strategy based on their own infrastructure.
Of course my case is focused on compression, but actually the principle is general and applicable to many other options.

 

[RolandK]

yep, but i would consider this low priority. it's typically not a problem for >99% of all users , so it's a special case

 

[mauro70]

yep, but i would consider this low priority. it's typically not a problem for >99% of all users , so it's a special case

well, arguable: there are few posts in the forum reporting slow performances... and probably there are also a lot of clients who don't realize the issue (or probably never spent some time to perform tests).

In any case I have submitted the enhancement request here --> https://bugzilla.proxmox.com/show_bug.cgi?id=4900 but never expected it "by tomorrow" , I can imagine that the team has surely more priority tasks to do... perhaps in 3/6 months (if accepted)? in the end, enhancement 4605 mentioned above was in the same context and managed with that timeframe more or less...

 

[Dunuin]

perhaps in 3/6 months (if accepted)? in the end, enhancement 4605 mentioned above was in the same context and managed with that timeframe more or less...

I wouldn't count on that. There are so many feature requests that took many years to be implemented. People for example waited 2.5 years for a dark mode and while not being that important it was probably one of the most requested features.

 

[rigel.local]

Wow, is using spinning Seagate Exos HDD 7200rpm drives something like 8 drives in configurations like this is a bad idea?
4 VDEVs with 2 drives in mirror

Enterprise SATA SSDs with DWPD > 1 are crazy expensive