Compression drastically affects Proxmox Backup Server performances

[Dunuin]

Calculate it yourself. Lets say this gives you 400 IOPS and you are using 8x 20TB HDDs. 80TB of backups would mean ~40 million chunk files spread in random order across the disks and when running a GC or full-verify each single chunk file needs to be randomly read + written. Just think of how fast it would be to sha512sum+touch 80TBs of bigger jpgs or smaller mp3s...thats the performance you can expect.

 

[EllyMae]

If I were to buy 25TB of SSDs, I would find better uses for them than backups. To me, your engineering equation doesn't make sense. You need to design a system that works well with inexpensive media.

 

[ness1602]

That is your logic, but if your customers asks for sub-60 min recovery, you buy enterprise SSD.

 

[Dunuin]

Another point was future-proofing. They invested many years of development to engineer a backup solution from scratch. Sooner or later SSDs will become cheaper than HDDs and then they won't be screwed because they didn't designed it for an ancient legacy storage, with lots of downsides, resulting is a worse product. Also a good way to differentiate the product from all the old competing backup solutions using differential backups that designed it with HDDs in mind.

 

[EllyMae]

So, I'm not a customer. Got it. This is not the first time I've been told in here to shut up. It's now a trend.

 

[Dunuin]

I didn't told you to shut up. If you are unhappy with the lack of HDD support, here is a good place to give the developers some feedback.
But we aren't doing this "SSDs are too expensive...PBS needs to work with HDDs..." conversation the first time. I'm just repeating what the staff explained in those other threads, why they designed it with SSDs in mind and even knowing that performance then will suck when using HDDs.
They discussed the pros and cons internally in the planning phase of the PBS and decided to focus on SSD. I doubt, they will change it, as this would probably mean doing all the work twice, as this should require fundamental changes. Like adding some Veeam-like differential backups using vdisks and skipping deduplication so the IO would be more sequential.

I personally hope they spend some more time on optimizations, so using HDDs for data + SSDs for metadata will perform a bit less worse.

 

[ness1602]

Nobody told you to shut up, just only that your logic doesn't make sense. All the compession,deduplication and others things are costly, and this cost is translated to IOPS and CPU. I have installed and maintained PBS nodes from 1-140TB, some pure hdd, some ssd metadata and some full ssd.
Every implementation is tailored for the guys paying,and what they expect. PBS is a great tool, albeit a little rigid, so there isn't any customization to say.

 

[EllyMae]

You say rigid. I say stubborn. It's easy to go 250 mph in a Farrari. I have a Honda and want to go 100. But the manufacurer has it governed at 50.

 

[ness1602]

You think you have honda,but it is rebranded trabant. If you have 100tb hdds, they work okay if you have 100 backups. But once you have 5k+ backups, you will see the pain.

 

[mauro70]

If I were to buy 25TB of SSDs, I would find better uses for them than backups. To me, your engineering equation doesn't make sense. You need to design a system that works well with inexpensive media.

in my view, your point generally speaking is fair, however I think that the discussion has diverged a bit from the original topic: what I tried to show was that backup options like "compression", which are pure CPU activity, might have not negligeable side effects even when your backup infrastructure has expensive SSD/NVME.
Performance of the backup process can be increased by acting on multiple single factors: CPUs, disks and on software side. The best solution will be surely achieved by getting the best of each single element, but optimal/satisfying solutions can be achieved also acting only on single elements. I focused my discussion on the software part, seen that disabling the compression in my old servers (with low-end CPUs) basically doubled the disk-write performances.
Cheers

 

[mauro2306]

in my view, your point generally speaking is fair, however I think that the discussion has diverged a bit from the original topic: what I tried to show was that backup options like "compression", which are pure CPU activity, might have not negligeable side effects even when your backup infrastructure has expensive SSD/NVME.
Performance of the backup process can be increased by acting on multiple single factors: CPUs, disks and on software side. The best solution will be surely achieved by getting the best of each single element, but optimal/satisfying solutions can be achieved also acting only on single elements. I focused my discussion on the software part, seen that disabling the compression in my old servers (with low-end CPUs) basically doubled the disk-write performances.
Cheers

hi there, i stumbled on your post searching why my backups were running that slow : https://forum.proxmox.com/threads/bad-backup-and-restore-performance.155108/ . Can i ask you how did you disabled compression ? I haven't found a way to do it, or maybe i did not understand what you were referring to with this. My opinion on it is that i get the statement about SSD replacing eventually HDD for the long shot, but still today the ratio price/storage is not in favor of SSD yet for most of the case, and for backup purpose, you may not need something that goes to multiple GB/s, most of the network link are 10Gbps and therefore i think that most use case would be satisfied with at least getting to that speed level, which is doable even with HDD when setting up a good RAIDZ infra. And about the argument of saying that compression save space, it also depends on what do you backup, not all data are even compressible, which then reduces the gain while you still process CPU around it.
So either PBS is really made for super high-end infra and it doesn't care about the lower-end one, or i think that some additional settings on removing (or at least adjusting) compression level and few other stuff will be required, because the cost for changing a server to a more powerful one in my case for example is way, way more expensive than adding more storage to compensate the absence of compression for example.

 

[mauro70]

hi there, i stumbled on your post searching why my backups were running that slow : https://forum.proxmox.com/threads/bad-backup-and-restore-performance.155108/ . Can i ask you how did you disabled compression ? I haven't found a way to do it, or maybe i did not understand what you were referring to with this. My opinion on it is that i get the statement about SSD replacing eventually HDD for the long shot, but still today the ratio price/storage is not in favor of SSD yet for most of the case, and for backup purpose, you may not need something that goes to multiple GB/s, most of the network link are 10Gbps and therefore i think that most use case would be satisfied with at least getting to that speed level, which is doable even with HDD when setting up a good RAIDZ infra. And about the argument of saying that compression save space, it also depends on what do you backup, not all data are even compressible, which then reduces the gain while you still process CPU around it.
So either PBS is really made for super high-end infra and it doesn't care about the lower-end one, or i think that some additional settings on removing (or at least adjusting) compression level and few other stuff will be required, because the cost for changing a server to a more powerful one in my case for example is way, way more expensive than adding more storage to compensate the absence of compression for example.

Hi, delpiero3,
>> because the cost for changing a server to a more powerful one in my case for example is way, way more expensive [...]
happy to read that, as I fully agree with your view point, it is exactly what I have always said since the beginning.

PBS does not have any way to disable compression or any other option.
I came to that conclusion by comparing the performances of PBS with the performance of the backup executed by PVE through VZDUMP.
When executed by PVE through VZDUMP, you can select which options to enable, including compression (none, LZO, GZIP, ZSTD)
Of course, the underlying storage used by PBS and by PVE was the same, otherwise the comparison would be fake.
And basically I discovered that the performance of the backup executed by PBS was comparable with the backup performed by PVE through VZDUMP with compression enabled. Instead, the performance of the backup performed by PVE through VZDUMP **without** compression was almost the double.
It was evident that, in my server, the compression was hurting the CPU and that was negatively affecting the overall performance of PBS, irrelevant if the storage was inexpensive SSD, enterprise SSD or even standard spinning disks.

I have raised an enhancement request to add to PBS the feature to enable/disable compression or any other option like it is possible to do with VZDUMP --> https://bugzilla.proxmox.com/show_bug.cgi?id=4900. As you can see, the request has been accepted, but nobody seems working on it, maybe it is not considered sufficiently important.

Cheers

 

[mauro2306]

Hi, delpiero3,
>> because the cost for changing a server to a more powerful one in my case for example is way, way more expensive [...]
happy to read that, as I fully agree with your view point, it is exactly what I have always said since the beginning.

PBS does not have any way to disable compression or any other option.
I came to that conclusion by comparing the performances of PBS with the performance of the backup executed by PVE through VZDUMP.
When executed by PVE through VZDUMP, you can select which options to enable, including compression (none, LZO, GZIP, ZSTD)
Of course, the underlying storage used by PBS and by PVE was the same, otherwise the comparison would be fake.
And basically I discovered that the performance of the backup executed by PBS was comparable with the backup performed by PVE through VZDUMP with compression enabled. Instead, the performance of the backup performed by PVE through VZDUMP **without** compression was almost the double.
It was evident that, in my server, the compression was hurting the CPU and that was negatively affecting the overall performance of PBS, irrelevant if the storage was inexpensive SSD, enterprise SSD or even standard spinning disks.

I have raised an enhancement request to add to PBS the feature to enable/disable compression or any other option like it is possible to do with VZDUMP --> https://bugzilla.proxmox.com/show_bug.cgi?id=4900. As you can see, the request has been accepted, but nobody seems working on it, maybe it is not considered sufficiently important.

Cheers

Sad indeed, i really hope that this will be taken in one of the next upgrade. I do really think that most of the use case is not the mission critical restore job within a couple of minutes. I don't think that most of the customers except to have backup machines that are as powerful as the one running their VMs (so to speak of course).

 

[g4sho]

Sad indeed, i really hope that this will be taken in one of the next upgrade. I do really think that most of the use case is not the mission critical restore job within a couple of minutes. I don't think that most of the customers except to have backup machines that are as powerful as the one running their VMs (so to speak of course).

Mine is and it is a home project, and the very reason I am reading this is because the backups from nvme to PSB nvme datatstore are slow.(even when I run PSB as a VM on same host)And from Vm on nvme to nvme without compression are 8 times as fast. 100MBs compared to 1GBps. My observation is if performance is what you want from backup, then you are better off not using Proxmox backup server data stores. I will see what Smb and Nfs shares performance will be over 40Gbe ethernet, for backup.

 

[g4sho]

Getting over 1GB per second backing up to SMB synology share from 1 NVME drive to 12- 3tb sas2 hard drives in a synologyVM raid 6. With no compression. Took 4 min 29 seconds to backup 200GB Vm and 11 min to restore same vm back to Host from Synology Vm.