PBS Performance

C

ChrisJM

Well-Known Member
Mar 12, 2018
67
2
48
39
Hello,

I'm running PBS on RAID Z2 over 4x Enterprise SATA with 50GBps Private Bandwidth.

My actual hosts are 256GB RAM with 64 Cores

I have noticed the restores are pretty slow on the decompression - is there a way to increase the performance on them?

Also where is the resources being used for the restore / backup? is it on the host or on the PBS server?

also note that the I/O doesnt seem to be the issue and doesnt seem to be the current bottleneck.

I have read other posts which are a few years old but doesnt really give an answer on increasing performance.

Kind Regards,
Chris
 
Are you using HDDs? Then you have your answer. RAIDz instead of mirror is also optimal for performance.
To speedup things with HDDs add two small ( 2% of your vpool size) enterprise- SSDs in a mirror as a special device to the zfs Pool.
Especially garbage collection should Profit a lot from that.
Please note that you will need to rewrite the data with zfs send/receive otherwise the special device will only help for New data.
 
Last edited:
Which RAID would you recommend for the ZFS - we need the HDD for the space.

this the benchmark

Uploaded 1327 chunks in 5 seconds.
Time per request: 3771 microseconds.
TLS speed: 1112.06 MB/s
SHA256 speed: 1873.80 MB/s
Compression speed: 563.47 MB/s
Decompress speed: 739.41 MB/s
AES256/GCM speed: 4169.04 MB/s
Verify speed: 527.82 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1112.06 MB/s (90%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 1873.80 MB/s (93%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed │ 563.47 MB/s (75%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed │ 739.41 MB/s (62%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed │ 527.82 MB/s (70%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed │ 4169.04 MB/s (114%) │
 
At the moment we are running PBS on top of proxmox 8 then i do the ZFS and make separate virtual disks per datastore - i do this as we have separate clusters and want backups secure and not go between clusters


What if i put the PBS on baremetal and then make a single ZFS within PBS - can i make separate storage so not each cluster can see each others backups?
 
ChrisJM said:
At the moment we are running PBS on top of proxmox 8 then i do the ZFS and make separate virtual disks per datastore - i do this as we have separate clusters and want backups secure and not go between clusters


What if i put the PBS on baremetal and then make a single ZFS within PBS - can i make separate storage so not each cluster can see each others backups?
Click to expand...
Yes that would be possible with namespaces and api-tokens or technical users/logins for every cluster : https://pbs.proxmox.com/docs/storage.html#backup-namespaces
You could also create a custom datastore for each cluster. Putting everything in one big datastore would however result in a higher deduplication ratio thus saving space though.
I think however that PBS as a VM instead of bare metal shouldn't have that much of an impact compared to the choice of HDD without ssd special devices for storage ( I'm quite sure that they are the main issue, everything else (be it the setup as VM or even the RAID layout) propably also impact things but not that much but the issue of limited IOPS with HDDs).
Now the PBS manual recommends against running PBS as a VM or paaralel to PVE on the same host but the main reason for that isn't so much the performance impact but the consequences for a disaster recoevery scenario: If your PBS is on the same machine and with the same storage as your production vms and data you might loose both in case of disaster. You also would need to first restore the PBS before actually starting the remaining restore operations.
 
Last edited:
  • Like
Reactions: waltar and _gabriel
Thank you, the PBS on top of proxmox is a completely different host anyway and is for the backup server.

the main reason why thinking to go direct to bare metal is the network as the virtio network device has a limit of 10Gbps
 
  • Like
Reactions: Johannes S
We are doing some troubleshooting and discussion in this thread:
D

Thread 'Abysmally slow restore from backup'

Hello,

My woes with my HP Microserver Gen8 continue (here's my previous thread, now thankfully solved... mostly), as I'm now getting terrible performance with backups.

I'm restoring backups to lvm-data that's located on a 7200rpm HDD. The SATA controller is this:

Code: 
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)

The backups are stored on PBS connected with gigabit ethernet. The restore starts fine, with write performance around 20-30 MB/s, which is what I'd expect. After a few minutes, write speed...
  • Like

I'm getting results from Veeam at the moment, already did my PBS testing. Seems like PBS is about 4-6x slower restoring than backing up.
 
i dont really get it.

I can backup up pretty fast on a fresh backup, but a restore will take hours - this make no sense
 
ChrisJM said:
I can backup up pretty fast on a fresh backup, but a restore will take hours - this make no sense
Click to expand...
For a backup only modified data is transferred to the PBS and written to disk. Usually that's only a small portion of the whole disk.

For a restore each and every "chunk" of the image has to be read on the PBS. (And all bytes have to be transferred through the network.) That is much slower and takes a lot of time...
 
  • Like
Reactions: Johannes S and waltar
UdoB said:
For a backup only modified data is transferred to the PBS and written to disk. Usually that's only a small portion of the whole disk.

For a restore each and every "chunk" of the image has to be read on the PBS. (And all bytes have to be transferred through the network.) That is much slower and takes a lot of time...
Click to expand...

As i said i can do a fresh backup with no backup before it and its fine and fast, restores are really slow. where does it do the decompression?
 
I got a cache working, dont think its making any difference what so ever.

where does the decompression happen? on the host or on the pbs?
 
It must do otherwise the CPU on the host wouldnt spike when doing a backup..

also it still makes no sense to why a fresh full backup can be done in an hour but a restore can take 4 hours?

if anything a write I/O is an issue, when i copy a file from the PBS to the host its very very fast, so why is the restore so bad on PBS?
 
ChrisJM said:
where does the decompression happen? on the host or on the pbs?
Click to expand...

Usually backups are encrypted. By the source! The destination-PBS shall not be able to decrypt this data = only the source-PVE knows the secret key.

Encrypted data is NOT compressible. (Perhaps a few percent, but that does not count here.) By definition encrypted data "looks like white noise".

So any compression must happen on the PVE - before encryption.

I am not sure where chunking actually happens in this chain. But... I can see the actual chunks are very dynamic in size - from a few hundred bytes up to 4 MB. This tells me that chunking happens before compression, basically as step zero.

Disclaimer: effectively just guessing - please correct me if I'm wrong...
 
JensF said:
Not by default. It can be activated but it's not standard.
Click to expand...
Sure. But is defines the sequence of operations. Even if disabled / not used...
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom