how to improve pbs backup speed

[juancarlo2596]

Hi, I have 2 servers running Proxmox. One is for production and one is for backup. Backup server has PBS installed. However backing up to pbs seems to be quite slow even with dedicated 10Gbpbs network. I also have a TrueNAS installed on the same server and backup from TrueNAS is way more faster than pbs.

pbs benchmark:

root@proxmox02:~# proxmox-backup-client benchmark --repository root@pam@172.254.254.20bsDrive
Password for "root@pam":
Uploaded 154 chunks in 5 seconds.
Time per request: 33459 microseconds.
TLS speed: 125.35 MB/s
SHA256 speed: 283.37 MB/s
Compression speed: 301.50 MB/s
Decompress speed: 268.80 MB/s
AES256/GCM speed: 581.42 MB/s
Verify speed: 160.91 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 125.35 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 283.37 MB/s (14%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 301.50 MB/s (40%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 268.80 MB/s (22%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 160.91 MB/s (21%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 581.42 MB/s (16%) │
└───────────────────────────────────┴───────────────────┘

PBS vm specs : 12 (2 sockets, 6 cores) [host.flags=+aes] [numa=1], 32GB RAM, 32GB for OS (On SSD), 6TB for Datastore (On HDD), dedicated 10Gb Network

host specs:
Dell PowerEdge T620
24 x Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz (2 Sockets)
200GB RAM
2x Samsung 870 Evo SSD 1TB connected on SATA Port (SW RAID 1 for OS)
8x 6TB WD Enterprise HDD connected on PERC H710 (HW RAID 6 for storage)

There are only 2 running VMs inside the host (PBS and TrueNAS)

Any way to improve my pbs backup speed?

 

[Dunuin]

Are you running PBS in a VM, LXC or bare metal next to PVE on the same machine? In case of a VM what config does that VM use? Stuff like kvm64 and E1000 are notoriously slow.

 

[sb-jw]

Are you running PBS in a VM, LXV or bare metal next to PVE on the same machine?

PBS vm specs : 12 (2 sockets, 6 cores) [host.flags=+aes] [numa=1], 32GB RAM, 32GB for OS (On SSD), 6TB for Datastore (On HDD), dedicated 10Gb Network

I would say it's a VM!

I also have a TrueNAS installed on the same server and backup from TrueNAS is way more faster than pbs.

A dull full backup to an NFS share cannot necessarily be compared with the intelligent handling of the backups. Often there are caches in between that are activated because of async, which trick your node into thinking that the data has been written, but in reality it is still being copied in the background. If you then switch off your node immediately afterwards, this could lead to a defective backup.

In this respect, you should always look at the protocols, caches and handling of the solutions in detail and not make 1:1 comparisons.

From my experience I can say that in terms of bandwidth utilization, backing up on PBS is significantly slower than before on a CephFS, but the real backup time has been drastically shortened. Where before I usually had 20 - 30 minutes per node, today it is sometimes only 8 minutes.

This also shows that bandwidth isn't everything. The incremental backup alone saves a lot of time.

8x 6TB WD Enterprise HDD connected on PERC H710 (HW RAID 6 for storage)

Which models do you use? How is the RAID configured?

 

[juancarlo2596]

Are you running PBS in a VM, LXC or bare metal next to PVE on the same machine? In case of a VM what config does that VM use? Stuff like kvm64 and E1000 are notoriously slow.

pbs is running on a different server, separated on my production server. screenshot of vm config attached

 

[juancarlo2596]

I would say it's a VM!


A dull full backup to an NFS share cannot necessarily be compared with the intelligent handling of the backups. Often there are caches in between that are activated because of async, which trick your node into thinking that the data has been written, but in reality it is still being copied in the background. If you then switch off your node immediately afterwards, this could lead to a defective backup.

In this respect, you should always look at the protocols, caches and handling of the solutions in detail and not make 1:1 comparisons.

From my experience I can say that in terms of bandwidth utilization, backing up on PBS is significantly slower than before on a CephFS, but the real backup time has been drastically shortened. Where before I usually had 20 - 30 minutes per node, today it is sometimes only 8 minutes.

This also shows that bandwidth isn't everything. The incremental backup alone saves a lot of time.


Which models do you use? How is the RAID configured?

1. Yes, it's a VM. Maybe a little change of config would help?

2. So those percentages from the NFS backups does not mean it's already being copied to the NFS Share? Like saving it temporarily then will transfer afterwards?

3. WD6003FRYZ WD Gold Enterprise , RAID is configured in RAID6

 

[sb-jw]

2. So those percentages from the NFS backups does not mean it's already being copied to the NFS Share? Like saving it temporarily then will transfer afterwards?

The current NFS versions have async as standard. Whether it is actually the case that the data has not yet been transferred or has already been transferred but is cached on the target ultimately depends on the configuration and the implementation used. In this respect, my statement is more generic and less specific to your application.
But you should also keep the topic of sync and async in mind, because not everything is as it seems

 

[Dunuin]

And also keep in mind that most stuff (hashing/compression/encryption) is done client-side when creating backups. So those PVE nodes need beefy performance too. You don't have that stuff when writing directly to a NFS share.

 

[Neobin]

how to improve pbs backup speed

Do not use:

• 12 year old hardware: [1], that is slower than a current 7W TDP CPU: [2]
• HDDs at all (on the source as well as the target)
• Parity raid, but e.g.: raid10
• (Virtualized PBS with all the additional layers in between, especially regarding the vDisk(s))

[1] https://ark.intel.com/content/www/u...0-15m-cache-2-30-ghz-7-20-gt-s-intel-qpi.html
[2] https://www.cpubenchmark.net/compare/1215vs5602