PBS Backups slow

Kiyo

Member
Jul 28, 2020
14
1
8
24
LXC Backups from my spinning rust RAID1 to a spinning rust RAID10 are ridiculously slow.
(CT 200)
Code:
INFO: root.pxar: had to backup 9.35 GiB of 5.071 TiB (compressed 8.186 GiB) in 19638.29s
INFO: root.pxar: average backup speed: 499.249 KiB/s
INFO: root.pxar: backup was done incrementally, reused 5.062 TiB (99.8%)
INFO: Uploaded backup catalog (10.711 MiB)
INFO: Duration: 19649.61s

Some are faster,
(CT 203)
Code:
INFO: root.pxar: had to backup 455.973 MiB of 2.758 GiB (compressed 181.175 MiB) in 9.15s
INFO: root.pxar: average backup speed: 49.832 MiB/s
INFO: root.pxar: backup was done incrementally, reused 2.313 GiB (83.9%)
INFO: Uploaded backup catalog (515.036 KiB)
INFO: Duration: 9.85s

(CT 205)
Code:
INFO: root.pxar: had to backup 743.333 MiB of 398.554 GiB (compressed 277.813 MiB) in 3662.18s
INFO: root.pxar: average backup speed: 207.847 KiB/s
INFO: root.pxar: backup was done incrementally, reused 397.828 GiB (99.8%)
INFO: Uploaded backup catalog (14.389 MiB)
INFO: Duration: 3663.05s


Is this simply because of the size of the containers?

Running a benchmark from my PVE host to the PBS box seems promising,

Code:
root@Ritsu:~# proxmox-backup-client benchmark --repository 192.168.0.253:RAID10
Uploaded 148 chunks in 5 seconds.
Time per request: 35671 microseconds.
TLS speed: 117.58 MB/s
SHA256 speed: 2163.67 MB/s
Compression speed: 832.12 MB/s
Decompress speed: 1021.85 MB/s
AES256/GCM speed: 2431.79 MB/s
Verify speed: 695.84 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 117.58 MB/s (10%)   │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2163.67 MB/s (107%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 832.12 MB/s (111%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1021.85 MB/s (85%)  │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 695.84 MB/s (92%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2431.79 MB/s (67%)  │
└───────────────────────────────────┴─────────────────────┘

One final note is that the boot disks of the PBS are incredibly slow, as they are small flash DOMs, but since it's just the boot disk this shouldn't affect backup speed?
 
Last edited:
I guess there are many small files in the container? Reading them just takes a long time because the RAID is slow.
 
the average backup speed is referring to the transfer of (new/changed) data, but the duration includes the time needed to read and process *all* data. if the incremental data is very small (good! lots of deduplication), but the total data (which needs to be read and hashed to determine that it is okay to skip) is big, the average speed will look very low. this doesn't mean that anything is wrong with either side or the network, just that very little data needed to actually be transferred, and it took a long time to determine this fact. since we read (and skip) and transfer in one go, we cannot really disentangle the time spent (and thus the speed) of both parts of the backup.

if you want to determine the theoretical max speed, you can just create a backup using all new (possibly random) data, so that all chunks need to be uploaded and processed and stored by the PBS server. the benchmark command does something similar, but a bit simplified - it will upload chunks, but the server doesn't actually store them, so it's purely testing throughput on the network layer.
 
Running a benchmark from my PVE host to the PBS box seems promising,

Code:
root@Ritsu:~# proxmox-backup-client benchmark --repository 192.168.0.253:RAID10
Uploaded 148 chunks in 5 seconds.
Your Storage seems very slow...
For comparing, an old HP ML10v2:
Code:
Uploaded 1050 chunks in 5 seconds.
 
Last edited:
Your Storage seems very slow...
For comparing, an old HP ML10v2:
Code:
Uploaded 1050 chunks in 5 seconds.
note that this benchmark doesn't in any way touch storage on either side - it's purely a measurement of

- TLS performance (if the required crypto is slow, data cannot transfer faster)
- HTTP2 performance (mainly affected by high latency or latency fluctuations)
- network performance (obviously ;))

a bottle neck on either of these three levels will show clearly in the benchmark. the crypto performance of the client side looks okay, so either the server is bad at crypto or the network is at fault. it could simply be a Gbit line that just can't transfer more data, for example.
 
the average backup speed is referring to the transfer of (new/changed) data, but the duration includes the time needed to read and process *all* data. if the incremental data is very small (good! lots of deduplication), but the total data (which needs to be read and hashed to determine that it is okay to skip) is big, the average speed will look very low. this doesn't mean that anything is wrong with either side or the network, just that very little data needed to actually be transferred, and it took a long time to determine this fact. since we read (and skip) and transfer in one go, we cannot really disentangle the time spent (and thus the speed) of both parts of the backup.

if you want to determine the theoretical max speed, you can just create a backup using all new (possibly random) data, so that all chunks need to be uploaded and processed and stored by the PBS server. the benchmark command does something similar, but a bit simplified - it will upload chunks, but the server doesn't actually store them, so it's purely testing throughput on the network layer.
I see, makes sense,

It is a lot of files, but generally large ones, so would make sense that it takes a long time

I might have to reconsider backing up all that data then, it's all mostly replaceable :p

Danke!
 
note that this benchmark doesn't in any way touch storage on either side - it's purely a measurement of

- TLS performance (if the required crypto is slow, data cannot transfer faster)
- HTTP2 performance (mainly affected by high latency or latency fluctuations)
- network performance (obviously ;))

a bottle neck on either of these three levels will show clearly in the benchmark. the crypto performance of the client side looks okay, so either the server is bad at crypto or the network is at fault. it could simply be a Gbit line that just can't transfer more data, for example.
PBS is most likely bad at crypto, it's an old Sandy Bridge i5 haha.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!