PBS parallel backups & PBS performance

[dompro]

Maybe my FC connection to Pure is, for whatever reason, not as fast as I thought it to be. What would be a good test on the datastore which makes it compareable to how PBS writes data?

Also in the meantime I ran some tests with a virtual PBS on PVE. Configured with Hosts CPU. Same datastore (FC LUN) I get those results:

Uploaded 898 chunks in 5 seconds.
Time per request: 5577 microseconds.
TLS speed: 752.02 MB/s
SHA256 speed: 458.16 MB/s
Compression speed: 452.69 MB/s
Decompress speed: 730.08 MB/s
AES256/GCM speed: 3343.92 MB/s
Verify speed: 277.85 MB/s

The improved TLS and AES throughput doesn't change much on the backup speed.

Before

INFO: transferred 200.00 GiB in 392 seconds (522.4 MiB/s)

After

INFO: transferred 200.00 GiB in 352 seconds (581.8 MiB/s)

PS: The calculated throughput in the task-information is "fake" right? Because he calculates the empty space on the disk into the throughput. Tasklog attached in case anyone is curious. I'm always picking the maximum write throughput as performance indicator.

 

[dompro]

Ok, I've finished preparing the desktop PC. Core Ultra 5 235. Samsung BM9C1 NVMe.

Local Benchmark results:

proxmox-backup-client benchmark --repository local-nvme
Uploaded 1815 chunks in 5 seconds.
Time per request: 2758 microseconds.
TLS speed: 1520.56 MB/s
SHA256 speed: 4683.31 MB/s
Compression speed: 693.29 MB/s
Decompress speed: 999.11 MB/s
AES256/GCM speed: 8647.78 MB/s
Verify speed: 834.23 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1520.56 MB/s (123%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 4683.31 MB/s (232%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 693.29 MB/s (92%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 999.11 MB/s (83%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 834.23 MB/s (110%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 8647.78 MB/s (237%) │
└───────────────────────────────────┴─────────────────────┘

SHA checksum got quite the bump from SHA-NI.

Benchmark on PVE:

proxmox-backup-client benchmark --repository 'x:8007:local-nvme'
Uploaded 1134 chunks in 5 seconds.
Time per request: 4439 microseconds.
TLS speed: 944.87 MB/s
SHA256 speed: 463.29 MB/s
Compression speed: 465.59 MB/s
Decompress speed: 773.36 MB/s
AES256/GCM speed: 3357.92 MB/s
Verify speed: 287.38 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 944.87 MB/s (77%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 463.29 MB/s (23%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 465.59 MB/s (62%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 773.36 MB/s (65%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 287.38 MB/s (38%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 3357.92 MB/s (92%) │
└───────────────────────────────────┴────────────────────┘

Testbackup is about the same speed like before.

INFO: transferred 200.00 GiB in 349 seconds (586.8 MiB/s)

Soooo. What does that tell me? That I need to replace my 12 Cluster Nodes with servers that have newer CPUs?

Edit: I'm a bit slow in the head. I can just install PVE on the same desktop and see if performance improves. Will report back.

 

[dompro]

Desktop PVE to my Testing-PBS:

INFO:  92% (352.0 GiB of 382.0 GiB) in 51s, read: 2.0 GiB/s, write: 494.7 MiB/s
INFO:  93% (355.3 GiB of 382.0 GiB) in 56s, read: 679.2 MiB/s, write: 679.2 MiB/s
INFO:  94% (359.2 GiB of 382.0 GiB) in 1m 2s, read: 670.7 MiB/s, write: 670.7 MiB/s
INFO:  95% (363.2 GiB of 382.0 GiB) in 1m 9s, read: 576.6 MiB/s, write: 576.6 MiB/s
INFO:  96% (367.0 GiB of 382.0 GiB) in 1m 16s, read: 558.3 MiB/s, write: 558.3 MiB/s
INFO:  97% (370.9 GiB of 382.0 GiB) in 1m 22s, read: 666.7 MiB/s, write: 566.7 MiB/s
INFO:  98% (376.4 GiB of 382.0 GiB) in 1m 26s, read: 1.4 GiB/s, write: 396.0 MiB/s
INFO: 100% (382.0 GiB of 382.0 GiB) in 1m 29s, read: 1.9 GiB/s, write: 212.2 MiB/s

So yea, seems like the CPU in the PVE Hosts is the culprit. Somehow kinda sad, up until now I considered my PVE Hosts with those 6226R CPUs beefy.