Hi!
we are experiencing a consistent restore performance limit of about 2.5 - 3 GB/s when restoring from PBS to PVE.
TL;DR: Restore speed from PBS to PVE is capped, independent of any configurations. CPU is not saturated and we cannot identify the bottleneck.
Environment
Observations
Benchmarks (Micron and Kioxia are almost the same)
Uploaded 1390 chunks in 5 seconds.
Time per request: 3605 microseconds.
TLS speed: 1163.44 MB/s
SHA256 speed: 2170.14 MB/s
Compression speed: 617.12 MB/s
Decompress speed: 824.37 MB/s
AES256/GCM speed: 9911.01 MB/s
Verify speed: 605.93 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1163.44 MB/s (94%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2170.14 MB/s (107%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed │ 617.12 MB/s (82%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed │ 824.37 MB/s (69%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed │ 605.93 MB/s (80%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed │ 9911.01 MB/s (272%) │
└───────────────────────────────────┴─────────────────────┘
Has anyone managed to increase the restore speed yet?
I appreciate any support and any advice.
Thanks and best wishes,
McShadow
we are experiencing a consistent restore performance limit of about 2.5 - 3 GB/s when restoring from PBS to PVE.
TL;DR: Restore speed from PBS to PVE is capped, independent of any configurations. CPU is not saturated and we cannot identify the bottleneck.
Environment
- Fast system overall
- 6x Micron NVMes (7500 PRO)
- 6x Kioxia NVMes (CD8P-R)
- AMD EPYC 9174F Processor
- 12x 16GB DDR5 4800MHz
- PVE 9.1 image installed on hardware
- Proxmox BS 4.1.4 software installed on PVE 9.1
- There is no virtualization layer involved. Everything runs on bare metal.
Observations
- 6 NVMe drives in a stripe (Micron and Kioxia) -> restore speed stays the same.
- 12 NVMe drives in a stripe -> restore speed stays the same.
- In 'top': CPU load appears evenly distributed across all cores, with no significant 'wa' spikes.
- Also tried giving PVE more threads in /etc/environment - no changes at all.
- Performance does not improve when PBS storage and the destination restore volume are placed on different disks.
Benchmarks (Micron and Kioxia are almost the same)
Uploaded 1390 chunks in 5 seconds.
Time per request: 3605 microseconds.
TLS speed: 1163.44 MB/s
SHA256 speed: 2170.14 MB/s
Compression speed: 617.12 MB/s
Decompress speed: 824.37 MB/s
AES256/GCM speed: 9911.01 MB/s
Verify speed: 605.93 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1163.44 MB/s (94%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2170.14 MB/s (107%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed │ 617.12 MB/s (82%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed │ 824.37 MB/s (69%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed │ 605.93 MB/s (80%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed │ 9911.01 MB/s (272%) │
└───────────────────────────────────┴─────────────────────┘
Has anyone managed to increase the restore speed yet?
I appreciate any support and any advice.
Thanks and best wishes,
McShadow