Improve verification process - change CPU

P

pszemek

New Member
Jan 20, 2024
11
0
1
Hello,

I'm in the process of tweaking the backup verification process and from what I see, quite large VMs (4TB and up) are problematic. Below is the output from the benchmark.

Code: 
# proxmox-backup-client benchmark --repository backup
Uploaded 388 chunks in 5 seconds.
Time per request: 12978 microseconds.
TLS speed: 323.16 MB/s
SHA256 speed: 212.31 MB/s
Compression speed: 253.78 MB/s
Decompress speed: 320.11 MB/s
AES256/GCM speed: 560.31 MB/s
Verify speed: 132.93 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 323.16 MB/s (26%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 212.31 MB/s (11%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 253.78 MB/s (34%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 320.11 MB/s (27%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 132.93 MB/s (18%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 560.31 MB/s (15%) │
└───────────────────────────────────┴───────────────────┘

Can you suggest me whether I should go for a more powerful, single core or look for a problem on the IO side? My setup is as follows:

CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
RAM: 256GB
Storage: HW RAID6, 16x16TB + 2x1.8T NVMe for special device on ZFS
 
pszemek said:
Can you suggest me whether I should go for a more powerful, single core or look for a problem on the IO side? My setup is as follows:
Click to expand...

How does I/O delay look like during verification? That should give an indication as to whether I/O or the CPU is the bottleneck.
 
Could you post the graph of the CPU usage during the verification process? That would make it easier to gauge.
 
pszemek said:
Storage: HW RAID6, 16x16TB + 2x1.8T NVMe for special device on ZFS
Click to expand...
After re-reading your post: Do I understand correctly that you are using ZFS on top of a hardware RAID?
 
Using ZFS on top of HW RAID is strongly discouraged and can lead to a multitude of issues, you can read more about it in the OpenZFS wiki [1]. I'd recommend checking whether your RAID controller has the possibility of running in HBA mode when you want to use ZFS, or use a filesystem other than ZFS if you want to keep using HW RAID.

Also it seems like your CPU does not yet support the Intel SHA Extensions, which means it will perform sub-par compared to even slightly newer CPUs with verifications jobs. Your speed is 132.93 MB/s and I'd assume that this would bottleneck the storage. But one would have to test I/O via fio for instance.

In order to really gauge I'd recommend getting the storage setup in order (either ZFS or HW Raid - not both) and then running a benchmark on the speed of the storage to see how fast it is. Then compare that to the chunk verification speed and see what is the bottleneck with your setup.

[1] https://openzfs.github.io/openzfs-docs/Performance and Tuning/Hardware.html#hardware-raid-controllers
 
Hello,

We change processor on older server but results wasn't be satisfactly. Time for verify backup, copy data from virtualizator to backup system didn't reduced. We must change backup politic and change server for create backup on '1st level'.

@shanreich i've a question. Are you have any good document about tweak for more performance ZFS on SSD/NVMe for PBS? We replaced the server and while the backup time decreased significantly, I noticed that the number of IOPS for any operations did not increase.
 
What kind of SSDs/NVMEs are you using and which layout (RAID10, RAIDZ, ...)? We recommend using RAID10 over RAIDZn, since RAIDZn does not provide additional IOPS when adding additional disks. Asides from recommendations regarding disk layout, we do not recommend any particular ZFS configuration performance tweaks - with proper disks and disk layout the performance of ZFS out-of-the-box is already quite good. Only with HDDs you can add a special device for instance, but this only makes sense when using HDDs.
 
  • Like
Reactions: pszemek
I am also suffering poor backup performance but am running powerful, enterprise hardware with all flash storage in hardware RAID 10...

I beleive the 'chunk verification' is the bottleneck. But these are modern Xeon Platinum's in my hosts....


┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 494.71 MB/s (24%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 539.17 MB/s (72%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 850.95 MB/s (71%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 305.07 MB/s (40%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1715.09 MB/s (47%) │
└───────────────────────────────────┴────────────────────┘

https://forum.proxmox.com/threads/pbs-performance-improvements-enterprise-all-flash.150514/
 
yes - hence the "and follow-up work". we've discussed internally how to remove bottle necks for systems where more parallelism would actually cause improved performance (because for most hardware PBS is running on, the inverse would unfortunately be true -> too much parallelism causes starvation and things get slower).
 
fabian said:
yes - hence the "and follow-up work". we've discussed internally how to remove bottle necks for systems where more parallelism would actually cause improved performance (because for most hardware PBS is running on, the inverse would unfortunately be true -> too much parallelism causes starvation and things get slower).
Click to expand...
Well thats good to hear, I will keep a close eye.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom