How fast is your backup datastore? Benchmark tool

joace · Oct 27, 2022

Here is my benchmark on newly setup PBS (test locally on the datastores)
PBS config (on a HPE DL380 Gen10 unit):

Xeon Gold 5117 14c/28t 2.00GHz x 2
128GB DDR4 RDIMM 2400MHz
2 datastores:
- SAS SSD 1.6TB x 16 (8 mirror vdev, this will be official datastore for backup)
- NVMe 960GB x 2 in mirror (for testing purpose)

Benchmark result:

On SAS SSD datastore

Uploaded 935 chunks in 5 seconds.
Time per request: 5372 microseconds.
TLS speed: 780.75 MB/s
SHA256 speed: 360.64 MB/s
Compression speed: 365.17 MB/s
Decompress speed: 484.88 MB/s
AES256/GCM speed: 1054.89 MB/s
Verify speed: 206.40 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 780.75 MB/s (63%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 360.64 MB/s (18%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 365.17 MB/s (49%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 484.88 MB/s (40%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 206.40 MB/s (27%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1054.89 MB/s (29%) │
└───────────────────────────────────┴────────────────────┘

On NVMe datastore

Uploaded 908 chunks in 5 seconds.
Time per request: 5514 microseconds.
TLS speed: 760.56 MB/s
SHA256 speed: 360.83 MB/s
Compression speed: 367.37 MB/s
Decompress speed: 482.57 MB/s
AES256/GCM speed: 1058.30 MB/s
Verify speed: 206.64 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 760.56 MB/s (62%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 360.83 MB/s (18%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 367.37 MB/s (49%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 482.57 MB/s (40%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 206.64 MB/s (27%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1058.30 MB/s (29%) │
└───────────────────────────────────┴────────────────────┘

Seems both are not getting nice number with these SSD drives, especially for NVMe datastore like other guys posted result here, not sure whether this is because of low processor speed, and whether PBS is single thread, so dual socket is not really helpful here. Can anybody share some ideas to help me improve the performance? Thanks!

Toranaga · Oct 27, 2022

Not fast, but working:

Supermicro X11SBA-LN4F, Intel N3700, 16GB, WD RED Pro 14 TB, PBS Datastore ZFS on luks

Uploaded 52 chunks in 5 seconds.
Time per request: 111703 microseconds.
TLS speed: 37.55 MB/s
SHA256 speed: 243.08 MB/s
Compression speed: 221.92 MB/s
Decompress speed: 291.81 MB/s
AES256/GCM speed: 745.87 MB/s
Verify speed: 125.93 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 37.55 MB/s (3%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 243.08 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 221.92 MB/s (30%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 291.81 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 125.93 MB/s (17%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 745.87 MB/s (20%) │
└───────────────────────────────────┴───────────────────┘

Felix. · Oct 29, 2022

Toranaga said:
PBS Datastore ZFS on luks

So you store your encrypted chunks on an encrypted ZFS pool? Why?

Toranaga · Oct 29, 2022

No, as far as I know is zfs not full crypted, some data stay uncrypted.
So I do not use the crypt part of zfs.

Dunuin · Oct 29, 2022

Toranaga said:
No, as far as I know is zfs not full crypted, some data stay uncrypted.
So I do not use the crypt part of zfs.

Jup, still the question why you don't simple use the encryption that PBS offers on the software level. Then you don't need t encrypt you filesystems.

Toranaga · Oct 29, 2022

I use the target disk for more than pbs
1. The disk is used as rsync target for backups that run to long to do it with pbs. These data needs to be crypted too.
2. The disk is datastore for the emergency pve installed parallel to the pbs, thats the reason for zfs instead of lvm.
3. I see problems with the crash recovery if the pve is complete gone? where are the keys to restore the vms? Maybe incomplete informations on my side. The passphrase of luks is entered by hand on boot, this works for me. Crash recovery needs only the pure disk with the datastore of the pbs.

Dunuin · Oct 29, 2022

Toranaga said:
3. I see problems with the crash recovery if the pve is complete gone? where are the keys to restore the vms? Maybe incomplete informations on my side. The passphrase of luks is entered by hand on boot, this works for me. Crash recovery needs only the pure disk with the datastore of the pbs.

You save the key when enabling encryption for the PBS datastore.

Felix. · Oct 30, 2022

Toranaga said:
I use the target disk for more than pbs
1. The disk is used as rsync target for backups that run to long to do it with pbs. These data needs to be crypted too.
2. The disk is datastore for the emergency pve installed parallel to the pbs, thats the reason for zfs instead of lvm.
3. I see problems with the crash recovery if the pve is complete gone? where are the keys to restore the vms? Maybe incomplete informations on my side. The passphrase of luks is entered by hand on boot, this works for me. Crash recovery needs only the pure disk with the datastore of the pbs.

If you did not download it from the UI, you can find the keys in /etc/pve/priv/storage.
Also you can enable the Master Key Feature, that way you only need to keep eyes on a single keypair: https://pve.proxmox.com/wiki/Storage:_Proxmox_Backup_Server#storage_pbs_encryption

Toranaga · Oct 30, 2022

Thanks for your help.
But I have reasons to do it with LUKS.

Robstarusa · Nov 2, 2022

Backup server is E3-1240v3 with 32G ram & sata ssds on 10Gbit/s Ethernet
VM Host is 5950x with 2.5Gbit/s ethernet & 3.84TB nvme ssd.

My biggest issue is occasional full verifies where the e3-1243v3 cpu goes to 400% for hours at a time. Typically 4-5 Hours.

Benchmark run from VM Host:

Uploaded 355 chunks in 5 seconds.
Time per request: 14438 microseconds.
TLS speed: 290.49 MB/s
SHA256 speed: 2397.32 MB/s
Compression speed: 924.12 MB/s
Decompress speed: 1209.97 MB/s
AES256/GCM speed: 2856.40 MB/s
Verify speed: 803.62 MB/s


┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 290.49 MB/s (24%)   │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2397.32 MB/s (119%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 924.12 MB/s (123%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1209.97 MB/s (101%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 803.62 MB/s (106%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2856.40 MB/s (78%)  │
└───────────────────────────────────┴─────────────────────┘

semihtore · Nov 3, 2022

Hi.

Help mee!!!!

very very very slow speed backup

I have 10Gb ethernet and 8Gb SAN Switch

Uploaded 43 chunks in 6 seconds.
Time per request: 159704 microseconds.
TLS speed: 26.26 MB/s
SHA256 speed: 215.65 MB/s
Compression speed: 268.07 MB/s
Decompress speed: 395.54 MB/s
AES256/GCM speed: 588.19 MB/s
Verify speed: 138.26 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 26.26 MB/s (2%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 215.65 MB/s (11%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 268.07 MB/s (36%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 395.54 MB/s (33%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 138.26 MB/s (18%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 588.19 MB/s (16%) │
└───────────────────────────────────┴───────────────────┘

And local disk performance for NFS : 260 MiB/s

please help me!!

liberodark · Nov 3, 2022

Result with a PBS VM

4x Cores
8 Go Ram
ZFS HDD (SATA Storage via Netapp NFS v3)

On PBS :

Code:

proxmox-backup-client benchmark --repository backup_c
Password for "root@pam": *****************
Uploaded 498 chunks in 5 seconds.
Time per request: 10073 microseconds.
TLS speed: 416.35 MB/s
SHA256 speed: 330.75 MB/s
Compression speed: 227.08 MB/s
Decompress speed: 270.28 MB/s
AES256/GCM speed: 530.00 MB/s
Verify speed: 147.19 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 416.35 MB/s (34%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 330.75 MB/s (16%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 227.08 MB/s (30%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 270.28 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 147.19 MB/s (19%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 530.00 MB/s (15%) │
└───────────────────────────────────┴───────────────────┘

On PVE :

Code:

proxmox-backup-client benchmark --repository root@pam@my_ip:backup_c
Password for "root@pam": *****************
Uploaded 148 chunks in 5 seconds.
Time per request: 35880 microseconds.
TLS speed: 116.90 MB/s
SHA256 speed: 306.85 MB/s
Compression speed: 307.21 MB/s
Decompress speed: 480.33 MB/s
AES256/GCM speed: 934.35 MB/s
Verify speed: 185.04 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 116.90 MB/s (9%)  │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 306.85 MB/s (15%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 307.21 MB/s (41%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 480.33 MB/s (40%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 185.04 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 934.35 MB/s (26%) │
└───────────────────────────────────┴───────────────────┘

VictorSTS · Nov 23, 2022

Hello,

I'm getting these numbers from a AMD Ryzen 7 PRO 3700 8-Core Processor:

Code:

SHA256 speed: 2114.85 MB/s
Compression speed: 593.17 MB/s
Decompress speed: 796.48 MB/s
AES256/GCM speed: 1866.55 MB/s
Verify speed: 588.18 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ not tested          │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2114.85 MB/s (105%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 593.17 MB/s (79%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 796.48 MB/s (66%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 588.18 MB/s (78%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 1866.55 MB/s (51%)  │
└───────────────────────────────────┴─────────────────────┘

I know that the "PRO" version is somewhat slower than the standard 3700x, but still these numbers seem to be too low, specially compared with the ones provided in the wiki: https://pve.proxmox.com/wiki/PBS_Client_CPU_Benchmark#AMD_Ryzen_7_3700X_8-Core_Processor

This server has a standalone PVE running a couple of very lightweight VMs (linux nameservers) and installed PBS alongside PVE. Anyone else have any data using Ryzen 7 CPUs? I would like to get some more numbers before I dwelve into the BIOS and other settings which may affect the performance.

Thanks!

Neobin · Nov 25, 2022

PBS: 2.2.7
Kernel: 5.19

AMD Ryzen 9 5950X:

Code:

┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ not tested          │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2454.96 MB/s (121%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 924.92 MB/s (123%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1203.11 MB/s (100%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 793.16 MB/s (105%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2813.84 MB/s (77%)  │
└───────────────────────────────────┴─────────────────────┘

AMD Ryzen 7 5700G:

Code:

┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ not tested          │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2268.14 MB/s (112%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 898.85 MB/s (120%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1147.84 MB/s (96%)  │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 773.07 MB/s (102%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2704.96 MB/s (74%)  │
└───────────────────────────────────┴─────────────────────┘

Intel Core i9-9900K:

Code:

┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested         │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 648.44 MB/s (32%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 690.37 MB/s (92%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 940.24 MB/s (78%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 381.77 MB/s (50%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2017.46 MB/s (55%) │
└───────────────────────────────────┴────────────────────┘

Intel Atom C2558:

Code:

┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ not tested        │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 116.25 MB/s (6%)  │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 144.09 MB/s (19%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 178.57 MB/s (15%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 69.40 MB/s (9%)   │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 215.83 MB/s (6%)  │
└───────────────────────────────────┴───────────────────┘

ITT · Nov 25, 2022

ITT said:

Xeon E3 1231 v3 / 32GB Ram, SSD

Schedutil:

Code:

Uploaded 936 chunks in 5 seconds.
Time per request: 5378 microseconds.
TLS speed: 779.85 MB/s
SHA256 speed: 444.46 MB/s
Compression speed: 434.62 MB/s
Decompress speed: 671.52 MB/s
AES256/GCM speed: 1268.67 MB/s
Verify speed: 264.32 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 779.85 MB/s (63%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 444.46 MB/s (22%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 434.62 MB/s (58%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 671.52 MB/s (56%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 264.32 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1268.67 MB/s (35%) │
└───────────────────────────────────┴────────────────────┘

After some time, same Hardware but switched to acpi_cpufreq:

Code:

Uploaded 1026 chunks in 5 seconds.
Time per request: 4884 microseconds.
TLS speed: 858.73 MB/s
SHA256 speed: 482.43 MB/s
Compression speed: 513.00 MB/s
Decompress speed: 819.40 MB/s
AES256/GCM speed: 1600.68 MB/s
Verify speed: 302.91 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 858.73 MB/s (70%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 482.43 MB/s (24%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 513.00 MB/s (68%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 819.40 MB/s (68%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 302.91 MB/s (40%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1600.68 MB/s (44%) │
└───────────────────────────────────┴────────────────────┘

I think Uploaded xxx chunks in 5 seconds. is very important (Disk-/Wirespeed)

t.lamprecht · Nov 25, 2022

ITT said:
I think Uploaded xxx chunks in 5 seconds. is very important (Disk-/Wirespeed)

It's also quite dependent of few other factors, especially connection latency and throughput can have a big impact; network throughput is the obvious upper limit independent on how much IO both ends can handle, and high-latencies can have a much higher impact than one might expect (keyword: TCP head of line blocking).

Neobin · Nov 28, 2022

Out of curiosity, I tested the Intel Core i9-9900K again with: mitigations=off:

Intel Core i9-9900K	Wiki reference	Default mitigations	Mitigations = off
PBS	1.0.8	2.2.7	2.2.7
Kernel	5.4	5.19	5.19
TLS (maximal backup upload speed)	not tested	not tested	not tested
SHA256 checksum computation speed	612.84 MB/s (30%)	648.44 MB/s (32%)	648.85 MB/s (32%)
ZStd level 1 compression speed	694.92 MB/s (92%)	690.37 MB/s (92%)	775.14 MB/s (103%)
ZStd level 1 decompression speed	1518.13 MB/s (127%)	940.24 MB/s (78%)	1149.87 MB/s (96%)
Chunk verification speed	438.38 MB/s (58%)	381.77 MB/s (50%)	413.62 MB/s (55%)
AES256 GCM encryption speed	4099.33 MB/s (112%)	2017.46 MB/s (55%)	2537.91 MB/s (70%)

Mitigations output from: lscpu:

Intel Core i9-9900K	Default mitigations	Mitigations = off
Vulnerability Itlb multihit	KVM: Mitigation: Split huge pages	KVM: Vulnerable
Vulnerability L1tf	Not affected	Not affected
Vulnerability Mds	Mitigation; Clear CPU buffers; SMT vulnerable	Vulnerable; SMT vulnerable
Vulnerability Meltdown	Not affected	Not affected
Vulnerability Mmio stale data	Mitigation; Clear CPU buffers; SMT vulnerable	Vulnerable
Vulnerability Retbleed	Mitigation; IBRS	Vulnerable
Vulnerability Spec store bypass	Mitigation; Speculative Store Bypass disabled via prctl	Vulnerable
Vulnerability Spectre v1	Mitigation; usercopy/swapgs barriers and __user pointer sanitization	Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2	Mitigation; IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected	Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Vulnerability Srbds	Mitigation; Microcode	Vulnerable
Vulnerability Tsx async abort	Mitigation; TSX disabled	Mitigation; TSX disabled

I am wondering, where the rather big difference for the "ZStd level 1 decompression speed" and especially the massive difference for the "AES256 GCM encryption speed" even with: mitigations=off compared to the wiki reference: [1] does come from.
@t.lamprecht: Could this be explained with changes made between the PBS- and/or kernel-versions?
One thing I could think of hardware-wise: Are those tests heavily RAM sensitive? Because the RAM in this machine runs at the CPU's official supported default/max speed of only 2666 MHz and if the reference machine from the wiki would have run with e.g.: 3200 MHz RAM, this might explain it? But would this really be such a massive difference in regard of the "AES256 GCM encryption speed"?
Not that I want to dig too deep in this, but a difference between ~4100 MB/s and ~2000/~2500 MB/s makes me a bit curious.

[1] https://pve.proxmox.com/wiki/PBS_Cl....28R.29_Core.28TM.29_i9-9900K_CPU_.40_3.60GHz

zaclys · Dec 5, 2023

hello

on mosts intel CPU , the proxmox-backup-client benchmark shows same results : the sha256 checksums and the chunk verification are very slow.

too late to replace my intel 48 cores cpu for an amd cpu , so :

- is it possible to adjust config to use other checksums ( in /etc/vzdump.conf or other config file ) ? to add threads ? other ideas ?

- do you know if promox 8 and linux kernel 6 are better than proxmox 7 and linux kernel 5.x on the same cpu for these benchmarks ?

- for backup CT , in vzdump.conf , the performance.pbs-entries-max parameter can accelerate these checksums ? it's only on Proxmox 8 ?

- for accelerate TLS maximal backup upload speed : is it possible to adjust size of chunks in config ? ( my PBS is distant of 300 km = latency )

thanks

note : before , i had use "rsync" linux shell old command to check and mirror 40 TB of datas ( every day ) . It had need less than 10 hours. now it needs 5x more time

meyergru · Dec 5, 2023

Funny that I just stumbled into the same problem, but with a Ryzen 7 5700G. Here are my results:

SHA256 speed: 2293.46 MB/s
Compression speed: 646.84 MB/s
Decompress speed: 772.44 MB/s
AES256/GCM speed: 1679.63 MB/s
Verify speed: 578.93 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ not tested │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2293.46 MB/s (113%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed │ 646.84 MB/s (86%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed │ 772.44 MB/s (64%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed │ 578.93 MB/s (76%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed │ 1679.63 MB/s (46%) │
└───────────────────────────────────┴─────────────────────┘

I frankly do not believe that my CPU should be much slower than a Ryzen 7 2700. Alas, I do not have the exact CPUs from the wiki to compare.

I know that some kernel routines for cryptography had problems with security mitigations, essentially disabling AVX extensions. I have not investigated if the routines in question are implemented directly in the application or if some kernel / library implementations are being used.
At least for the AES256/GCM part, it seems that the Rust implementation is poorly optimized, see here.

FWIW, from the surface of it:

1. It seems like the Ryzen 7 2700 comparison values are now unrealistic (although I cannot say for sure). In my specific case, the decompression speed gives rise for concern, as well.
2. One might ask if the current implementations could be improved when they used to be faster back in the day.

zaclys · Dec 6, 2023

But it's a good score for you on "SHA256 checksum computation speed" test !

On intel cpus , slow is really slow :

1/ intel Bi Xeon E5-2697 v4 ( 36 cores / 72 threads ) , proxmox 6 , linux kernel 5.4 :

Bash:

┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested         │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 359.33 MB/s (18%)  │ X
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 359.20 MB/s (48%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 771.63 MB/s (64%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 248.70 MB/s (33%)  │ X
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1963.75 MB/s (54%) │
└───────────────────────────────────┴────────────────────┘

2/ intel Bi Xeon Gold 6248R ( 48 cores / 96 threads) , promox VE 7 , linux kernel 5.15 :

Bash:

+===================================+====================+
| Name                              | Value              |
+===================================+====================+
| TLS (maximal backup upload speed) | not tested         |
+-----------------------------------+--------------------+
| SHA256 checksum computation speed | 464.94 MB/s (23%)  | X
+-----------------------------------+--------------------+
| ZStd level 1 compression speed    | 530.75 MB/s (71%)  |
+-----------------------------------+--------------------+
| ZStd level 1 decompression speed  | 761.70 MB/s (64%)  |
+-----------------------------------+--------------------+
| Chunk verification speed          | 285.24 MB/s (38%)  | X
+-----------------------------------+--------------------+
| AES256 GCM encryption speed       | 1722.09 MB/s (47%) |
+===================================+====================+

3/ AMD Ryzen 7 PRO 3700 ( 8 cores / 16 threads) , proxmox BS 3 , linux kernel 6.2 :

Bash:

+===================================+=====================+
| Name                              | Value               |
+===================================+=====================+
| TLS (maximal backup upload speed) | not tested          |
+-----------------------------------+---------------------+
| SHA256 checksum computation speed | 2174.90 MB/s (108%) | X
+-----------------------------------+---------------------+
| ZStd level 1 compression speed    | 540.87 MB/s (72%)   |
+-----------------------------------+---------------------+
| ZStd level 1 decompression speed  | 710.93 MB/s (59%)   |
+-----------------------------------+---------------------+
| Chunk verification speed          | 531.99 MB/s (70%)   | X
+-----------------------------------+---------------------+
| AES256 GCM encryption speed       | 1530.12 MB/s (42%)  |
+===================================+=====================+

If you read previous posts on this topic , you'll find same results : proxmox backup server is not designed for Intel cpus , AMD cpus only

( ref : https://pve.proxmox.com/wiki/PBS_Client_CPU_Benchmark#PBS_Client_Benchmark )

@proxmox staff : so please, is it possible or not to use other settings on PBS clients , to avoid intel cpu's bottlenecks ? ( due to kernel's mitigations for intel's vulnerabilities before 2020 )

thanks

How fast is your backup datastore? Benchmark tool

Renowned Member

Well-Known Member

Renowned Member

Well-Known Member

Distinguished Member

Well-Known Member

Distinguished Member

Renowned Member

Well-Known Member

Renowned Member

Member

Member

Famous Member

Distinguished Member

Well-Known Member

Proxmox Staff Member

Distinguished Member

New Member

Member

New Member