How fast is your backup datastore? Benchmark tool

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,122
1,583
164
Code:
Uploaded 1095 chunks in 5 seconds.
Time per request: 4571 microseconds.
TLS speed: 917.49 MB/s
SHA256 speed: 599.43 MB/s
Compression speed: 1684.74 MB/s
Decompress speed: 4631.42 MB/s
AES256/GCM speed: 3236.10 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 917.49 MB/s (133%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 599.43 MB/s (28%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1684.74 MB/s (78%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 4631.42 MB/s (57%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 3236.10 MB/s (85%) │
└───────────────────────────────────┴────────────────────┘

with current git master on an i9-10900X

there were quite some improvements on the TLS throughput side which are available up to pbstest / pve-no-subscription:
  • proxmox-backup-client/server 0.8.15-1
  • proxmox-backup-qemu 0.7.0-1
restarting PBS server and VMs is necessary to fully benefit from the changes.

would be curious to see results from those that were previously affected by low TLS speedtest results!
 
  • Like
Reactions: flames

sukerman

Member
Aug 29, 2019
50
4
13
50
Much better.... at least with the benchmark tool. Good work. This is over a link with 30ms latency, but before I got 2.5Mbps.

Code:
root@wopr:~# proxmox-backup-client benchmark --repository root@pam@xxxxxxxx:xxxxxxxxx

Uploaded 101 chunks in 5 seconds.
Time per request: 53192 microseconds.
TLS speed: 78.85 MB/s
SHA256 speed: 2025.31 MB/s
Compression speed: 2309.24 MB/s
Decompress speed: 9946.93 MB/s
AES256/GCM speed: 3634.58 MB/s

+===================================+=====================+
| Name                              | Value               |
+===================================+=====================+
| TLS (maximal backup upload speed) | 78.85 MB/s (13%)    |
+-----------------------------------+---------------------+
| SHA256 checksum computation speed | 2025.31 MB/s (96%)  |
+-----------------------------------+---------------------+
| ZStd level 1 compression speed    | 2309.24 MB/s (107%) |
+-----------------------------------+---------------------+
| ZStd level 1 decompression speed  | 9946.93 MB/s (123%) |
+-----------------------------------+---------------------+
| AES256 GCM encryption speed       | 3634.58 MB/s (96%)  |
+===================================+=====================+

Good work
 
  • Like
Reactions: r.jochum and fabian

shelter

New Member
Aug 24, 2020
7
2
3
The below benchmarks are using:

PBS Server - Dell T420 - Two Xeon E5-2440 - 12 Core - 128 GB DDR3 RAM - 10 GbE - ZFS w/ mirrored vdevs - SAS HDDs
PVE Server - HP DL360p - Two Xeon E5-2640 - 12 Core - 128 GB DDR3 RAM - 10 GbE

A. Local test

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘


B. Test from PVE to PBS with 10 GbE link

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘

Does anyone have any thoughts on what might be my bottleneck? I'm guessing old hardware and slow disks, but I'd love some input from those more savvy than I.
 

sukerman

Member
Aug 29, 2019
50
4
13
50
The below benchmarks are using:

PBS Server - Dell T420 - Two Xeon E5-2440 - 12 Core - 128 GB DDR3 RAM - 10 GbE - ZFS w/ mirrored vdevs - SAS HDDs
PVE Server - HP DL360p - Two Xeon E5-2640 - 12 Core - 128 GB DDR3 RAM - 10 GbE

A. Local test

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘


B. Test from PVE to PBS with 10 GbE link

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘

Does anyone have any thoughts on what might be my bottleneck? I'm guessing old hardware and slow disks, but I'd love some input from those more savvy than I.

I'm not that savvy but...... I get similar numbers to a remote machine, the worse the ping, the lower TLS speed I get, but if you're getting that locally I think it's your hardware, on a Ryzen 3600 with 4x raid 5 HDD's I get :

Code:
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1056.22 MB/s (153%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2016.49 MB/s (100%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 770.87 MB/s (103%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1362.65 MB/s (114%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 806.14 MB/s (106%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3614.14 MB/s (99%)  │
└───────────────────────────────────┴─────────────────────┘

With ping 30ms to remote over 1Gbps link:

Code:
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 17.18 MB/s (2%)     │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2057.26 MB/s (102%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 795.13 MB/s (106%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1379.15 MB/s (115%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 829.26 MB/s (109%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3721.90 MB/s (102%) │
└───────────────────────────────────┴─────────────────────┘
 

Felix.

Active Member
May 13, 2018
154
18
38
22
Wanted to share my results. :D

Hardware Specs:
Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
16GB ECC DDR3 RAM
2x Seagate ST33000650NS 3TB Disks as ZFS Mirror (checksum set to edonr, compression lz4, ashift=12)
Intel NIC with 1 Gbit/s Uplink

Code:
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 466.45 MB/s (68%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 372.51 MB/s (18%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 558.56 MB/s (74%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 956.81 MB/s (80%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 267.78 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1129.82 MB/s (31%) │
└───────────────────────────────────┴────────────────────┘
 

AngryAdm

Member
Sep 5, 2020
145
17
18
91
We provide a small benchmark tool providing some metrics about your datastore on your Proxmox Backup Server, here is an example:

Code:
root@pbs-20:~# proxmox-backup-client benchmark --repository mybackupstore

Uploaded 703 chunks in 5 seconds.
Time per request: 7131 microseconds.
TLS speed: 588.11 MB/s
SHA256 speed: 1629.45 MB/s
Compression speed: 1855.22 MB/s
Decompress speed: 8156.74 MB/s
AES256/GCM speed: 2949.58 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 588.11 MB/s (100%)  │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum comptation speed  │ 1629.45 MB/s (77%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 1855.22 MB/s (86%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 8156.74 MB/s (101%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2949.58 MB/s (78%)  │
└───────────────────────────────────┴─────────────────────┘
root@pbs-20:~#

This is an AMD EPYC 7302P 16-Core Processor (1 Socket), the datastore is a ZFS mirror on 2 x 6 TB HDDs.

You are benchmarking your RAM. Think about it, which HDD do you know that can write more than (average) 120Mb/second? Mirroring does not make that twice as fast..
 

AngryAdm

Member
Sep 5, 2020
145
17
18
91
Real world date, not benchmarking RAM. It is terribly slow.
INFO: 0% (208.1 MiB of 1000.0 GiB) in 3s, read: 69.4 MiB/s, write: 33.4 MiB/s
INFO: 1% (10.0 GiB of 1000.0 GiB) in 5m 18s, read: 31.9 MiB/s, write: 31.3 MiB/s
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,484
1,090
164
34
Vienna
You are benchmarking your RAM. Think about it, which HDD do you know that can write more than (average) 120Mb/second? Mirroring does not make that twice as fast..
yes, if you use slow disks, they will be your bottleneck, the idea of the tool is to provide a best case scenario (e.g. if you have fast ssds)
 
  • Like
Reactions: AngryAdm

oversite

Active Member
Jul 13, 2011
148
25
38
Real world date, not benchmarking RAM. It is terribly slow.
INFO: 0% (208.1 MiB of 1000.0 GiB) in 3s, read: 69.4 MiB/s, write: 33.4 MiB/s
INFO: 1% (10.0 GiB of 1000.0 GiB) in 5m 18s, read: 31.9 MiB/s, write: 31.3 MiB/s

well, this is also real world...
INFO: 18% (3.0 GiB of 16.0 GiB) in 3s, read: 1.0 GiB/s, write: 214.7 MiB/s
INFO: 41% (6.6 GiB of 16.0 GiB) in 6s, read: 1.2 GiB/s, write: 268.0 MiB/s
...
 
Sep 28, 2020
98
5
13
This is an VServer with 8 vCores, 16 GB RAM, the datastore is a 1,6 TB SAS (RAID10).
Code:
Uploaded 368 chunks in 5 seconds.
Time per request: 13652 microseconds.
TLS speed: 307.21 MB/s
SHA256 speed: 218.82 MB/s
Compression speed: 301.93 MB/s
Decompress speed: 476.84 MB/s
AES256/GCM speed: 1297.52 MB/s
Verify speed: 154.96 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 307.21 MB/s (45%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 218.82 MB/s (11%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 301.93 MB/s (40%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 476.84 MB/s (40%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 154.96 MB/s (20%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1297.52 MB/s (36%) │
└───────────────────────────────────┴────────────────────┘
 

symmcom

Renowned Member
Oct 28, 2012
1,087
38
68
Calgary, Canada
www.symmcom.com
Strange that this benchmark result shows so low given the specs. Perhaps because of the drives.
CPU: Xeon E5520 Dual Socket 16 Cores
RAM: 48GB RAM
HDD: 10x2TB 3.5" Seagate Barracuda in ZFS Raid-Z3
NIC: 2x10Gbps SFP+

Code:
Uploaded 94 chunks in 5 seconds.
Time per request: 54014 microseconds.
TLS speed: 77.65 MB/s
SHA256 speed: 175.80 MB/s
Compression speed: 297.50 MB/s
Decompress speed: 504.30 MB/s
AES256/GCM speed: 93.10 MB/s
Verify speed: 131.24 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 77.65 MB/s (6%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 175.80 MB/s (9%)  │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 297.50 MB/s (40%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 504.30 MB/s (42%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 131.24 MB/s (17%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 93.10 MB/s (3%)   │
└───────────────────────────────────┴───────────────────┘
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,484
1,090
164
34
Vienna
Strange that this benchmark result shows so low given the specs. Perhaps because of the drives.
its probably more due to the low single core speed of the E5520 (~2.5 Ghz on an 11 year old architecture)
also are you running the benchmark directly on that box? besides the tls benchmark, all others are run on the machine the command is invoked
 

sarada

Member
Sep 29, 2020
4
0
6
25
Not the fastest, but it does its job nicely!
I'm glad I installed it, I was able to use it twice for restoration.
Thank you very much for the work of the team!

It currently has 22 VMs and 3 host backups.

Old low power hw, I know the limitations. :)
HP N36L, 8G ECC, 3x3T SATA drive in RAIDZ


Code:
Uploaded 42 chunks in 5 seconds.
Time per request: 123303 microseconds.
TLS speed: 34.02 MB/s
SHA256 speed: 87.72 MB/s
Compression speed: 97.25 MB/s
Decompress speed: 58.63 MB/s
AES256/GCM speed: 37.02 MB/s
Verify speed: 34.97 MB/s
┌───────────────────────────────────┬──────────────────┐
│ Name                              │ Value            │
╞═══════════════════════════════════╪══════════════════╡
│ TLS (maximal backup upload speed) │ 34.02 MB/s (3%)  │
├───────────────────────────────────┼──────────────────┤
│ SHA256 checksum computation speed │ 87.72 MB/s (4%)  │
├───────────────────────────────────┼──────────────────┤
│ ZStd level 1 compression speed    │ 97.25 MB/s (13%) │
├───────────────────────────────────┼──────────────────┤
│ ZStd level 1 decompression speed  │ 58.63 MB/s (5%)  │
├───────────────────────────────────┼──────────────────┤
│ Chunk verification speed          │ 34.97 MB/s (5%)  │
├───────────────────────────────────┼──────────────────┤
│ AES256 GCM encryption speed       │ 37.02 MB/s (1%)  │
└───────────────────────────────────┴──────────────────┘
 

cromatn5

Active Member
Mar 26, 2018
69
8
28
37
France
Xeon E3-1230V6, 32 GB ECC, PVE + PBS on same host.
Intel S4500 960 GB for proxmox and VM, lvm-thin.
ZFS mirror with 2 WD RED 5400 rpm for backups.


Code:
Uploaded 1310 chunks in 5 seconds.
Time per request: 3824 microseconds.
TLS speed: 1096.64 MB/s
SHA256 speed: 504.58 MB/s
Compression speed: 766.60 MB/s
Decompress speed: 1249.32 MB/s
AES256/GCM speed: 3408.73 MB/s
Verify speed: 358.00 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1096.64 MB/s (89%)  │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 504.58 MB/s (25%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 766.60 MB/s (102%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1249.32 MB/s (104%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 358.00 MB/s (47%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3408.73 MB/s (94%)  │
└───────────────────────────────────┴─────────────────────┘
 

RolandK

Active Member
Mar 5, 2019
307
56
33
49
since disk iops is a very important factor regarding pbs (if not the most important one, i have to admit we underestimated that) - what about adding a speedtest for verify & garbage collection ?
 

Brandito

New Member
Jan 12, 2022
12
3
3
38
I get the following error when using the --repository flag in PBS running in an LXC

Error: Permission denied (os error 13)

Without the flag the benchmark runs. I have two other instances of PBS, one in a VM on my nas and another hosted remotely that have no issues.

Is it possible running in an LXC is the issue?

Test results when connecting to my PBS VM hosted on a DS918 with a 5GBe USB adapter below

Code:
Uploaded 157 chunks in 5 seconds.
Time per request: 33147 microseconds.
TLS speed: 126.53 MB/s
SHA256 speed: 375.54 MB/s
Compression speed: 407.40 MB/s
Decompress speed: 600.51 MB/s
AES256/GCM speed: 1296.21 MB/s
Verify speed: 245.63 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 126.53 MB/s (10%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 375.54 MB/s (19%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 407.40 MB/s (54%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 600.51 MB/s (50%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 245.63 MB/s (32%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1296.21 MB/s (36%) │
└───────────────────────────────────┴────────────────────┘
 

Dunuin

Famous Member
Jun 30, 2020
7,997
1,993
149
Germany
Client:
PVE 7.2
Supermicro X10SRM-F
1x Xeon E5-2683 v4; 32 Threads @ shedutil 1.2-3.0 GHz
4x 16GB ECC DDR4 2133MHz
10Gbit + Gbit NICs

Server:
PBS 2.2 ontop of Debian 11.4 inside VM using 2GB RAM and 3 vCPUs on a TrueNAS server with datastore over NFS from same server on raidz1 ZFS pool of 4x 8TB 7200RPM HDDs + 2x S3710 200GB as mirrored special devices. Ashift=12, recordsize=1M, relatime=on, encrypted pool.
Supermicro X10SLL-F
1x Xeon E3-1230 v3; 8 threads @ powerd 0.8-3.3GHz
4x 8GB ECC DDR3 1600MHz
10Gbit + Gbit NICs

Code:
Uploaded 148 chunks in 5 seconds.
Time per request: 35689 microseconds.
TLS speed: 117.52 MB/s
SHA256 speed: 323.79 MB/s
Compression speed: 329.19 MB/s
Decompress speed: 459.30 MB/s
AES256/GCM speed: 1029.28 MB/s
Verify speed: 192.18 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 117.52 MB/s (10%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 323.79 MB/s (16%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 329.19 MB/s (44%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 459.30 MB/s (38%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 192.18 MB/s (25%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1029.28 MB/s (28%) │
└───────────────────────────────────┴────────────────────┘

Edit:
Good that I benchmarked it. TLS speed of 117MB/s indicated that I'm limited to Gbit speeds...indeed...I used the IP of the Gbit NIC to add my PBS to PVE. Changed it now to the IP of the 10Gbit NIC and got over double the speed :)
Code:
Time per request: 15056 microseconds.
TLS speed: 278.56 MB/s
SHA256 speed: 321.90 MB/s
Compression speed: 315.83 MB/s
Decompress speed: 440.22 MB/s
AES256/GCM speed: 991.42 MB/s
Verify speed: 183.73 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 278.56 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 321.90 MB/s (16%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 315.83 MB/s (42%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 440.22 MB/s (37%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 183.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 991.42 MB/s (27%) │
└───────────────────────────────────┴───────────────────┘
 
Last edited:
  • Like
Reactions: fabian

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,122
1,583
164
I get the following error when using the --repository flag in PBS running in an LXC

Error: Permission denied (os error 13)

Without the flag the benchmark runs. I have two other instances of PBS, one in a VM on my nas and another hosted remotely that have no issues.

Is it possible running in an LXC is the issue?
maybe open a new thread with all the relevant details (PBS client version, PBS server version, full output of command, do other commands like status --repository .. work, ..)
 

James Crook

Active Member
Jul 28, 2017
142
2
38
Uploaded 171 chunks in 5 seconds.
Time per request: 29563 microseconds.
TLS speed: 141.88 MB/s
SHA256 speed: 332.99 MB/s
Compression speed: 355.48 MB/s
Decompress speed: 493.69 MB/s
AES256/GCM speed: 185.94 MB/s
Verify speed: 197.33 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 141.88 MB/s (11%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 332.99 MB/s (16%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 355.48 MB/s (47%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 493.69 MB/s (41%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 197.33 MB/s (26%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 185.94 MB/s (5%) │
└───────────────────────────────────┴───────────────────┘

i3-3240 CPU @ 3.40GHz
16GB RAM
4x 2TB WD RED (CMR) ZFS Raid 0
 

ITT

Active Member
Mar 19, 2021
284
67
28
42
Xeon E3 1231 v3 / 32GB Ram, SSD


Powersave:

Code:
Uploaded 197 chunks in 5 seconds.
Time per request: 25609 microseconds.
TLS speed: 163.78 MB/s
SHA256 speed: 92.52 MB/s
Compression speed: 89.93 MB/s
Decompress speed: 142.32 MB/s
AES256/GCM speed: 268.44 MB/s
Verify speed: 57.31 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 163.78 MB/s (13%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 92.52 MB/s (5%)   │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 89.93 MB/s (12%)  │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 142.32 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 57.31 MB/s (8%)   │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 268.44 MB/s (7%)  │
└───────────────────────────────────┴───────────────────┘



Schedutil:

Code:
Uploaded 936 chunks in 5 seconds.
Time per request: 5378 microseconds.
TLS speed: 779.85 MB/s
SHA256 speed: 444.46 MB/s
Compression speed: 434.62 MB/s
Decompress speed: 671.52 MB/s
AES256/GCM speed: 1268.67 MB/s
Verify speed: 264.32 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 779.85 MB/s (63%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 444.46 MB/s (22%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 434.62 MB/s (58%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 671.52 MB/s (56%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 264.32 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1268.67 MB/s (35%) │
└───────────────────────────────────┴────────────────────┘



Ondemand:

Code:
Uploaded 928 chunks in 5 seconds.
Time per request: 5425 microseconds.
TLS speed: 773.08 MB/s
SHA256 speed: 445.40 MB/s
Compression speed: 444.50 MB/s
Decompress speed: 670.40 MB/s
AES256/GCM speed: 1282.66 MB/s
Verify speed: 270.87 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 773.08 MB/s (63%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 445.40 MB/s (22%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 444.50 MB/s (59%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 670.40 MB/s (56%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 270.87 MB/s (36%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1282.66 MB/s (35%) │
└───────────────────────────────────┴────────────────────┘


Performance:

Code:
Uploaded 930 chunks in 5 seconds.
Time per request: 5383 microseconds.
TLS speed: 779.16 MB/s
SHA256 speed: 443.14 MB/s
Compression speed: 438.01 MB/s
Decompress speed: 688.25 MB/s
AES256/GCM speed: 1288.04 MB/s
Verify speed: 273.85 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 779.16 MB/s (63%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 443.14 MB/s (22%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 438.01 MB/s (58%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 688.25 MB/s (57%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 273.85 MB/s (36%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1288.04 MB/s (35%) │
└───────────────────────────────────┴────────────────────┘
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!