How fast is your backup datastore? Benchmark tool

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,702
1,452
164
Code:
Uploaded 1095 chunks in 5 seconds.
Time per request: 4571 microseconds.
TLS speed: 917.49 MB/s
SHA256 speed: 599.43 MB/s
Compression speed: 1684.74 MB/s
Decompress speed: 4631.42 MB/s
AES256/GCM speed: 3236.10 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 917.49 MB/s (133%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 599.43 MB/s (28%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1684.74 MB/s (78%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 4631.42 MB/s (57%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 3236.10 MB/s (85%) │
└───────────────────────────────────┴────────────────────┘

with current git master on an i9-10900X

there were quite some improvements on the TLS throughput side which are available up to pbstest / pve-no-subscription:
  • proxmox-backup-client/server 0.8.15-1
  • proxmox-backup-qemu 0.7.0-1
restarting PBS server and VMs is necessary to fully benefit from the changes.

would be curious to see results from those that were previously affected by low TLS speedtest results!
 
  • Like
Reactions: flames

sukerman

Member
Aug 29, 2019
50
4
13
50
Much better.... at least with the benchmark tool. Good work. This is over a link with 30ms latency, but before I got 2.5Mbps.

Code:
root@wopr:~# proxmox-backup-client benchmark --repository root@pam@xxxxxxxx:xxxxxxxxx

Uploaded 101 chunks in 5 seconds.
Time per request: 53192 microseconds.
TLS speed: 78.85 MB/s
SHA256 speed: 2025.31 MB/s
Compression speed: 2309.24 MB/s
Decompress speed: 9946.93 MB/s
AES256/GCM speed: 3634.58 MB/s

+===================================+=====================+
| Name                              | Value               |
+===================================+=====================+
| TLS (maximal backup upload speed) | 78.85 MB/s (13%)    |
+-----------------------------------+---------------------+
| SHA256 checksum computation speed | 2025.31 MB/s (96%)  |
+-----------------------------------+---------------------+
| ZStd level 1 compression speed    | 2309.24 MB/s (107%) |
+-----------------------------------+---------------------+
| ZStd level 1 decompression speed  | 9946.93 MB/s (123%) |
+-----------------------------------+---------------------+
| AES256 GCM encryption speed       | 3634.58 MB/s (96%)  |
+===================================+=====================+

Good work
 
  • Like
Reactions: r.jochum and fabian

shelter

New Member
Aug 24, 2020
7
2
3
The below benchmarks are using:

PBS Server - Dell T420 - Two Xeon E5-2440 - 12 Core - 128 GB DDR3 RAM - 10 GbE - ZFS w/ mirrored vdevs - SAS HDDs
PVE Server - HP DL360p - Two Xeon E5-2640 - 12 Core - 128 GB DDR3 RAM - 10 GbE

A. Local test

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘


B. Test from PVE to PBS with 10 GbE link

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘

Does anyone have any thoughts on what might be my bottleneck? I'm guessing old hardware and slow disks, but I'd love some input from those more savvy than I.
 

sukerman

Member
Aug 29, 2019
50
4
13
50
The below benchmarks are using:

PBS Server - Dell T420 - Two Xeon E5-2440 - 12 Core - 128 GB DDR3 RAM - 10 GbE - ZFS w/ mirrored vdevs - SAS HDDs
PVE Server - HP DL360p - Two Xeon E5-2640 - 12 Core - 128 GB DDR3 RAM - 10 GbE

A. Local test

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘


B. Test from PVE to PBS with 10 GbE link

Code:
Uploaded 206 chunks in 5 seconds.
Time per request: 24857 microseconds.
TLS speed: 168.73 MB/s
SHA256 speed: 249.48 MB/s
Compression speed: 371.69 MB/s
Decompress speed: 609.95 MB/s
AES256/GCM speed: 824.96 MB/s
Verify speed: 175.85 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 168.73 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 249.48 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 371.69 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 609.95 MB/s (51%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 175.85 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 824.96 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘

Does anyone have any thoughts on what might be my bottleneck? I'm guessing old hardware and slow disks, but I'd love some input from those more savvy than I.

I'm not that savvy but...... I get similar numbers to a remote machine, the worse the ping, the lower TLS speed I get, but if you're getting that locally I think it's your hardware, on a Ryzen 3600 with 4x raid 5 HDD's I get :

Code:
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1056.22 MB/s (153%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2016.49 MB/s (100%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 770.87 MB/s (103%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1362.65 MB/s (114%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 806.14 MB/s (106%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3614.14 MB/s (99%)  │
└───────────────────────────────────┴─────────────────────┘

With ping 30ms to remote over 1Gbps link:

Code:
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 17.18 MB/s (2%)     │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2057.26 MB/s (102%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 795.13 MB/s (106%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1379.15 MB/s (115%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 829.26 MB/s (109%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3721.90 MB/s (102%) │
└───────────────────────────────────┴─────────────────────┘
 

Felix.

Active Member
May 13, 2018
142
16
38
22
Wanted to share my results. :D

Hardware Specs:
Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
16GB ECC DDR3 RAM
2x Seagate ST33000650NS 3TB Disks as ZFS Mirror (checksum set to edonr, compression lz4, ashift=12)
Intel NIC with 1 Gbit/s Uplink

Code:
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 466.45 MB/s (68%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 372.51 MB/s (18%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 558.56 MB/s (74%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 956.81 MB/s (80%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 267.78 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1129.82 MB/s (31%) │
└───────────────────────────────────┴────────────────────┘
 

AngryAdm

Member
Sep 5, 2020
145
15
18
91
We provide a small benchmark tool providing some metrics about your datastore on your Proxmox Backup Server, here is an example:

Code:
root@pbs-20:~# proxmox-backup-client benchmark --repository mybackupstore

Uploaded 703 chunks in 5 seconds.
Time per request: 7131 microseconds.
TLS speed: 588.11 MB/s
SHA256 speed: 1629.45 MB/s
Compression speed: 1855.22 MB/s
Decompress speed: 8156.74 MB/s
AES256/GCM speed: 2949.58 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 588.11 MB/s (100%)  │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum comptation speed  │ 1629.45 MB/s (77%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 1855.22 MB/s (86%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 8156.74 MB/s (101%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2949.58 MB/s (78%)  │
└───────────────────────────────────┴─────────────────────┘
root@pbs-20:~#

This is an AMD EPYC 7302P 16-Core Processor (1 Socket), the datastore is a ZFS mirror on 2 x 6 TB HDDs.

You are benchmarking your RAM. Think about it, which HDD do you know that can write more than (average) 120Mb/second? Mirroring does not make that twice as fast..
 

AngryAdm

Member
Sep 5, 2020
145
15
18
91
Real world date, not benchmarking RAM. It is terribly slow.
INFO: 0% (208.1 MiB of 1000.0 GiB) in 3s, read: 69.4 MiB/s, write: 33.4 MiB/s
INFO: 1% (10.0 GiB of 1000.0 GiB) in 5m 18s, read: 31.9 MiB/s, write: 31.3 MiB/s
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,062
993
163
34
Vienna
You are benchmarking your RAM. Think about it, which HDD do you know that can write more than (average) 120Mb/second? Mirroring does not make that twice as fast..
yes, if you use slow disks, they will be your bottleneck, the idea of the tool is to provide a best case scenario (e.g. if you have fast ssds)
 
  • Like
Reactions: AngryAdm

oversite

Active Member
Jul 13, 2011
148
25
38
Real world date, not benchmarking RAM. It is terribly slow.
INFO: 0% (208.1 MiB of 1000.0 GiB) in 3s, read: 69.4 MiB/s, write: 33.4 MiB/s
INFO: 1% (10.0 GiB of 1000.0 GiB) in 5m 18s, read: 31.9 MiB/s, write: 31.3 MiB/s

well, this is also real world...
INFO: 18% (3.0 GiB of 16.0 GiB) in 3s, read: 1.0 GiB/s, write: 214.7 MiB/s
INFO: 41% (6.6 GiB of 16.0 GiB) in 6s, read: 1.2 GiB/s, write: 268.0 MiB/s
...
 
Sep 28, 2020
95
5
8
This is an VServer with 8 vCores, 16 GB RAM, the datastore is a 1,6 TB SAS (RAID10).
Code:
Uploaded 368 chunks in 5 seconds.
Time per request: 13652 microseconds.
TLS speed: 307.21 MB/s
SHA256 speed: 218.82 MB/s
Compression speed: 301.93 MB/s
Decompress speed: 476.84 MB/s
AES256/GCM speed: 1297.52 MB/s
Verify speed: 154.96 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 307.21 MB/s (45%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 218.82 MB/s (11%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 301.93 MB/s (40%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 476.84 MB/s (40%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 154.96 MB/s (20%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1297.52 MB/s (36%) │
└───────────────────────────────────┴────────────────────┘
 

symmcom

Renowned Member
Oct 28, 2012
1,084
36
68
Calgary, Canada
www.symmcom.com
Strange that this benchmark result shows so low given the specs. Perhaps because of the drives.
CPU: Xeon E5520 Dual Socket 16 Cores
RAM: 48GB RAM
HDD: 10x2TB 3.5" Seagate Barracuda in ZFS Raid-Z3
NIC: 2x10Gbps SFP+

Code:
Uploaded 94 chunks in 5 seconds.
Time per request: 54014 microseconds.
TLS speed: 77.65 MB/s
SHA256 speed: 175.80 MB/s
Compression speed: 297.50 MB/s
Decompress speed: 504.30 MB/s
AES256/GCM speed: 93.10 MB/s
Verify speed: 131.24 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 77.65 MB/s (6%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 175.80 MB/s (9%)  │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 297.50 MB/s (40%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 504.30 MB/s (42%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 131.24 MB/s (17%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 93.10 MB/s (3%)   │
└───────────────────────────────────┴───────────────────┘
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,062
993
163
34
Vienna
Strange that this benchmark result shows so low given the specs. Perhaps because of the drives.
its probably more due to the low single core speed of the E5520 (~2.5 Ghz on an 11 year old architecture)
also are you running the benchmark directly on that box? besides the tls benchmark, all others are run on the machine the command is invoked
 

sarada

New Member
Sep 29, 2020
4
0
1
25
Not the fastest, but it does its job nicely!
I'm glad I installed it, I was able to use it twice for restoration.
Thank you very much for the work of the team!

It currently has 22 VMs and 3 host backups.

Old low power hw, I know the limitations. :)
HP N36L, 8G ECC, 3x3T SATA drive in RAIDZ


Code:
Uploaded 42 chunks in 5 seconds.
Time per request: 123303 microseconds.
TLS speed: 34.02 MB/s
SHA256 speed: 87.72 MB/s
Compression speed: 97.25 MB/s
Decompress speed: 58.63 MB/s
AES256/GCM speed: 37.02 MB/s
Verify speed: 34.97 MB/s
┌───────────────────────────────────┬──────────────────┐
│ Name                              │ Value            │
╞═══════════════════════════════════╪══════════════════╡
│ TLS (maximal backup upload speed) │ 34.02 MB/s (3%)  │
├───────────────────────────────────┼──────────────────┤
│ SHA256 checksum computation speed │ 87.72 MB/s (4%)  │
├───────────────────────────────────┼──────────────────┤
│ ZStd level 1 compression speed    │ 97.25 MB/s (13%) │
├───────────────────────────────────┼──────────────────┤
│ ZStd level 1 decompression speed  │ 58.63 MB/s (5%)  │
├───────────────────────────────────┼──────────────────┤
│ Chunk verification speed          │ 34.97 MB/s (5%)  │
├───────────────────────────────────┼──────────────────┤
│ AES256 GCM encryption speed       │ 37.02 MB/s (1%)  │
└───────────────────────────────────┴──────────────────┘
 

cromatn5

Active Member
Mar 26, 2018
69
8
28
36
France
Xeon E3-1230V6, 32 GB ECC, PVE + PBS on same host.
Intel S4500 960 GB for proxmox and VM, lvm-thin.
ZFS mirror with 2 WD RED 5400 rpm for backups.


Code:
Uploaded 1310 chunks in 5 seconds.
Time per request: 3824 microseconds.
TLS speed: 1096.64 MB/s
SHA256 speed: 504.58 MB/s
Compression speed: 766.60 MB/s
Decompress speed: 1249.32 MB/s
AES256/GCM speed: 3408.73 MB/s
Verify speed: 358.00 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1096.64 MB/s (89%)  │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 504.58 MB/s (25%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 766.60 MB/s (102%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1249.32 MB/s (104%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 358.00 MB/s (47%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3408.73 MB/s (94%)  │
└───────────────────────────────────┴─────────────────────┘
 

RolandK

Member
Mar 5, 2019
162
19
23
49
since disk iops is a very important factor regarding pbs (if not the most important one, i have to admit we underestimated that) - what about adding a speedtest for verify & garbage collection ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!