How fast is your backup datastore? Benchmark tool

[fabian]

PBS is not designed for AMD CPUs. some CPUs are better at certain compute heavy tasks, some are slower, that's just how it is.

- for accelerate TLS maximal backup upload speed : is it possible to adjust size of chunks in config ? ( my PBS is distant of 300 km = latency )

that one will hit you pretty hard, since the HTTP/2 connection used for actually sending the backup data (or retrieving it in case of a restore) is pretty slow over higher latency links. I'd suggest setting up a local PBS system and syncing that to your remote PBS instance, to decouple the backup of the guest from the slow link.

 

[meyergru]

But it's a good score for you on "SHA256 checksum computation speed" test !

That is why I do not trust the comparison numbers to the Ryzen 7 2700. I bet that this very CPU itself would not give 100% scores any more.

As I wrote, there are two problems: 1. The comparability suffers because the historical scores do not fit any more and 2. This is caused by the fact that the implementation now seems to be even worse than before (whatever the cause is or what manufacturer gets hit by what). It just seems that this benchmark needs to be looked at and the implementations could be improved along the way.

Actually, I do not know if any of this is / could be handled in the kernel or be outsourced in a library that does implementations based on available CPU features. Linux definitely does this for AES (OpenZFS uses it), probably also for SHA256. I would guess that one could use OpenSSL library routines, too.

 

[xoma11]

Uploaded 1370 chunks in 5 seconds.
Time per request: 3655 microseconds.
TLS speed: 1147.34 MB/s
SHA256 speed: 618.54 MB/s
Compression speed: 646.70 MB/s
Decompress speed: 866.91 MB/s
AES256/GCM speed: 1868.19 MB/s
Verify speed: 355.95 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 1147.34 MB/s (93%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 618.54 MB/s (31%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 646.70 MB/s (86%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 866.91 MB/s (72%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 355.95 MB/s (47%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1868.19 MB/s (51%) │
└───────────────────────────────────┴────────────────────┘


8 x Intel(R) Xeon(R) E-2244G CPU @ 3.80GHz (1 Socket) 64GB 4*10TB HDD

 

[jamarsa]

My data:

proxmox-backup-client benchmark --repository root@pam@localhost:8007:Backup
Uploaded 1486 chunks in 5 seconds.
Time per request: 3370 microseconds.
TLS speed: 1244.29 MB/s
SHA256 speed: 539.57 MB/s
Compression speed: 597.08 MB/s
Decompress speed: 837.27 MB/s
AES256/GCM speed: 1865.33 MB/s
Verify speed: 326.39 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1244.29 MB/s (101%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 539.57 MB/s (27%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed │ 597.08 MB/s (79%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed │ 837.27 MB/s (70%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed │ 326.39 MB/s (43%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed │ 1865.33 MB/s (51%) │
└───────────────────────────────────┴─────────────────────┘

CPU:
Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz

ZFS RAID 0 disks (2), with a NVMe disk as cache/special.
Device Model: HGST HUS728T8TALN6L4
Firmware Version: V8GNW980
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Size: 4096 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)

capacity operations bandwidth
pool alloc free read write read write
------------------------------------------------------------------------------ ----- ----- ----- ----- ----- -----
Backup 5.79T 8.79T 306 86 33.5M 12.7M
sdb 2.97T 4.30T 131 7 16.8M 5.87M
sda 2.79T 4.48T 128 8 16.5M 6.23M
special - - - - - -
pbs-meta2 33.5G 16.0G 47 70 245K 687K
cache - - - - - -
pbs-cache2 80.0G 17.6M 84 46 637K 4.68M
------------------------------------------------------------------------------ ----- ----- ----- ----- ----- -----

I'm a bit perplexed by the real verify speed... Never over 150 MB/s, and an average of about 100 MB/s. The same speed when I write to tape (which in theory is capable of 300 MB/s, LTO-8 over SAS). I have seen read speeds of 300-500 MB/s when ZFS was doing a scrub... (not reflected in the web interface):


But on a verify/tape backup is much worse:

I can attach fio tests if anyone is interested. In short, a random read gives about 210 MB/s, a sequential read about 1400 MB/s.

Of course, all the tests were done in local over the server. Backup speed over the network is currently limited by the NIC to 1Gbps; just waiting for a 10Gbps Switch to improve backup time.
Another paradox is that average write speeds are double read speeds (i.e., write speeds are as expected)

 

[juancarlo2596]

root@proxmox02:~# proxmox-backup-client benchmark --repository root@pam@172.254.254.20bsDrive
Password for "root@pam":
Uploaded 154 chunks in 5 seconds.
Time per request: 33459 microseconds.
TLS speed: 125.35 MB/s
SHA256 speed: 283.37 MB/s
Compression speed: 301.50 MB/s
Decompress speed: 268.80 MB/s
AES256/GCM speed: 581.42 MB/s
Verify speed: 160.91 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 125.35 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 283.37 MB/s (14%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 301.50 MB/s (40%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 268.80 MB/s (22%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 160.91 MB/s (21%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 581.42 MB/s (16%) │
└───────────────────────────────────┴───────────────────┘

pbs running on a vm
vm specs: 12 (2 sockets, 6 cores) [host.flags=+aes] [numa=1], 32GB RAM, 32GB for OS (On SSD), 6TB for Datastore (On HDD), dedicated 10Gb Network

host specs:
Dell PowerEdge T620
24 x Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz (2 Sockets)
200GB RAM
2x Samsung 870 Evo SSD 1TB connected on SATA Port (SW RAID 1 for OS)
8x 6TB WD Enterprise HDD connected on PERC H710 (HW RAID 6 for storage)

Quite slow when backing up vms and lxc even if I have a dedicated 10Gb network and getting about 120-150 MB/s, my TrueNAS vm also on a dedicated 10Gb is much faster. I can get about 350-400 MB/s on average speed on transfer. Any recommendations for a fast backup speed on PBS?

 

[jamarsa]

pbs running on a vm
vm specs: 12 (2 sockets, 6 cores) [host.flags=+aes] [numa=1], 32GB RAM, 32GB for OS (On SSD), 6TB for Datastore (On HDD), dedicated 10Gb Network

host specs:
Dell PowerEdge T620
24 x Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz (2 Sockets)
200GB RAM
2x Samsung 870 Evo SSD 1TB connected on SATA Port (SW RAID 1 for OS)
8x 6TB WD Enterprise HDD connected on PERC H710 (HW RAID 6 for storage)

Quite slow when backing up vms and lxc even if I have a dedicated 10Gb network and getting about 120-150 MB/s, my TrueNAS vm also on a dedicated 10Gb is much faster. I can get about 350-400 MB/s on average speed on transfer. Any recommendations for a fast backup speed on PBS?

I'm not by any means an expert, but it seems that your CPU is not powerful enough to handle more than 125/160 GB/s encryption/decryption. Perhaps it is because of the emulation type (not host), or maybe the total number of cores assigned. Also, ¿how is the load of the host, are other VMs soaking up CPU cycles?

 

[juancarlo2596]

I'm not by any means an expert, but it seems that your CPU is not powerful enough to handle more than 125/160 GB/s encryption/decryption. Perhaps it is because of the emulation type (not host), or maybe the total number of cores assigned. Also, ¿how is the load of the host, are other VMs soaking up CPU cycles?

hi, cpu emulation is set to host. i have only 2 vms running on my server, TrueNAS and PBS. Host is only taking up 2-3% load

 

[Reifide]

Network: 10Gbps
CPU PVE: 32 x Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz (1 Socket) (128GB RAM)
CPU PBS: 8 x 12th Gen Intel(R) Core(TM) i3-12100 (1 Socket) (32GB RAM)
Storage: M.2 NVME

Benchmark: Localhost on PBS:

Uploaded 1767 chunks in 5 seconds.
Time per request: 2832 microseconds.
TLS speed: 1480.68 MB/s
SHA256 speed: 2036.89 MB/s
Compression speed: 694.73 MB/s
Decompress speed: 1021.38 MB/s
AES256/GCM speed: 2498.39 MB/s
Verify speed: 680.79 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1480.68 MB/s (120%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2036.89 MB/s (101%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 694.73 MB/s (92%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1021.38 MB/s (85%)  │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 680.79 MB/s (90%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2498.39 MB/s (69%)  │
└───────────────────────────────────┴─────────────────────┘

Benchmark: PVE Server, over the network, to the PBS server

Uploaded 1345 chunks in 5 seconds.
Time per request: 3725 microseconds.
TLS speed: 1125.97 MB/s
SHA256 speed: 461.63 MB/s
Compression speed: 435.40 MB/s
Decompress speed: 654.90 MB/s
AES256/GCM speed: 1313.35 MB/s
Verify speed: 269.51 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 1125.97 MB/s (91%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 461.63 MB/s (23%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 435.40 MB/s (58%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 654.90 MB/s (55%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 269.51 MB/s (36%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1313.35 MB/s (36%) │
└───────────────────────────────────┴────────────────────┘

 

[Gysmayo]

Uploaded 306 chunks in 5 seconds.
Time per request: 16434 microseconds.
TLS speed: 255.21 MB/s
SHA256 speed: 271.52 MB/s
Compression speed: 310.80 MB/s
Decompress speed: 473.66 MB/s
AES256/GCM speed: 852.06 MB/s
Verify speed: 189.51 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 255.21 MB/s (21%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 271.52 MB/s (13%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 310.80 MB/s (41%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 473.66 MB/s (40%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 189.51 MB/s (25%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 852.06 MB/s (23%) │
└───────────────────────────────────┴───────────────────┘

CPU:
48 x Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz (2 Sockets)

Memory:
64Gb

disk:
1 x crucial 250GB SSD (root)
15 x 18TB Seagate EXO drives (raid6 p

net:
2 x Intel Corporation 82599ES 10-Gigabit SFI/SFP+

How is this getting such a bad performance especially restore performance is very slow. (38MB/s)

 

[iffster]

Guys, seriously, this "benchmark" doesn't test your storage (ssd/hdd/nvme) or your server. It tests single core of your CPU and that is all.
I mean why are we still posting results here?

 

[Dunuin]

Guys, seriously, this "benchmark" doesn't test your storage (ssd/hdd/nvme) or your server. It tests single core of your CPU and that is all.
I mean why are we still posting results here?

Shouldn't "Chunk verification speed" also account for storage performance as chunks need to be read?
And ZSTD (de)compression should be able to make use of multithreaded performance?

 

[iffster]

Shouldn't "Chunk verification speed" also account for storage performance as chunks need to be read?
And ZSTD (de)compression should be able to make use of multithreaded performance?

We don't know what were the intentions of the developer of "benchmark" option but from the official proxmox documentation https://pbs.proxmox.com/docs/backup-client.html :
"The backup client also comes with a benchmarking tool. This tool measures various metrics relating to compression and encryption speeds. If a ProxmoxBackup repository (remote or local) is specified, the TLS upload speed will get measured too."

As I understand it is indeed supposed to measure only compression/encryption and TLS speed (and in single thread - but it's not stated).
My previous statement (or "poke" if you want) was addressed to forum members who think that they are testing actual storage speed
and that with this test they will find out how fast they will backup/restore.

Also, I run multiple backup servers with 7+years cpu generation difference - for example X5650 Xeon (from 2010) and Gold 6148 from 2017 and both of them report almost identical results (this benchmark)

Also something to think about - X5650 is backed by raid10 zfs SAS 7200k storage (12x disks) and Xeon Gold 6148 is backed by 2x NVMe Samsung PM1733 7.68TB drives in raid1 (zfs) and both of them perform identical when you write backups (300MB/sec input limit). I believe that some process responsible for writing backups on the PBS server(i'm talking about receiving part) is still single threaded.

But read speeds are better on nvme/xeon gold of course, all the verification processes and restore will work faster.