Increasing Backup Speed?

M

mmidgett

Renowned Member
Proxmox Subscriber
Dec 31, 2012
65
8
73
Mooresville, North Carolina
www.trick-solutions.com
We've just recently started testing PBS and I really like the simplicity of the software. But I feel that something must be wrong because backups are much slower than vzdump to a NFS share.

Network Bonded 10G to all servers to the TOR using LACP with L3+4

TOR switches = Arista in MLAG fashion with MTU 9000

Typical vzdump will consume about 5Gbits of the network while processing the backups.

PBS is only using about 350Mbits.

Hypervisors are all Dell R610,620'son PVE 6.3

PBS 1.0 is running on a Dell R710 with Dual quad cores at 2.26GHZ with 16Gb ram. I have 6x 2TB drives in a raid 10 with 128bit stripe with writeback cache enabled. PBS average backup speed is 50MB according to the job.

I've used this server and before as an NFS mount point and it will do 250MB/s (ish) of disk writes which I should be seeing 3Gbits of network speed.

Here is the PBS speed test.

Uploaded 118 chunks in 5 seconds.
Time per request: 44925 microseconds.
TLS speed: 93.36 MB/s
SHA256 speed: 258.49 MB/s
Compression speed: 433.75 MB/s
Decompress speed: 727.66 MB/s
AES256/GCM speed: 799.33 MB/s
Verify speed: 191.80 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 93.36 MB/s (8%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 258.49 MB/s (13%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 433.75 MB/s (58%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 727.66 MB/s (61%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 191.80 MB/s (25%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 799.33 MB/s (22%) │


TLS seems very slow for Dual Xeon X5670's at 2.93GHZ
 
AES enabled in your BIOS?

Check with:

> grep -m1 -o aes /proc/cpuinfo
 
Last edited:
Just to post a benchmark before the CPU's are upgraded I ran this on the PBS server against itself.

Uploaded 107 chunks in 5 seconds.
Time per request: 47380 microseconds.
TLS speed: 88.52 MB/s
SHA256 speed: 202.00 MB/s
Compression speed: 337.06 MB/s
Decompress speed: 561.61 MB/s
AES256/GCM speed: 106.60 MB/s
Verify speed: 148.01 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 88.52 MB/s (7%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 202.00 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 337.06 MB/s (45%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 561.61 MB/s (47%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 148.01 MB/s (20%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 106.60 MB/s (3%) │
└───────────────────────────────────┴───────────────────┘
 

Attachments

  • 1616705775557.png
    1616705775557.png
    32.9 KB · Views: 47
After the CPU has been upgraded.

Uploaded 562 chunks in 5 seconds.
Time per request: 8914 microseconds.
TLS speed: 470.51 MB/s
SHA256 speed: 265.23 MB/s
Compression speed: 444.40 MB/s
Decompress speed: 726.08 MB/s
AES256/GCM speed: 788.28 MB/s
Verify speed: 193.03 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 470.51 MB/s (38%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 265.23 MB/s (13%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 444.40 MB/s (59%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 726.08 MB/s (61%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 193.03 MB/s (25%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 788.28 MB/s (22%) │
 
mmidgett said:
What directly relates to the slow chunk verification speed?
Click to expand...
this is the combination of zstd decompress + sha256 speed (since each chunk must be decompressed and then hashed)
you can check that it is exactly that:

VerifyRate = 1/((1/zstd decompression rate)+(1/sha256 rate))

Code: 
1/((1/726.08)+(1/265.23)) ~ 194
which is your chunk verification speed (rounding/precision errors aside)
 
dcsapak said:
this is the combination of zstd decompress + sha256 speed (since each chunk must be decompressed and then hashed)
you can check that it is exactly that:

VerifyRate = 1/((1/zstd decompression rate)+(1/sha256 rate))

Code: 
1/((1/726.08)+(1/265.23)) ~ 194
which is your chunk verification speed (rounding/precision errors aside)
Click to expand...
Hi,
I don't know if this is still true, but we have 2 x 3 nodes PVE 8 clusters + PBS with lots of timeout, both built on same hardware, and by the same guy. Both clusters suffers abyssal TLS perf a soon as traffic goes through network stack.

Something seems wrong with TLS :
Bash: 
[root@pbs1:~]# proxmox-backup-client benchmark --repository root@pam@10.11.12.13:data
Uploaded 10 chunks in 35 seconds.
Time per request: 3512695 microseconds.
TLS speed: 1.19 MB/s
SHA256 speed: 886.22 MB/s
Compression speed: 358.33 MB/s
Decompress speed: 530.14 MB/s
AES256/GCM speed: 1143.77 MB/s
Verify speed: 396.64 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ [B]1.19[/B] MB/s (0%)     │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 886.22 MB/s (44%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 358.33 MB/s (48%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 530.14 MB/s (44%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 396.64 MB/s (52%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1143.77 MB/s (31%) │
└───────────────────────────────────┴────────────────────┘

Of course aes in enabled :

Bash: 
[root@pbs1:~]# grep -m1 -o aes /proc/cpuinfo
aes

And a different repository access leads to :


Bash: 
[root@pbs1:~]# proxmox-backup-client benchmark --repository data
Uploaded 467 chunks in 5 seconds.
Time per request: 10738 microseconds.
TLS speed: 390.60 MB/s
SHA256 speed: 947.21 MB/s
Compression speed: 390.41 MB/s
Decompress speed: 529.54 MB/s
AES256/GCM speed: 1156.67 MB/s
Verify speed: 395.60 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │[B] 390.60 MB/s[/B] (32%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 947.21 MB/s (47%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 390.41 MB/s (52%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 529.54 MB/s (44%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 395.60 MB/s (52%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1156.67 MB/s (32%) │
└───────────────────────────────────┴────────────────────┘

Network physical links tested :
- 2 x 1GbE
- 2 x 10G optical

Becoming crazy...
 
No, nothing.

2 NICs.
- One quad port I350 Gigabit Network Connection (1Gbps)
- One dual port FastLinQ QL41000 Series 10/25/40/50GbE Controller.
_gabriel said:
firewall between pve and pbs ?
Click to expand...
No, tests aree now done locally from PBS to PBS, via localhost, or IP.
Via localhost : 390MB/s
Via "real" IP : 1,20MB/s. nothing special about this one, netmask /21 excepted. With, or without physical link.
Via "fake" IP with /24 mask : 390MB/s!!

Tried all 3 kernels on this PBS :
6.5.11-7-pve
5.11.22-7-pve
6.2.16-20-pve

Christophe.
 
Well, problem seems solved by adding an IP alias in same subnet than the main one (/21), then bench via this alias was ok!!

Tested again on main IP : ok!

No more bad test, even after a reboot.

I have NO idea where does the problem came from, still there after 2 reboots...

Christophe.
 
Try disabling network offloading on both ends and in the host in case of PBS was inside a PVE.

ethtool -K eth0 tx off gso off gro off
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom