Slow backup performance

ntimo · Jul 12, 2020

Hello,
I am trying to backup my virtual machines to a pbs server with 2x 3 TB Enterprise HDDs. When I start the backup process on my PVE Server the transfer speed of the backup data is around 8 MiB/s. I have already tested the network connection between the two servers and that doesn't seems to be the problem (here is an iperf3 test):

Code:

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.03 GBytes   881 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  1.03 GBytes   878 Mbits/sec                  receiver

I also did a local storage benchmark on the pbs server as described here: https://forum.proxmox.com/threads/how-fast-is-your-backup-datastore-benchmark-tool.72750/

Code:

Uploaded 514 chunks in 5 seconds.
Time per request: 9756 microseconds.
TLS speed: 429.90 MB/s
SHA256 speed: 384.45 MB/s
Compression speed: 1253.74 MB/s
Decompress speed: 4510.64 MB/s
AES256/GCM speed: 1201.71 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 429.90 MB/s (73%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed  │ 384.45 MB/s (18%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1253.74 MB/s (58%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 4510.64 MB/s (56%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1201.71 MB/s (32%) │
└───────────────────────────────────┴────────────────────┘

Do you have any idea why the backup is so slow? This is really a huge bootle neck for me since I have VMs that are a few hundred gb, and doing to the first backup this way would take ages to finish, also restore operations are not that fast.

Servers:

Code:

PBS Server:
Core i7 3770
16 GB DDR3 RAM
2x WDC_WD3000FYYZ-01UL1B2 3TB

Thanks for your help.

t.lamprecht · Jul 12, 2020

Hi,

ntimo said:
Do you have any idea why the backup is so slow?

On what storage are the VMs disks on? Maybe that's a bottleneck for the initial backup?

ntimo · Jul 12, 2020

t.lamprecht said:
On what storage are the VMs disks on? Maybe that's a bottleneck for the initial backup?

The source storage of my pvs host uses two KXD51RUE1T92 TOSHIBA NVME SSDs. The disks in my pve are much faster then the ones in my backup server. So those for sure can't be the bottleneck. Also when I do the backup to a network attached smb share (Hetzner storage box) I get around 150 MiB/s.

DerDanilo · Jul 12, 2020

Did you configure the MTU for the vswitch correctly on all attached devices? The network speed looks like Hetzner vSwitch performance.

ntimo · Jul 12, 2020

DerDanilo said:
Did you configure the MTU for the vswitch correctly on all attached devices? The network speed looks like Hetzner vSwitch performance.

I am actually not using a vswitch to connect the two servers. They just talk to each other using their public IP. On the interface I haven't set any specific MTU.

DerDanilo · Jul 12, 2020

Can you try with a vswitch NIC and an MTU of 1400 for the additional vNIC? I've had some weird network issues that I could resolve by using an MTU of 1400 even for forwarded IPs (not vswitch) at Hetzner, which boosted performance quiet a lot and removed random connectivity issues. Then again this problems are not present on all nodes, might even be an issue at some of the Hetzner DCs.

ntimo · Jul 12, 2020

DerDanilo said:
Can you try with a vswitch NIC and an MTU of 1400 for the additional vNIC? I've had some weird network issues that I could resolve by using an MTU of 1400 even for forwarded IPs (not vswitch) at Hetzner, which boosted performance quiet a lot and removed random connectivity issues. Then again this problems are not present on all nodes, might even be an issue at some of the Hetzner DCs.

Hey, I have now setup a vswitch with MTU 1400 and now the performance is somewhat better but not great yet:

Code:

INFO: status: 3% (816.1 MiB of 20.0 GiB), duration 3, read: 272.0 MiB/s, write: 272.0 MiB/s
INFO: status: 4% (856.1 MiB of 20.0 GiB), duration 6, read: 13.3 MiB/s, write: 13.3 MiB/s
INFO: status: 5% (1.0 GiB of 20.0 GiB), duration 75, read: 2.6 MiB/s, write: 2.6 MiB/s
INFO: status: 6% (1.3 GiB of 20.0 GiB), duration 125, read: 6.3 MiB/s, write: 6.3 MiB/s
INFO: status: 7% (1.5 GiB of 20.0 GiB), duration 128, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: status: 8% (1.7 GiB of 20.0 GiB), duration 139, read: 22.9 MiB/s, write: 22.9 MiB/s
INFO: status: 9% (1.8 GiB of 20.0 GiB), duration 144, read: 16.8 MiB/s, write: 16.8 MiB/s

fabian · Jul 13, 2020

ntimo said:
The source storage of my pvs host uses two KXD51RUE1T92 TOSHIBA NVME SSDs. The disks in my pve are much faster then the ones in my backup server. So those for sure can't be the bottleneck. Also when I do the backup to a network attached smb share (Hetzner storage box) I get around 150 MiB/s.

150MiB/s would be more than line speed of your NIC according to iperf (which I assume is 1Gbit?). not saying that there is not still room for improvement (your limit is probably around ~100 MiB/s). the calculations for old-style vzdump and pbs-style vzdump are not the same, so values are not 1:1 comparable.

fichte · Jul 24, 2020

hi i have mainly the same problem

host A is the client node
host B is the pbs node

host A to host B communication is fast with iperf3 a proxmox-backup-client benchmark is slow

iperf3 hosta to hostb:
Connecting to host public_ip_of_hostb, port 5201
[ 5] local ip_of_hosta port 54966 connected to ip_of_hostb port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 86.5 MBytes 725 Mbits/sec 0 3.19 MBytes
[ 5] 1.00-2.00 sec 108 MBytes 902 Mbits/sec 0 3.19 MBytes
[ 5] 2.00-3.00 sec 108 MBytes 902 Mbits/sec 0 3.19 MBytes
[ 5] 3.00-4.00 sec 108 MBytes 902 Mbits/sec 0 3.19 MBytes
[ 5] 4.00-5.00 sec 106 MBytes 891 Mbits/sec 0 3.19 MBytes
[ 5] 5.00-6.00 sec 108 MBytes 902 Mbits/sec 0 3.19 MBytes
[ 5] 6.00-7.00 sec 108 MBytes 902 Mbits/sec 0 3.19 MBytes
[ 5] 7.00-8.00 sec 108 MBytes 902 Mbits/sec 0 3.19 MBytes
[ 5] 8.00-9.00 sec 108 MBytes 902 Mbits/sec 0 3.19 MBytes
[ 5] 9.00-10.00 sec 92.5 MBytes 776 Mbits/sec 171 2.46 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.01 GBytes 871 Mbits/sec 171 sender
[ 5] 0.00-10.03 sec 1.01 GBytes 868 Mbits/sec receiver

iperf Done.

proxmox-backup-client hosta to hostb:
Uploaded 10 chunks in 16 seconds.
Time per request: 1691462 microseconds.
TLS speed: 2.48 MB/s
SHA256 speed: 1482.63 MB/s
Compression speed: 1507.84 MB/s
Decompress speed: 5041.29 MB/s
AES256/GCM speed: 2675.20 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 2.48 MB/s (0%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed │ 1482.63 MB/s (70%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 1507.84 MB/s (70%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 5041.29 MB/s (63%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 2675.20 MB/s (70%) │
└───────────────────────────────────┴────────────────────┘

maybe someone had the same problem and fixed it

thank you

fichte · Jul 26, 2020

i figured out the problem. it has something to do with 10Gbe network cards, but i dont know where to search, or how to investigate that.

i made a tcpdump but everything looks good

pbs connection 1Gbe to 1Gbe is fast
pbs connection 10Gbe to 1Gbe or 1Gbe to 10Gbe is slow max 2MB/s.

both servers are at hetzner and connect to their public ip address, no vswitch or special network configuration

fabian · Jul 27, 2020

what about iperf from 1G to 10G or vice versa? did you try to ask hetzner what is going on?

fichte · Jul 27, 2020

fabian said:
what about iperf from 1G to 10G or vice versa? did you try to ask hetzner what is going on?

iperf 1G to 10G and vice versa works as expected, as i mentioned 2 posts earlier. yesterday evening i asked hetzner to change the network card to be absolutely sure that its not an hardware issue. but unfortunately that wasnt the case. same results with the new 10Gbe card.

fabian · Jul 27, 2020

that was not very clear since you never specified which interfaces you got on which server

anyhow, we don't do anything special network wise, so I'd still ask hetzner whether they block/ratelimit something in particular. PBS uses http/2 + TLS for client <-> server communication.

fichte · Jul 27, 2020

fabian said:
that was not very clear since you never specified which interfaces you got on which server anyhow, we don't do anything special network wise, so I'd still ask hetzner whether they block/ratelimit something in particular. PBS uses http/2 + TLS for client <-> server communication.

sorry for not being detailed enough. im in contact with finnish hetzner support, but they are not helpful in this case. getting standard answers like "we need an MTR report"

i would like to do another test, id like to put nginx in streams mode in front of the pbs proxy but i cannot configure the listen address in pbs because it listens on all interfaces per default. its hardcoded in the source code. i never got in contact with compiling a rust application. i have a clean buster container and would like to build it from source with cargo but a simple make after installing cargo didnt work out

.

do you have some documentation on how to build proxmox-backup on debian buster?

thank you

fabian · Jul 27, 2020

basically add deb http://download.proxmox.com/debian/devel buster main as repo, clone the proxmox-backup git repository, install all the missing build dependencies and 'make deb'

fichte · Jul 27, 2020

fabian said:
basically add deb http://download.proxmox.com/debian/devel buster main as repo, clone the proxmox-backup git repository, install all the missing build dependencies and 'make deb'

what i mentioned did the trick. i recompiled proxmox-backup-proxy to listen on 127.0.0.1:8007 only and put a nginx in stream mode in front of the proxmox-backup-proxy. now i have full speed on the 10Gbe nic

proxmoxrks · Jul 27, 2020

fichte said:
what i mentioned did the trick. i recompiled proxmox-backup-proxy to listen on 127.0.0.1:8007 only and put a nginx in stream mode in front of the proxmox-backup-proxy. now i have full speed on the 10Gbe nic

1) Is your nginx terminating the TLS?
2) Do you have issue (slow) when restoring?
3) Could you provide instruction on recompiling?
4) Instead of recompiling, could we "pvesm add" with a different port? For example 8017?

[0.0.0.0:8017] -> nginx (stream) proxy -> PBS [localhost:8007]

Thanks.

fichte · Jul 27, 2020

proxmoxrks said:
1) Is your nginx terminating the TLS?
2) Do you have issue (slow) when restoring?
3) Could you provide instruction on recompiling?
4) Instead of recompiling, could we "pvesm add" with a different port? For example 8017?

[0.0.0.0:8017] -> nginx (stream) proxy -> PBS [localhost:8007]

Thanks.

1) not when running in stream mode, then pbs terminates tls but it also works in http proxy mode, then nginx terminates tls
2) yes restoring is still slow
3) yes i could provide this tomorrow
4) i didnt try that

Luckyj · Jul 27, 2020

I'm seeing this exact same issue with my setup. PVE on a server with 10g, backup to remote PBS maxes out around 15-17Mbps (~2MBps). WAN links are 1Gbps up/down ethernet, iperf3 shows near full line speed, both ways between all hosts involved. PBS sync job also hits the same speed limit. Local backups to PBS are normal/expected speeds.

Edit: also happens between a PVE server with 1g, on a different network, going to remote PBS. So I'm not sure if this related to 10g.

Edit 2: PBS is virtual on PVE. Local backups to PBS are fast.

proxmoxrks · Jul 28, 2020

Luckyj said:
I'm seeing this exact same issue with my setup. PVE on a server with 10g, backup to remote PBS maxes out around 15-17Mbps (~2MBps). WAN links are 1Gbps up/down ethernet, iperf3 shows near full line speed, both ways between all hosts involved. PBS sync job also hits the same speed limit. Local backups to PBS are normal/expected speeds.

Edit: also happens between a PVE server with 1g, on a different network, going to remote PBS. So I'm not sure if this related to 10g.

Edit 2: PBS is virtual on PVE. Local backups to PBS are fast.

Did you try this benchmark?
proxmox-backup-client benchmark --repository root@pam@IP:REMOTE_STORAGE

Mine is showing around 7MB/s (TLS) performance.
https://forum.proxmox.com/threads/is-pbs-using-single-thread-only.73612/

Slow backup performance

Member

Proxmox Staff Member

Member

Renowned Member

Member

Renowned Member

Member

Proxmox Staff Member

Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Member

Member

Member

Member