Slow throughput on PBS running as a VM behind a FTTC connection

[krikey]

I have a locally hosted lab setup consisting of PVE 7.4-15 which in turn is running a VM of PBS 3.0-1 called "backup.local". PVE is also running a few other VMs and LXCs.

I have a relatively slow 60mbit/20mbit FTTC connection to my lab. I've tested this and verified that I'm getting close to these speeds (52mbit/16mbit) using speedtest-cli on the PBS instance.

If I backup any of the local VMs or LXCs to backup.local, I'm getting a throughput of ~100MiB/s according to the log, admittedly its not really leaving the physical server which seems fine.

If i pass through and port redirect 8007 through my local router I can then mount backup.local as a PBS device on a PVE instance hosted in a datacentre, lets call this pve.remote. This datacentre provides a 1Gbps/1Gbps connection, again tested.

When I try to run a backup between pve.remote > backup.local, I'm only getting ~900KiB/s which seems quite a bit lower than the FTTC connection will provide.

When I run backups between pve.remote and a separate PBS server in the Datacentre, I'm also getting good understandable speeds.

Looking at the status of the local PVE and backup.local, there's no evidence of a bottleneck there that I can see as network, CPU and memory all look ok. According to the local lab router, it's using ~7000Kbps of a theoretical max of 53,000Kbps when a backup is running, some of which is other traffic not related to PVE or PBS.

Can anyone help with identifying where the issue might be?

 

[greybeard]

WAG. network latency, router load, mismatched MTU. Run iperf between datacentre and local server.

 

[krikey]

WAG. network latency, router load, mismatched MTU. Run iperf between datacentre and local server.

Using the following iperf3 commands below on both the server and client helped me to narrow down the issue to a specific datacentre server itself as the other datacentre server test exhibited a more appropriate speed. I'm in touch with the DC now to see if they can shed any more light and will update this ticket accordingly if I think it's useful info.

on server (receiving request)

iperf3 -s -p [determine a port if you don't want to use the default 5201]

on client (sending the request)

iperf3 -c [ip address of server] -p [determine a port if you don't want to use the default 5201] -bidir

If either ends are behind a firewall or NAT, don't forget to poke a hole through for the chosen port.

 

[fabian]

how is the latency of your link? the actual backup and restore flow are using HTTP/2.0 , which is quite latency sensitive unfortunately..

could you also maybe try `proxmox-backup-client benchmark --repository ....` both from client to server and on the server itself? this should give you a rough ballpark of HTTP/2.0 and crypto performance that is relevant for PBS operations..

 

[krikey]

how is the latency of your link? the actual backup and restore flow are using HTTP/2.0 , which is quite latency sensitive unfortunately..

could you also maybe try `proxmox-backup-client benchmark --repository ....` both from client to server and on the server itself? this should give you a rough ballpark of HTTP/2.0 and crypto performance that is relevant for PBS operations..

I've run ther benckmark as suggested and these all look ok, at least they seem faster than the throughput to my specific IP address anyway.

 

[fabian]

so this was between pve.remote and backup.local (the problematic combination)? could you also post a backup task log for that combination?

 

[krikey]

Here's the test between pve.remote1 (the slow one) and backup.local



and here's the test between pve.remote2 and backup.local

 

[fabian]

ah, okay. so yeah, it's definitely HTTP/2.0 that is the cause. either somebody is throttling that along the way (intentionally or unintentionally), or the link has a high enough latency to negatively affect throughput..