[SOLVED] Very slow networking in guests

[fagi]

Hi all

I was trying to find a solution by using multiple searches, however, i couldn't find a solution to my problem, so i decided to write in here, in hopes for some support.

On my PM host server (PM host is on PVE-OS), i have a 1Gbit/s connection to the internet, which is working well (i get around 950Mbit/s up/down on the host).
However, on the guest machines (all of them, all debian 11 netinst), the speed is at an average of 20Mbit/s. My guests are connected to the internet using a Linux Bridge and static IPs NAT.
Connectivity is good so far, the servers are reachable. However, if i try to download a file from one of the servers, i get a maximum 2MB/s (or ~20Mbit/s).

I will happily provide as much information as required, but i have no idea anymore, where to start. Does anyone know where I could start? From my suspicion, my linux bridge is breaking things down somehow.

The only thing i found is that there may be issues with Realtek NICs, which my server is using. But i unfortunately didn't find a working solution on how to fix that.
I found guides outlining how to disable hardware offloading, which i tried. However, it didn't help at all.

There is no further configuration of the linux bridge, except some port forwardings onto the guests.

I can send you all the configs and whatnot for troubleshooting, so let me know what you need and i'll happily deliver it.

Im thankful for any kind of support

Best regards

edit: The "static IPs" is misleading. It's a NAT environment with only the host having a public ip. the clients within the NATs have static private IPs.

 

[spirit]

do you use virtio nic for yours vm nics ?

 

[fagi]

do you use virtio nic for yours vm nics ?

i can confirm that. i also tried other nics though, (the intel and realtek ones, aswell as the vmware one). all to the same result, where VirtIO NIC performs just ever so slightly better than the other options. currently all guests running are using the VirtIO NIC, though.

 

[fagi]

Sorry for doubleposting, but i can deliver slightly more information, i guess. I was now able to confirm on a test vm, that a second, newly created linux bridge behaves in the exact same way. i have checked on my debian, it has automatically installed the qemu-guest-agent and in the VM options, the relevant setting is enabled (without trim...). i checked with qm agent <vid> ping and it returned without error.

the speed on the second bridge guest is the same (around 20MBit/s) as on the first bridge guests. it feels like theres throttling happening, but neither did i invoke that anywhere on purpose, nor would i know how to find out if it is even the case on proxmox.

edit: didn't know where to attach that, so:

a little summary of iperf3 if that helps:

this is from one of my vms to a public iperf-server:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  51.0 MBytes   428 Mbits/sec    4   3.92 MBytes
[  5]   1.00-2.00   sec  77.5 MBytes   650 Mbits/sec    0   3.92 MBytes
[  5]   2.00-3.00   sec  78.8 MBytes   661 Mbits/sec    0   3.92 MBytes
[  5]   3.00-4.00   sec  78.8 MBytes   661 Mbits/sec    0   3.92 MBytes
[  5]   4.00-4.31   sec  23.8 MBytes   637 Mbits/sec    0   3.92 MBytes

this generally seems to work quite well.
the opposite example, from one of my home debian machines to one of the vms:

[  5]   0.00-1.00   sec  16.1 MBytes   135 Mbits/sec    0   1.29 MBytes
[  5]   1.00-2.00   sec  16.2 MBytes   136 Mbits/sec    0   2.11 MBytes
[  5]   2.00-3.00   sec  16.2 MBytes   136 Mbits/sec    0   2.87 MBytes
[  5]   3.00-4.00   sec  16.2 MBytes   136 Mbits/sec    0   3.00 MBytes

already looks a bit worse.
from my home win10 pc iperf shows this:

[  4]   0.00-1.01   sec  2.12 MBytes  17.7 Mbits/sec
[  4]   1.01-2.01   sec  2.00 MBytes  16.8 Mbits/sec
[  4]   2.01-3.01   sec  1.88 MBytes  15.7 Mbits/sec
[  4]   3.01-4.01   sec  2.12 MBytes  17.9 Mbits/sec
[  4]   4.01-4.24   sec   384 KBytes  13.1 Mbits/sec

so that's way worse.

the interconnection between the vms looks good:

[  5]   0.00-1.00   sec  3.13 GBytes  26.9 Gbits/sec
[  5]   1.00-2.00   sec  1.96 GBytes  16.9 Gbits/sec
[  5]   2.00-3.00   sec  1.96 GBytes  16.8 Gbits/sec
[  5]   3.00-4.00   sec  1.93 GBytes  16.6 Gbits/sec

as well as the connection between the host and the guests:

[  5]   0.00-1.00   sec  3.36 GBytes  28.9 Gbits/sec    0   2.75 MBytes
[  5]   1.00-2.00   sec  3.37 GBytes  28.9 Gbits/sec    0   2.98 MBytes
[  5]   2.00-3.00   sec  3.38 GBytes  29.0 Gbits/sec    0   2.98 MBytes
[  5]   3.00-4.00   sec  3.38 GBytes  29.0 Gbits/sec    0   2.98 MBytes

pm host to public test server:

[  5]   0.00-1.00   sec  53.5 MBytes   449 Mbits/sec    0   4.78 MBytes
[  5]   1.00-2.00   sec  76.2 MBytes   640 Mbits/sec    0   4.78 MBytes
[  5]   2.00-3.00   sec  77.5 MBytes   650 Mbits/sec    0   4.78 MBytes
[  5]   3.00-4.00   sec  77.5 MBytes   650 Mbits/sec    0   4.78 MBytes
[  5]   4.00-5.00   sec  76.2 MBytes   640 Mbits/sec    0   4.78 MBytes

running speedtest-cli on the host:

Testing download speed................................................................................
Download: 935.31 Mbit/s
Testing upload speed......................................................................................................
Upload: 933.65 Mbit/s

shows good results however, running speedtest-cli on a vm:

Testing download speed................................................................................
Download: 31.77 Mbit/s
Testing upload speed......................................................................................................
Upload: 9.59 Mbit/s

gives me this. which matches about up with the 2MB/s download rate i get when i try downloading something from one of my web services.

i dont know if any of this helps, but it seems to be some problem when connecting to the host from outbound. my home internet connection is around 650MBit/s down and around 150MBit/s up

the server's internet connection is symmetrical gigabit.