Proxmox 10Gb/bit network speed is stuck at around 400Mb

aerodomigue · Oct 28, 2024

Hey, I'm having a network issue that's starting to drive me crazy. The server can't maintain a stable throughput above ~300 Mbits/sec (upload file, content, ...). The server is a Proxmox (8.2.7, kernel 6.8.12-2-pve) with a network card (AQC100 10GB).

If I do a local test (10 Gb <-> 2.5 Gb), there’s no problem. But if I try to communicate outside the local network (e.g., to a public iperf server), the speed collapses.

- I’ve already changed the cables between the switches and the router.
- I reset the switch and the router.
- Tested with live ISOs (Ubuntu 20, 22, 24; Fedora 39, etc.).
- Tested iperf3 on the same server with 2 Macs (good speed at 2.5 Gbit).
- No specific network configuration.
- Static IP on each device.

Strangely, if I route the traffic through my Mac, the problem disappears (Proxmox <-> Mac <-> public).

Proxmox to Public

mac to public

proxmox to mac

krikey · Oct 29, 2024

My first (uneducated) guess is perhaps something to do with MTU?

aerodomigue · Oct 29, 2024

i already try with some value (1500, 1300, 9000, ...) and on my macs it's set to 1500
I also try with another network card (same result)

my current network conf:

Code:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet manual

auto enp5s0
iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.100/24
        gateway 192.168.10.254
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
        mtu 1500

krikey · Oct 29, 2024

whats the MTU ~~for your Mac and whats the MTU~~ for your public network?

aerodomigue · Oct 29, 2024

I have check on my mac and i have a MTU set to default 1500 but for my public network I don't know (i need to check with my network provider ?).

krikey · Oct 29, 2024

If it is MTU, you could try the following from your Proxmox CLI

Code:

ping -c 4 -M do -s 1500 ping.online.net

My MTU here is 1472 but your ISP/Network provider may be different.

You could also try adjusting the window size in iperf3

Code:

iperf3 -c ping.online.net -w [64,128,256]

and see if this gives you different results?

aerodomigue · Oct 29, 2024

For the MTU (tested with 1500, 1473, 1472)
work fine with 1472 and not 1473

Code:

root@proxmox:~# ping -c 4 -M do -s 1472 ping.online.net
PING ping.online.net (51.158.1.21) 1472(1500) bytes of data.
1480 bytes from 51.158.1.21 (51.158.1.21): icmp_seq=1 ttl=59 time=11.8 ms
1480 bytes from 51.158.1.21 (51.158.1.21): icmp_seq=2 ttl=59 time=12.2 ms
1480 bytes from 51.158.1.21 (51.158.1.21): icmp_seq=3 ttl=59 time=12.3 ms
1480 bytes from 51.158.1.21 (51.158.1.21): icmp_seq=4 ttl=59 time=11.5 ms

and for the iperf3

Code:

root@proxmox:~# iperf3 -c ping.online.net -w [64,128,256]
Connecting to host ping.online.net, port 5201
[  5] local 192.168.10.100 port 32838 connected to 51.158.1.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   250 MBytes  2.10 Gbits/sec    2   3.73 MBytes       
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec    2   1.94 MBytes       
[  5]   2.00-3.00   sec   116 MBytes   975 Mbits/sec    2   1.02 MBytes       
[  5]   3.00-4.00   sec  85.0 MBytes   713 Mbits/sec    1    788 KBytes       
[  5]   4.00-5.00   sec  66.2 MBytes   556 Mbits/sec    0    850 KBytes       
[  5]   5.00-6.00   sec  55.0 MBytes   461 Mbits/sec    1    649 KBytes       
[  5]   6.00-7.00   sec  53.8 MBytes   451 Mbits/sec    1    499 KBytes       
[  5]   7.00-8.00   sec  43.8 MBytes   367 Mbits/sec    0    559 KBytes       
[  5]   8.00-9.00   sec  46.2 MBytes   388 Mbits/sec    1    434 KBytes       
[  5]   9.00-10.00  sec  38.8 MBytes   325 Mbits/sec    0    505 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   948 MBytes   795 Mbits/sec   10             sender
[  5]   0.00-10.03  sec   945 MBytes   791 Mbits/sec                  receiver

and for the iperf3 (wihout -w [64,128,256])

Code:

Connecting to host ping.online.net, port 5201
[  5] local 192.168.10.100 port 41992 connected to 51.158.1.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   236 MBytes  1.98 Gbits/sec    3   2.64 MBytes       
[  5]   1.00-2.00   sec   175 MBytes  1.47 Gbits/sec    2   1.38 MBytes       
[  5]   2.00-3.00   sec   104 MBytes   870 Mbits/sec    1   1.03 MBytes       
[  5]   3.00-4.00   sec  82.5 MBytes   692 Mbits/sec    1    795 KBytes       
[  5]   4.00-5.00   sec  62.5 MBytes   524 Mbits/sec    0    857 KBytes       
[  5]   5.00-6.00   sec  53.8 MBytes   451 Mbits/sec    1    649 KBytes       
[  5]   6.00-7.00   sec  52.5 MBytes   440 Mbits/sec    0    708 KBytes       
[  5]   7.00-8.00   sec  41.2 MBytes   346 Mbits/sec    1    550 KBytes       
[  5]   8.00-9.00   sec  45.0 MBytes   377 Mbits/sec    0    608 KBytes       
[  5]   9.00-10.00  sec  37.5 MBytes   315 Mbits/sec    1    492 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   889 MBytes   746 Mbits/sec   10             sender
[  5]   0.00-10.03  sec   887 MBytes   742 Mbits/sec                  receiver

krikey · Oct 29, 2024

sorry I should have been more specific with the iperf3 command. the -w switch requires just one value I believe, not all of them in the [] brackets.

Try using different values for -w

aerodomigue · Oct 29, 2024

my fault I didn't check the -w parameter

so i test with some value (64,128,256,2048,20480,81920), the speed is stable but verrry slow

with 64

Code:

root@proxmox:~# iperf3 -c ping.online.net -w 64 -p 5202 -t 3
Connecting to host ping.online.net, port 5202
[  5] local 192.168.10.100 port 58232 connected to 51.158.1.21 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  48.4 KBytes   396 Kbits/sec    0   5.62 KBytes    
[  5]   1.00-2.00   sec  46.1 KBytes   378 Kbits/sec    0   5.62 KBytes    
[  5]   2.00-3.00   sec  47.2 KBytes   387 Kbits/sec    0   5.62 KBytes    
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   142 KBytes   387 Kbits/sec    0             sender
[  5]   0.00-3.03   sec   140 KBytes   379 Kbits/sec                  receiver

with 256

Code:

root@proxmox:~# iperf3 -c ping.online.net -w 256 -p 5202 -t 3
Connecting to host ping.online.net, port 5202
[  5] local 192.168.10.100 port 56578 connected to 51.158.1.21 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  46.1 KBytes   378 Kbits/sec    0   5.62 KBytes    
[  5]   1.00-2.00   sec  43.9 KBytes   359 Kbits/sec    0   5.62 KBytes    
[  5]   2.00-3.00   sec  43.9 KBytes   359 Kbits/sec    0   5.62 KBytes    
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   134 KBytes   366 Kbits/sec    0             sender
[  5]   0.00-3.02   sec   132 KBytes   358 Kbits/sec                  receiver

with 2048

Code:

root@proxmox:~# iperf3 -c ping.online.net -w 2048 -p 5202 -t 3
Connecting to host ping.online.net, port 5202
[  5] local 192.168.10.100 port 59878 connected to 51.158.1.21 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 KBytes   921 Kbits/sec    0   7.07 KBytes    
[  5]   1.00-2.00   sec   110 KBytes   904 Kbits/sec    0   7.07 KBytes    
[  5]   2.00-3.00   sec   110 KBytes   904 Kbits/sec    0   7.07 KBytes    
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   333 KBytes   909 Kbits/sec    0             sender
[  5]   0.00-3.03   sec   333 KBytes   901 Kbits/sec                  receiver

with 20480

Code:

root@proxmox:~# iperf3 -c ping.online.net -w 20480 -p 5202 -t 3
Connecting to host ping.online.net, port 5202
[  5] local 192.168.10.100 port 39420 connected to 51.158.1.21 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.41 MBytes  11.9 Mbits/sec    0   38.2 KBytes    
[  5]   1.00-2.00   sec  1.40 MBytes  11.8 Mbits/sec    0   38.2 KBytes    
[  5]   2.00-3.00   sec  1.38 MBytes  11.6 Mbits/sec    0   38.2 KBytes    
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec  4.20 MBytes  11.7 Mbits/sec    0             sender
[  5]   0.00-3.02   sec  4.18 MBytes  11.6 Mbits/sec                  receiver

with 81920

Code:

root@proxmox:~# iperf3 -c ping.online.net -w 81920 -p 5202 -t 3
Connecting to host ping.online.net, port 5202
[  5] local 192.168.10.100 port 38960 connected to 51.158.1.21 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.99 MBytes  50.3 Mbits/sec    0    163 KBytes    
[  5]   1.00-2.00   sec  6.00 MBytes  50.3 Mbits/sec    0    163 KBytes    
[  5]   2.00-3.00   sec  6.00 MBytes  50.4 Mbits/sec    0    163 KBytes    
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec  18.0 MBytes  50.3 Mbits/sec    0             sender
[  5]   0.00-3.03   sec  17.9 MBytes  49.8 Mbits/sec                  receiver

Update:
with iptraf-ng, i lot of oversized packet

aerodomigue · Oct 30, 2024

I have resolve my probleme, i have set inside /etc/sysctl.conf

Code:

# Enable TCP window scaling to handle large TCP windows
net.ipv4.tcp_window_scaling = 1

# TCP read buffer (min/default/max), with a maximum of 16 MB for high-throughput
net.ipv4.tcp_rmem = 4096 4194304 16777216

# TCP write buffer (min/default/max), with a maximum of 16 MB for high-throughput
net.ipv4.tcp_wmem = 4096 4194304 16777216

# Maximum receive buffer size for sockets
net.core.rmem_max = 16777216

# Maximum send buffer size for sockets
net.core.wmem_max = 16777216

# Increase the max queue of packets for each network interface
net.core.netdev_max_backlog = 250000

# Use TCP BBR congestion control (optional, try if performance issues persist)
net.ipv4.tcp_congestion_control = bbr

in my case `net.ipv4.tcp_congestion_control=bbr` has a really big impact

but I don't know why I need to set the `/etc/sysctl.conf` for each VM I have, any idea?

krikey · Oct 30, 2024

There's a similar thread here https://forum.proxmox.com/threads/10g-ethernet-config-questions.51428/ which may have some information?

aerodomigue · Oct 30, 2024

I read the thread, the problem is not so similar because I have exactly the same problem that I had on the host in the vm ... But if I do an iperf from the VM to the host or the vm to a mac the speed is goot but if it's vm to public the speed drop.

On the other hand, is it normal that the virtio network card in the vm does not have a spec?

Code:

Settings for enp6s18:
    Supported ports: [  ]
    Supported link modes:   Not reported
    Supported pause frame use: No
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: Other
    PHYAD: 0
    Transceiver: internal
    Link detected: yes

Search

Search

Proxmox 10Gb/bit network speed is stuck at around 400Mb

aerodomigue

New Member

krikey

Renowned Member

aerodomigue

New Member

krikey

Renowned Member

aerodomigue

New Member

krikey

Renowned Member

aerodomigue

New Member

krikey

Renowned Member

aerodomigue

New Member

aerodomigue

New Member

krikey

Renowned Member

aerodomigue

New Member