[SOLVED] 10Gb link not running at full speed

B

bobmc

Renowned Member
May 17, 2018
718
122
88
67
I have just installed a 10Gb dual NIC into my server and assigned port 1 to Vmbr10 and port 2 has a static IP assigned which I intend to be used for NFS and SMB traffic.
Both ports are going into a 10Gb switch which is connected to a second 10Gb switch which has my workstation connected. All links are reported to be running at 10G by the switches. All IP addresses are in the same subnet.

Code: 
ethtool enp4s0f0
Settings for enp4s0f0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x00002000 (8192)
                               hw
        Link detected: yes

iperf3 tests between workstation to an LXC container on Vmbr10 run at the full speed (although I do have to run 3 parallel connections to get the full bandwith)



Code: 
.\iperf3 -c 10.100.10.201 -P 3
Connecting to host 10.100.10.201, port 5201
[  4] local 10.100.10.20 port 50125 connected to 10.100.10.201 port 5201
[  6] local 10.100.10.20 port 50126 connected to 10.100.10.201 port 5201
[  8] local 10.100.10.20 port 50127 connected to 10.100.10.201 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   419 MBytes  3.52 Gbits/sec
[  6]   0.00-1.00   sec   364 MBytes  3.05 Gbits/sec
[  8]   0.00-1.00   sec   329 MBytes  2.76 Gbits/sec
[SUM]   0.00-1.00   sec  1.09 GBytes  9.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec   415 MBytes  3.48 Gbits/sec
[  6]   1.00-2.00   sec   363 MBytes  3.05 Gbits/sec
[  8]   1.00-2.00   sec   342 MBytes  2.86 Gbits/sec
[SUM]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec   413 MBytes  3.47 Gbits/sec
[  6]   2.00-3.00   sec   368 MBytes  3.09 Gbits/sec
[  8]   2.00-3.00   sec   334 MBytes  2.81 Gbits/sec
[SUM]   2.00-3.00   sec  1.09 GBytes  9.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   411 MBytes  3.44 Gbits/sec
[  6]   3.00-4.00   sec   369 MBytes  3.10 Gbits/sec
[  8]   3.00-4.00   sec   337 MBytes  2.82 Gbits/sec
[SUM]   3.00-4.00   sec  1.09 GBytes  9.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec   412 MBytes  3.45 Gbits/sec
[  6]   4.00-5.00   sec   371 MBytes  3.11 Gbits/sec
[  8]   4.00-5.00   sec   340 MBytes  2.85 Gbits/sec
[SUM]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec   420 MBytes  3.52 Gbits/sec
[  6]   5.00-6.00   sec   368 MBytes  3.09 Gbits/sec
[  8]   5.00-6.00   sec   336 MBytes  2.82 Gbits/sec
[SUM]   5.00-6.00   sec  1.10 GBytes  9.43 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec   414 MBytes  3.47 Gbits/sec
[  6]   6.00-7.00   sec   370 MBytes  3.11 Gbits/sec
[  8]   6.00-7.00   sec   336 MBytes  2.82 Gbits/sec
[SUM]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec   414 MBytes  3.47 Gbits/sec
[  6]   7.00-8.00   sec   366 MBytes  3.07 Gbits/sec
[  8]   7.00-8.00   sec   335 MBytes  2.81 Gbits/sec
[SUM]   7.00-8.00   sec  1.09 GBytes  9.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   414 MBytes  3.47 Gbits/sec
[  6]   8.00-9.00   sec   370 MBytes  3.10 Gbits/sec
[  8]   8.00-9.00   sec   334 MBytes  2.80 Gbits/sec
[SUM]   8.00-9.00   sec  1.09 GBytes  9.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec   416 MBytes  3.49 Gbits/sec
[  6]   9.00-10.00  sec   371 MBytes  3.11 Gbits/sec
[  8]   9.00-10.00  sec   338 MBytes  2.83 Gbits/sec
[SUM]   9.00-10.00  sec  1.10 GBytes  9.43 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  4.05 GBytes  3.48 Gbits/sec                  sender
[  4]   0.00-10.00  sec  4.05 GBytes  3.48 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  3.59 GBytes  3.09 Gbits/sec                  sender
[  6]   0.00-10.00  sec  3.59 GBytes  3.09 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  3.28 GBytes  2.82 Gbits/sec                  sender
[  8]   0.00-10.00  sec  3.28 GBytes  2.82 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver

But tests from workstation to port 2 seemed to be capped at 1Gb/s

Code: 
iperf 3.1.3
CYGWIN_NT-10.0 DESKTOP-HNLTT61 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Wed, 27 Jan 2021 20:23:11 GMT
Connecting to host 10.100.10.101, port 5201
      Cookie: DESKTOP-HNLTT61.1611778991.353433.06
      TCP MSS: 0 (default)
[  4] local 10.100.10.20 port 50144 connected to 10.100.10.101 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   113 MBytes   949 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   945 Mbits/sec
[  4]   2.00-3.00   sec   113 MBytes   947 Mbits/sec
[  4]   3.00-4.00   sec   113 MBytes   949 Mbits/sec
[  4]   4.00-5.00   sec   113 MBytes   946 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   947 Mbits/sec
[  4]   6.00-7.00   sec   113 MBytes   949 Mbits/sec
[  4]   7.00-8.00   sec   113 MBytes   947 Mbits/sec
[  4]   8.00-9.00   sec   113 MBytes   948 Mbits/sec
[  4]   9.00-10.00  sec   113 MBytes   947 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.10 GBytes   947 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.10 GBytes   947 Mbits/sec                  receiver
CPU Utilization: local/sender 6.4% (0.8%u/5.6%s), remote/receiver 17.2% (1.4%u/15.8%s)

Finally my network config
Code: 
auto lo
iface lo inet loopback

iface enp4s0f0 inet manual
        mtu 9000

auto enp4s0f1
iface enp4s0f1 inet static
        address 10.100.10.101/24
        mtu 9000

iface enp6s0f0 inet manual

iface enp6s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.100.10.100/24
        gateway 10.100.10.254
        bridge-ports enp6s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#LAN

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp6s0f1
        bridge-stp off
        bridge-fd 0
#WAN

auto vmbr10
iface vmbr10 inet manual
        bridge-ports enp4s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
 
  • Like
Reactions: ProxmoxEasy
OK, so today's update

Networking hardware is HP552SFP dual 10Gb cards which are rebadged Emulex BE3 connected via fibre to a pair of QNAP QSW-M408S switches.

This is my current network setup on my Proxmox host
network.jpg

Currently vmbr10 and vmbr11 each have a host IP and I have a pair of containers running, one with eth0 on vmbr10 and the other with eth0 on vmbr11

iperf3 tests from desktop client to either container get around 6Gb/s
iperf3 tests from desktop client to either vmbr10 or vmbr11 get the same

iperf3 tests from container to desktop client get around 9Gb/s
** but **
iperf3 tests from vmbr to desktop client get around 950MB/s and this seems to lock speeds between client and vmbr to 1Gb in both directions until the nic interface on the desktop is reset, then client to vmbr goes back to 6Gb/s. However, container to client speed is unaffected remaining at 6Gb/s down / 9Gb/s up as does container to client which returns 18Gb/s

So it looks like it may be a hardware/driver issue except that client to container speeds are completely rock solid.
 
I now have two physical clients, a Windows 10 PC and Debian 10 PC, they are both on the same switch and subnet and same network cards

Debian to Windows and Windows to Debian are both the same speed, around 8Gb/s as are Containers

But Proxmox host to Debian or Windows is 1Gb/s while Debian/Windows to Proxmox host is full speed.

Does anyone have an idea what is happening here? And what the solution might be?
 
Figured it out.

Seems the problem is having the default gateway defined on vmbr0 (1Gb port). If I move the gateway onto vmbr10 or vmbr11 then I can get 10Gb speeds from the host in both directions.

network2.JPG
 
  • Like
Reactions: ProxmoxEasy, _gabriel and i_am_jam
bobmc said:
Figured it out.

Seems the problem is having the default gateway defined on vmbr0 (1Gb port). If I move the gateway onto vmbr10 or vmbr11 then I can get 10Gb speeds from the host in both directions.
Click to expand...

Your solution has solved the same issue for me. Thank you!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back