Abysmal 10Gbit network speed with MTU 9000

[chryzo]

I have set up 10Gbit network between my three Proxmox hosts for NFS and VM migration. When testing with iperf3, i get nowhere near the link speed with MTU 9000.

All three servers connected to a Mikrotik CRS326-24S+2Q+ switch with 10Gbit DAC.

Any tips on whats wrong or how i can troubleshoot further?

root@hyp2:~# lspci | grep Intel
<-- snip -->
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

root@hyp2:~# ethtool enp2s0f1
Settings for enp2s0f1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

root@hyp2:~# ip a
<-- snip -->
4: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master vmbr400 state UP group default qlen 1000
    link/ether 90:e2:ba:49:49:6d brd ff:ff:ff:ff:ff:ff
    altname enx90e2ba49496d
<-- snip -->
17: vmbr400: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 90:e2:ba:49:49:6d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::92e2:baff:fe49:496d/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
18: vmbr400.400@vmbr400: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 90:e2:ba:49:49:6d brd ff:ff:ff:ff:ff:ff
    inet 172.31.254.102/24 scope global vmbr400.400
       valid_lft forever preferred_lft forever
    inet6 fe80::92e2:baff:fe49:496d/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

root@hyp3:~# ping -M do -s 8972 -c 3 172.31.254.102
PING 172.31.254.102 (172.31.254.102) 8972(9000) bytes of data.
8980 bytes from 172.31.254.102: icmp_seq=1 ttl=64 time=0.654 ms
8980 bytes from 172.31.254.102: icmp_seq=2 ttl=64 time=0.582 ms
8980 bytes from 172.31.254.102: icmp_seq=3 ttl=64 time=0.687 ms

--- 172.31.254.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2049ms
rtt min/avg/max/mdev = 0.582/0.641/0.687/0.043 ms

root@hyp3:~# iperf3 -c 172.31.254.102 -t 30 -P 4
Connecting to host 172.31.254.102, port 5201
[  5] local 172.31.254.103 port 32868 connected to 172.31.254.102 port 5201
[  7] local 172.31.254.103 port 32884 connected to 172.31.254.102 port 5201
[  9] local 172.31.254.103 port 32886 connected to 172.31.254.102 port 5201
[ 11] local 172.31.254.103 port 32896 connected to 172.31.254.102 port 5201
<-- snip -->
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec   423 MBytes   118 Mbits/sec  307            sender
[  5]   0.00-30.01  sec   422 MBytes   118 Mbits/sec                  receiver
[  7]   0.00-30.00  sec   435 MBytes   122 Mbits/sec  270            sender
[  7]   0.00-30.01  sec   434 MBytes   121 Mbits/sec                  receiver
[  9]   0.00-30.00  sec   403 MBytes   113 Mbits/sec  280            sender
[  9]   0.00-30.01  sec   401 MBytes   112 Mbits/sec                  receiver
[ 11]   0.00-30.00  sec   393 MBytes   110 Mbits/sec  279            sender
[ 11]   0.00-30.01  sec   392 MBytes   110 Mbits/sec                  receiver
[SUM]   0.00-30.00  sec  1.62 GBytes   462 Mbits/sec  1136             sender
[SUM]   0.00-30.01  sec  1.61 GBytes   461 Mbits/sec                  receiver

iperf Done.

 

[g.moffatt]

Hi Chryzo

Bit of an interesting one, my curiosity got the better of me. Thanks for posting all that information by the way, was insightful. After having a look into your switch I stumbled upon a few things. Your 462 Mbit/s result caught my eye because MikroTik lists about 454 Mbit/s for CPU bridging on that model, under “Ethernet test results” on the product page. Their test uses 1518-byte frames, so it’s not directly comparable to yours, but it makes offloading worth checking.

Are you running RouterOS on the MikroTik? If so, can you post the output of:

/interface bridge port print

And run this on the switch while iperf is running between the hosts:

/tool profile

You’re looking for an H flag next to the bridge ports connected to your hosts, which indicates hardware offloading is active. If that’s missing and the switch CPU maxes out during the test, it would suggest the traffic is going through the CPU and could explain the low throughput.