I have set up 10Gbit network between my three Proxmox hosts for NFS and VM migration. When testing with iperf3, i get nowhere near the link speed with MTU 9000.
All three servers connected to a Mikrotik CRS326-24S+2Q+ switch with 10Gbit DAC.
Any tips on whats wrong or how i can troubleshoot further?
All three servers connected to a Mikrotik CRS326-24S+2Q+ switch with 10Gbit DAC.
Any tips on whats wrong or how i can troubleshoot further?
Code:
root@hyp2:~# lspci | grep Intel
<-- snip -->
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Code:
root@hyp2:~# ethtool enp2s0f1
Settings for enp2s0f1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Code:
root@hyp2:~# ip a
<-- snip -->
4: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master vmbr400 state UP group default qlen 1000
link/ether 90:e2:ba:49:49:6d brd ff:ff:ff:ff:ff:ff
altname enx90e2ba49496d
<-- snip -->
17: vmbr400: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 90:e2:ba:49:49:6d brd ff:ff:ff:ff:ff:ff
inet6 fe80::92e2:baff:fe49:496d/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
18: vmbr400.400@vmbr400: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether 90:e2:ba:49:49:6d brd ff:ff:ff:ff:ff:ff
inet 172.31.254.102/24 scope global vmbr400.400
valid_lft forever preferred_lft forever
inet6 fe80::92e2:baff:fe49:496d/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
Code:
root@hyp3:~# ping -M do -s 8972 -c 3 172.31.254.102
PING 172.31.254.102 (172.31.254.102) 8972(9000) bytes of data.
8980 bytes from 172.31.254.102: icmp_seq=1 ttl=64 time=0.654 ms
8980 bytes from 172.31.254.102: icmp_seq=2 ttl=64 time=0.582 ms
8980 bytes from 172.31.254.102: icmp_seq=3 ttl=64 time=0.687 ms
--- 172.31.254.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2049ms
rtt min/avg/max/mdev = 0.582/0.641/0.687/0.043 ms
Code:
root@hyp3:~# iperf3 -c 172.31.254.102 -t 30 -P 4
Connecting to host 172.31.254.102, port 5201
[ 5] local 172.31.254.103 port 32868 connected to 172.31.254.102 port 5201
[ 7] local 172.31.254.103 port 32884 connected to 172.31.254.102 port 5201
[ 9] local 172.31.254.103 port 32886 connected to 172.31.254.102 port 5201
[ 11] local 172.31.254.103 port 32896 connected to 172.31.254.102 port 5201
<-- snip -->
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 423 MBytes 118 Mbits/sec 307 sender
[ 5] 0.00-30.01 sec 422 MBytes 118 Mbits/sec receiver
[ 7] 0.00-30.00 sec 435 MBytes 122 Mbits/sec 270 sender
[ 7] 0.00-30.01 sec 434 MBytes 121 Mbits/sec receiver
[ 9] 0.00-30.00 sec 403 MBytes 113 Mbits/sec 280 sender
[ 9] 0.00-30.01 sec 401 MBytes 112 Mbits/sec receiver
[ 11] 0.00-30.00 sec 393 MBytes 110 Mbits/sec 279 sender
[ 11] 0.00-30.01 sec 392 MBytes 110 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.62 GBytes 462 Mbits/sec 1136 sender
[SUM] 0.00-30.01 sec 1.61 GBytes 461 Mbits/sec receiver
iperf Done.