MTU on Physical Interface not Taking

[dontek]

I'm having an interesting issue I have not seen before, and I have not been able to find a similar situation by searching.

On my core NIC on one of my PVE nodes (9.0.10) the MTU setting will not take. I set 9000; I get 1500.



ip a output:

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master brmgmt state UP group default qlen 1000
    link/ether d0:50:99:db:88:61 brd ff:ff:ff:ff:ff:ff
    altname enp66s0f0
    altname enxd05099db8861

If I manually edit /etc/network/interfaces and change the device id from enp66s0f0 to eno1, then MTU 9000 sticks, however, then I have a rogue eno1 device in the web UI Networking for the node alongside the enp66s0f0 entry, and any vlans/bridges mapped to enp66s0f0 break. When things are working, less the MTU being 9000 like I want it, vlans using enp66s0f0 as their raw device also get MTU 1500, despite also being set to MTU 9000 in the web GUI.

If it matters, the NIC in question is a dual-port onboard Intel X550 on an ASRock Rack ROMED8-2T motherboard.

If there are some logs I can include to help, please let me know where to look.

Any ideas on how to troubleshoot this? I'm not having this issue with my other nodes, however, they are on different hardware.

 

[dontek]

I should add, I can successfully set MTU to the desired 9000 by command:

ip link set dev enp66s0f0 mtu 9000

However, this does not survive a reboot. Nor does it survive:

ifreload -a

The entry in /etc/network/interfaces for this device is simply:

iface enp66s0f0 inet manual
        mtu 9000
#core

...and the corresponding bridge:

auto brmgmt
iface brmgmt inet static
        address 172.16.0.6/24
        gateway 172.16.0.1
        bridge-ports enp66s0f0
        bridge-stp off
        bridge-fd 0
        mtu 9000
#management

 

[shanreich]

Any errors in the journal? (journalctl -b) - what does the debug output of ifreload say? (ifreload -avd)

 

[dontek]

Nothing error-looking in journalctl -b output. I see the normal ixgbe driver stuff (cherry-picked):

ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
ixgbe: Copyright (c) 1999-2016 Intel Corporation.
ixgbe 0000:42:00.0: enabling device (0000 -> 0002)
ixgbe 0000:42:00.0: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
ixgbe 0000:42:00.0: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
ixgbe 0000:42:00.0: MAC: 4, PHY: 0, PBA No: 000000-000
ixgbe 0000:42:00.0: d0:50:99:db:88:61
ixgbe 0000:42:00.0: Intel(R) 10 Gigabit Network Connection
ixgbe 0000:42:00.1: enabling device (0000 -> 0002)
ixgbe 0000:42:00.1: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
ixgbe 0000:42:00.1: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
ixgbe 0000:42:00.1: MAC: 4, PHY: 0, PBA No: 000000-000
ixgbe 0000:42:00.1: d0:50:99:db:88:62
ixgbe 0000:42:00.1: Intel(R) 10 Gigabit Network Connection
ixgbe 0000:42:00.0 eno1: renamed from eth0
ixgbe 0000:42:00.1 eno2: renamed from eth1
ixgbe 0000:42:00.1: registered PHC device on eno2
ixgbe 0000:42:00.0 eno1: entered allmulticast mode
ixgbe 0000:42:00.0 eno1: entered promiscuous mode
ixgbe 0000:42:00.0: registered PHC device on eno1
ixgbe 0000:42:00.1 eno2: NIC Link is Up 10 Gbps, Flow Control: None
ixgbe 0000:42:00.0 eno1: NIC Link is Up 10 Gbps, Flow Control: None

I see warnings about the MTU mismatch between vlans and realdev, like this:

/usr/sbin/ifup[3639]: warning: vlan1666: vlan dev mtu 9000 is greater than lower realdev eno1 mtu 1500
/usr/sbin/ifup[3639]: warning: vlan17: vlan dev mtu 9000 is greater than lower realdev eno1 mtu 1500

I also see these lines when I grep MTU

corosync[4666]:   [KNET  ] pmtud: MTU manually set to: 0
corosync[4666]:   [KNET  ] link: Resetting MTU for link 0 because host 2 joined
corosync[4666]:   [KNET  ] link: Resetting MTU for link 0 because host 1 joined
corosync[4666]:   [KNET  ] pmtud: PMTUD link change for host: 1 link: 0 from 469 to 8885
corosync[4666]:   [KNET  ] pmtud: Global data MTU changed to: 8885

ifreload -avd output attached at text file, since it exceeds message length threshold.