[ovs] mtu on several devices are "wrong" after a reboot

[stefan.runkel]

Hey there,

after testing the behavior a bit, there are probably two bugs I noticed.
One "bug" is resolved with an ifreload or with ifup -a.

1. A ovs_bonds Interface have the mtu of 1500 - despite setting it to 9000 (ovs_mtu 9000).
netstat -i (only bond0 and vmbr9)

Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0     1500        0      0      0 0             8      0      0      0 BMRU
[…]
vmbr9     9000      261      0      0 0             8      0      0      0 BMRU

Both bonding devices have a set MTU of 9000 and are properly set after a reboot.
/etc/network/interfaces (only bond0)

auto bond0
iface bond0 inet manual
    ovs_bonds enp194s0 enp197s0f1
    ovs_type OVSBond
    ovs_bridge vmbr9
    ovs_mtu 9000
    ovs_options vlan_mode=native-untagged tag=999 bond_mode=active-backup

Expected behavior:
bond0 should have an MTU of 9000 right after booting up.

2. One ovs vlan port (vlan099) has a set mtu of 9000.
Several other ovs vlan ports are having an mtu of 9000 except of a few, despite of the actual default of 1500.
netstat -i (only vlans)

Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
vlan010_  9000     3950      0      0 0          5706      0      0      0 BMRU
[… all with MTU 9000 …]
vlan099_  9000        2      0      0 0             8      0      0      0 BMRU
vlan100_  1500       25      0      0 0             8      0      0      0 BMRU
vlan101_  1500        0      0      0 0             9      0      0      0 BMRU
vlan999_  1500       15      0      0 0             9      0      0      0 BMRU

Expected behavior:
Every ovs vlan port (except with a set mtu) should have a MTU of 1500.


Using ifup -a resolves the first bug, bond0 is set up with a MTU of 9000.
The second bug is different though. After that every single ovs vlan port is set to MTU 9000.
This is ok - since they all use vmbr9, which is set to MTU 9000 too, but then the default of 1500 is not consistent.

Greetings
Alex


edit: Testing system is pve-manager/7.1-10/6ddebafe - it is nearly completely up2date. The bugs should be in ifconfig2 or ovs or some Proxmox script.

 

[spirit]

can you send you full /etc/network/interfaces ?

 

[stefan.runkel]

There you go - I had to censor IPs, remove some comments and routes as well as some names.
The namechanges (the vlans) were done properly via regex, so you could reuse this config in theory.

Spoiler: /etc/network/interfaces

# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp197s0f0
iface enp197s0f0 inet static
    address 168.119.xxx.xxx/xx
    gateway 168.119.xxx.xxx
#WAN

auto enp194s0
iface enp194s0 inet manual
    post-up ovs-appctl bond/set-active-slave bond0 enp194s0
#Switch-1

auto enp197s0f1
iface enp197s0f1 inet manual
#Switch-2

auto enp197s0f0.4000
iface enp197s0f0.4000 inet manual
    mtu 1400

auto enp197s0f0.4001
iface enp197s0f0.4001 inet manual
    mtu 1400

auto enp197s0f0.4003
iface enp197s0f0.4003 inet manual
    mtu 1400

auto enp197s0f0.4004
iface enp197s0f0.4004 inet manual
    mtu 1400

auto enp197s0f0.4005
iface enp197s0f0.4005 inet manual
    mtu 1400

auto vlan010_xxx
iface vlan010_xxx inet static
    address 172.xxx.xxx.xxx/xx
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=10

auto vlan019_xxx
iface vlan019_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=19

auto vlan020_xxx
iface vlan020_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=20

auto vlan021_xxx
iface vlan021_xxx inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=21

auto vlan022_xxx
iface vlan022_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=22

auto vlan023_xxx
iface vlan023_xxx inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=23

auto vlan024_xxx
iface vlan024_xxx inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=24

auto vlan025_xxx
iface vlan025_xxx inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=25

auto vlan026_xxx
iface vlan026_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=26

auto vlan027_xxx
iface vlan027_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=27

auto vlan028_xxx
iface vlan028_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=28

auto vlan040_xxx
iface vlan040_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=40

auto vlan050_xxx
iface vlan050_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=50

auto vlan060_xxx
iface vlan060_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=60

auto vlan099_xxx
iface vlan099_xxx inet static
    address 192.168.xxx.xxx/xx
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_mtu 9000
    ovs_options tag=99

auto vlan100_xxx
iface vlan100_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=100

auto vlan101_xxx
iface vlan101_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=101

auto vlan999_xxx
iface vlan999_xxx inet manual
    ovs_type OVSIntPort
    ovs_bridge vmbr9
    ovs_options tag=999

auto bond0
iface bond0 inet manual
    ovs_bonds enp194s0 enp197s0f1
    ovs_type OVSBond
    ovs_bridge vmbr9
    ovs_mtu 9000
    ovs_options vlan_mode=native-untagged tag=999 bond_mode=active-backup
#bond Switch-1 + Switch-2

auto vmbr9
iface vmbr9 inet manual
    ovs_type OVSBridge
    ovs_ports bond0 vlan010_xxx vlan019_xxx vlan020_xxx vlan021_xxx vlan022_xxx vlan023_xxx vlan024_xxx vlan025_xxx vlan026_xxx vlan027_xxx vlan028_xxx vlan040_xxx vlan050_xxx vlan060_xxx vlan099_xxx vlan100_xxx vlan101_xxx vlan999_xxx
    ovs_mtu 9000

auto vmbr4000
iface vmbr4000 inet manual
    bridge-ports enp197s0f0.4000
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/conf/enp197s0f0.4000/proxy_arp

auto vmbr4001
iface vmbr4001 inet manual
    bridge-ports enp197s0f0.4001
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/conf/enp197s0f0.4001/proxy_arp

auto vmbr4003
iface vmbr4003 inet manual
    bridge-ports enp197s0f0.4003
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/conf/enp197s0f0.4003/proxy_arp

auto vmbr4004
iface vmbr4004 inet manual
    bridge-ports enp197s0f0.4004
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/conf/enp197s0f0.4004/proxy_arp

auto vmbr4005
iface vmbr4005 inet static
    address 172.xxx.xxx.xxx/xx
    bridge-ports enp197s0f0.4005
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/conf/enp197s0f0.4005/proxy_arp

 

[spirit]

Hi,

why do you defined all theses vlanxxx ovsintport interfaces ? Do you have address ip on it (maked here for the forum ?)
because you don't need to define vlan ovsintpot, to have vlan working in the vm nic gui.

Also, some of them are "inet static" , some other "inet manual".

inet static -> you need to have an "address ..." configured , "inet manual"---> no address

 

[stefan.runkel]

I've copied most of the config at the beginning and expanded it later.
We need a good overview of all vlans and their purpose, so it's easier to put the VMs NIC to the according vlan.
Some vlans are used for the Proxmox hosts (vlan010 and vlan099), that's why there is an IP assigned

The static/manual difference isn't on purpose - this probably happened some time with the Proxmox WebUI, which was used often back then (before I've gotten involved).
I can change most of them to "manual" (those without an IP) - but this shouldn't matter for the mtu-issue.

Greetings
Alex

 

[stefan.runkel]

Hi,

why do you defined all theses vlanxxx ovsintport interfaces ? Do you have address ip on it (maked here for the forum ?)
because you don't need to define vlan ovsintpot, to have vlan working in the vm nic gui.

Also, some of them are "inet static" , some other "inet manual".

inet static -> you need to have an "address ..." configured , "inet manual"---> no address

Ah - sorry for my rudeness.
Thank you for the hint - I should be more thankful

I'll correct that and I'll look for other misconfigurations I might have in the network.

Greetings
Alex

 

[stefan.runkel]

@spirit what mtu will the vlans have, if I remove them from the interfaces-config?
I think they should have the mtu from vmbr9 -> 9000.

 

[spirit]

@spirit what mtu will the vlans have, if I remove them from the interfaces-config?
I think they should have the mtu from vmbr9 -> 9000.

The vm interfaces in vlanX on vmbr9, should have same mtu than vmbr9.

I'm not sure it's related, but I found a bug some weeks ago about mtu setup on vm nic on ovs. (not mtu in /etc/network/interfaces directly, but when proxmox set mtu on vm interfaces && ovs ports when vm is starting)
https://forum.proxmox.com/threads/o...-on-vms-assigned-to-vmbr0.105172/#post-452824

I have provided a patched .deb on this version. (I have send modification to pve-devel mailing list, but it's not yet upstream)

 

[stefan.runkel]

After some severe networking issues with corosync, we had to revert the mtu changes.
We are switching back to 1500 mtu.

The problem is simple. PMTUD (Path MTU Discovery) only works with TCP.
corosync works with UDP - thus MTU will definitely be an issue if there are differences in the network.

We won't try 9000 mtu in the near future again - therefore we won't do more experiments with this.