Shutdown of VM/CT changes Bridge MTU

[xmartx]

Hello PVE-Community,

I recently setup a three node cluster and I have this weird issue where the shutdown of a VM or CT changes the MTU of the bridge that is used for communication between VM/CTs and also for the cluster link. The MTU is changed to whatever is configured at the primary interface of the host (In this case the default 1500).

As the whole thing is running at Hetzner using their Vswitch feature it is mandatory to keep the MTU at 1400 or the whole node is not able to communicate on the VLAN which breaks the cluster. Here is the relevant config of one node, as long as I don't shutdown a VM/CT everything works flawlessly and if I reboot the corresponding host and thus resetting the MTU everything is fine again. I'd be glad for any help or even an explanation why Proxmox would have to touch the MTU of this bridge...

auto lo
iface lo inet loopback

auto enp98s0f0
iface enp98s0f0 inet static
    address  X.X.X.X
    netmask  255.255.255.255
    gateway  X.X.X.Y
    pointopoint X.X.X.Y

iface enp98s0f0.4000 inet manual
    vlan_raw_device enp98s0f0
    mtu 1400

auto vmbr0
iface vmbr0 inet static
    bridge_ports enp98s0f0.4000
    bridge_waitport 0
    mtu 1400
    address 10.10.23.4
    netmask  255.255.255.0

And here the config of a CT that triggers the MTU change:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 10.10.23.6
    netmask 255.255.255.0
    mtu 1400

 

[spirit]

does it happen only on shutdown ?
can you post you vmid.conf ?
do you use pve-firewall ? (I found a bug recently with mtu and pve-firewall)

 

[xmartx]

So far I have only noticed it on shutdown or better on every shutdown as it is 100% reproducible. Here is one of my confs, it is for a lxc container but I also tested with a VM and it didn't make a difference. I'm not using pve-firewall ie. it is still in defaultconfig as it was after the pve install:

arch: amd64
cores: 48
hostname: db03.xxx.com
memory: 114441
net0: name=eth0,bridge=vmbr0,hwaddr=D6:8B:67:1D:31:4D,ip=10.10.23.6/24,type=veth
net1: name=eth1,bridge=vmbr1,gw=xxx.xxx.xxx.xxx,hwaddr=C2:71:A1:B7:91:A0,ip=xxx.xxx.xxx.xxy/xx,type=veth
ostype: debian
rootfs: vg0:vm-100-disk-0,size=600G
searchdomain: db03.xxx.com
swap: 16384
lxc.mount.entry: /dev/net dev/net none bind,create=dir
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.apparmor.profile: unconfined

Fwiw, as a workaround we set the MTU of the host interface (enp98s0f0 in the first post) to 1400 as well and it seems to run well so far even though there is no headroom if there is a frame using the full mtu coming from one of the bridges...

 

[spirit]

So far I have only noticed it on shutdown or better on every shutdown as it is 100% reproducible. Here is one of my confs, it is for a lxc container but I also tested with a VM and it didn't make a difference. I'm not using pve-firewall ie. it is still in defaultconfig as it was after the pve install:

arch: amd64
cores: 48
hostname: db03.xxx.com
memory: 114441
net0: name=eth0,bridge=vmbr0,hwaddr=D6:8B:67:1D:31:4D,ip=10.10.23.6/24,type=veth
net1: name=eth1,bridge=vmbr1,gw=xxx.xxx.xxx.xxx,hwaddr=C2:71:A1:B7:91:A0,ip=xxx.xxx.xxx.xxy/xx,type=veth
ostype: debian
rootfs: vg0:vm-100-disk-0,size=600G
searchdomain: db03.xxx.com
swap: 16384
lxc.mount.entry: /dev/net dev/net none bind,create=dir
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.apparmor.profile: unconfined

Fwiw, as a workaround we set the MTU of the host interface (enp98s0f0 in the first post) to 1400 as well and it seems to run well so far even though there is no headroom if there is a frame using the full mtu coming from one of the bridges...

mmm, could be related to my patch
https://git.proxmox.com/?p=pve-comm...;hpb=0a3de87e0f68078652ca3293c1bd1cc377c27f9d

it was to fix veth_pair on pve-firewall, but lxc use same code.

can you try to edit
/usr/share/perl5/PVE/Network.pm

replace

my $cmd = "/sbin/ip link add name $veth type veth peer name $vethpeer mtu $bridgemtu";

by

my $cmd = "/sbin/ip link add name $veth mtu $bridgemtu type veth peer name $vethpeer mtu $bridgemtu";


then restart pvedaemon service,

and try to start/stop CT again.


(the patch is already included in pve-common 5.0-43, but I'm not sure it's already in repos)

 

[xmartx]

mmm, could be related to my patch

Nah, nothing to do with the patch, my colleague found out that this is expected behaviour of the "ip" tool and nothing to do with Proxmox. When the VM/CT is shutdown a "ip link del dev $veth" is issued and this sets the bridge mtu to the mtu of the underlying physical interface, in our case enp98s0f0 while ignoring the vlan interface (enp98s0f0.4000) that requires the reduced mtu.
To be honest I'm a bit surprised nobody encountered (or posted about) this before as Proxmox on Hetzner servers seems to be quite popular but maybe not many people use the Vswitch feature yet. We're gonna stick with the host interface on 1400 for the time being as it seems to run well and the transfer rates using TCP look fine with ~80MB/s.

 

[spirit]

Nah, nothing to do with the patch, my colleague found out that this is expected behaviour of the "ip" tool and nothing to do with Proxmox. When the VM/CT is shutdown a "ip link del dev $veth" is issued and this sets the bridge mtu to the mtu of the underlying physical interface, in our case enp98s0f0 while ignoring the vlan interface (enp98s0f0.4000) that requires the reduced mtu.
To be honest I'm a bit surprised nobody encountered (or posted about) this before as Proxmox on Hetzner servers seems to be quite popular but maybe not many people use the Vswitch feature yet. We're gonna stick with the host interface on 1400 for the time being as it seems to run well and the transfer rates using TCP look fine with ~80MB/s.

interesting, I dind't known about that. I'll look if it's possible to make a patch on proxmox when interface is going down, but I'm not they are hook currently in code at vm/ct shutdown.

 

[mira]

No ETA for VM/CT shutdown hooks yet, sorry.

 

[xmartx]

Not sure if a shutdown hook would be of help here as it breaks with the "ip link del" command being issued and we found nothing we could do to get it running again besides recreating the bridge and everything connected to it.