Shutdown of VM/CT changes Bridge MTU

Discussion in 'Proxmox VE: Networking and Firewall' started by xmartx, Nov 28, 2018.

  1. xmartx

    xmartx New Member

    Joined:
    Nov 28, 2018
    Messages:
    4
    Likes Received:
    0
    Hello PVE-Community,

    I recently setup a three node cluster and I have this weird issue where the shutdown of a VM or CT changes the MTU of the bridge that is used for communication between VM/CTs and also for the cluster link. The MTU is changed to whatever is configured at the primary interface of the host (In this case the default 1500).

    As the whole thing is running at Hetzner using their Vswitch feature it is mandatory to keep the MTU at 1400 or the whole node is not able to communicate on the VLAN which breaks the cluster. Here is the relevant config of one node, as long as I don't shutdown a VM/CT everything works flawlessly and if I reboot the corresponding host and thus resetting the MTU everything is fine again. I'd be glad for any help or even an explanation why Proxmox would have to touch the MTU of this bridge...

    Code:
    auto lo
    iface lo inet loopback
    
    auto enp98s0f0
    iface enp98s0f0 inet static
        address  X.X.X.X
        netmask  255.255.255.255
        gateway  X.X.X.Y
        pointopoint X.X.X.Y
    
    iface enp98s0f0.4000 inet manual
        vlan_raw_device enp98s0f0
        mtu 1400
    
    auto vmbr0
    iface vmbr0 inet static
        bridge_ports enp98s0f0.4000
        bridge_waitport 0
        mtu 1400
        address 10.10.23.4
        netmask  255.255.255.0
    And here the config of a CT that triggers the MTU change:

    Code:
    auto lo
    iface lo inet loopback
    
    auto eth0
    iface eth0 inet static
        address 10.10.23.6
        netmask 255.255.255.0
        mtu 1400
    
     
  2. spirit

    spirit Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 2, 2010
    Messages:
    3,196
    Likes Received:
    110
    does it happen only on shutdown ?
    can you post you vmid.conf ?
    do you use pve-firewall ? (I found a bug recently with mtu and pve-firewall)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. xmartx

    xmartx New Member

    Joined:
    Nov 28, 2018
    Messages:
    4
    Likes Received:
    0
    So far I have only noticed it on shutdown or better on every shutdown as it is 100% reproducible. Here is one of my confs, it is for a lxc container but I also tested with a VM and it didn't make a difference. I'm not using pve-firewall ie. it is still in defaultconfig as it was after the pve install:

    Code:
    arch: amd64
    cores: 48
    hostname: db03.xxx.com
    memory: 114441
    net0: name=eth0,bridge=vmbr0,hwaddr=D6:8B:67:1D:31:4D,ip=10.10.23.6/24,type=veth
    net1: name=eth1,bridge=vmbr1,gw=xxx.xxx.xxx.xxx,hwaddr=C2:71:A1:B7:91:A0,ip=xxx.xxx.xxx.xxy/xx,type=veth
    ostype: debian
    rootfs: vg0:vm-100-disk-0,size=600G
    searchdomain: db03.xxx.com
    swap: 16384
    lxc.mount.entry: /dev/net dev/net none bind,create=dir
    lxc.cgroup.devices.allow: c 10:200 rwm
    lxc.apparmor.profile: unconfined
    
    Fwiw, as a workaround we set the MTU of the host interface (enp98s0f0 in the first post) to 1400 as well and it seems to run well so far even though there is no headroom if there is a frame using the full mtu coming from one of the bridges...
     
  4. spirit

    spirit Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 2, 2010
    Messages:
    3,196
    Likes Received:
    110

    mmm, could be related to my patch
    https://git.proxmox.com/?p=pve-comm...;hpb=0a3de87e0f68078652ca3293c1bd1cc377c27f9d

    it was to fix veth_pair on pve-firewall, but lxc use same code.

    can you try to edit
    /usr/share/perl5/PVE/Network.pm

    replace

    my $cmd = "/sbin/ip link add name $veth type veth peer name $vethpeer mtu $bridgemtu";

    by

    my $cmd = "/sbin/ip link add name $veth mtu $bridgemtu type veth peer name $vethpeer mtu $bridgemtu";


    then restart pvedaemon service,

    and try to start/stop CT again.


    (the patch is already included in pve-common 5.0-43, but I'm not sure it's already in repos)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. xmartx

    xmartx New Member

    Joined:
    Nov 28, 2018
    Messages:
    4
    Likes Received:
    0
    Nah, nothing to do with the patch, my colleague found out that this is expected behaviour of the "ip" tool and nothing to do with Proxmox. When the VM/CT is shutdown a "ip link del dev $veth" is issued and this sets the bridge mtu to the mtu of the underlying physical interface, in our case enp98s0f0 while ignoring the vlan interface (enp98s0f0.4000) that requires the reduced mtu.
    To be honest I'm a bit surprised nobody encountered (or posted about) this before as Proxmox on Hetzner servers seems to be quite popular but maybe not many people use the Vswitch feature yet. We're gonna stick with the host interface on 1400 for the time being as it seems to run well and the transfer rates using TCP look fine with ~80MB/s.
     
    #5 xmartx, Dec 4, 2018
    Last edited: Dec 4, 2018
  6. spirit

    spirit Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 2, 2010
    Messages:
    3,196
    Likes Received:
    110
    interesting, I dind't known about that. I'll look if it's possible to make a patch on proxmox when interface is going down, but I'm not they are hook currently in code at vm/ct shutdown.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. dlimbeck

    dlimbeck Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2018
    Messages:
    65
    Likes Received:
    2
    No ETA for VM/CT shutdown hooks yet, sorry.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. xmartx

    xmartx New Member

    Joined:
    Nov 28, 2018
    Messages:
    4
    Likes Received:
    0
    Not sure if a shutdown hook would be of help here as it breaks with the "ip link del" command being issued and we found nothing we could do to get it running again besides recreating the bridge and everything connected to it.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice