[SOLVED] Proxmox bridge MTU issue

Jens Schleede · Jan 28, 2019

Hello,

I cannot add any new bridges to my cluster neither from the webGUI nor CLI.
The error messages I always get are like:
"interface 'vmbr270' - mtu 1500 is lower than 'eno1' - mtu 9000 (500)"
varying by vmbr* randomly.

Here eno1 has mtu 9000 and vmbr270 also has mtu 9000 as seen by "ip link".
So I don't understand what's the problem here?

Best regards
Jens

dcsapak · Jan 28, 2019

does vmbr270 have mtu 9000 configured in /etc/network/interfaces?

Jens Schleede · Jan 28, 2019

Not explicitly. I think they inherent the mtu of the bridge_port.
I'll try the explicit version and report back.

Jens Schleede · Jan 29, 2019

What's the correct way to set MTU size for a bridge in /etc/network/interfaces?

Stoiko Ivanov · Jan 29, 2019

adding a : `mtu 9000` line to the bridge stanza ought to do the trick:

Code:

auto vmbr1
iface vmbr1 inet manual
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    mtu 9000

Jens Schleede · Jan 30, 2019

Great, that worked. Thanks

Stoiko Ivanov · Jan 30, 2019

Nice - glad to hear! Please mark the post as SOLVED - so that others know what to expect. Thanks!

alatteri · Sep 20, 2019

Would it still be necessary to set MTU 9000 in the guest also, if you set it in the host bridge?

Stoiko Ivanov · Sep 20, 2019

Usually yes - since the guest OS normally also assumes a MTU of 1500

spirit · Sep 20, 2019

alatteri said:
Would it still be necessary to set MTU 9000 in the guest also, if you set it in the host bridge?

if you have mtu 9000 on bridge, and 1500 on guest -> 1500 (but ok)
if you have mtu 1500 on bridge, and 9000 on guest -> 1500 but with bad fragmentation

(I personnaly setup mtu 9000 on all my bridges and network switch, then setup mtu in guest between 1500-9000, depending on the application need)

Tekuno-Kage · Jan 28, 2020

I'm confronting similar message:

interface 'vmbr100' - mtu 9216 is lower than 'vmbr0' - mtu 9218 (500)

The configuration and implementation had no issues, everything is working if configure on shell, even can applying using the new Apply Configuration button. The issue probably is a configuration check just for the GUI. what is strange is that complains about an mtu size that fit on a bigger one, that mean that the generated traffic will be lower than the MTU Path.

The network solution is using QinQ therefore on the linux switch we are implementing this concept

L2 topology: PVE-01 (vmbr101 <-> (vmbr0.101|vmbr0)) <Trunk Link> Physical L2 Switch <Trunk Link> ((vmbr0|vmbr0.101) <-> vmbr101) PVE-02
In summary all traffic (including tagged frames on vmbr101 are encapsulated on vlan 101 of the vmbr0.

L1 Simplified Network Topology: PVE-01 <-> L2 Switch (MTU 9216) <-> PVE-02

Path Verification with Jumbo MTU Set:

root@PVE-01:~# ping 10.196.101.12 -M do -c 4 -s 9190
PING 10.196.101.12 (10.196.101.12) 9190(9218) bytes of data.
9198 bytes from 10.196.101.12: icmp_seq=1 ttl=64 time=0.259 ms
9198 bytes from 10.196.101.12: icmp_seq=2 ttl=64 time=0.251 ms
9198 bytes from 10.196.101.12: icmp_seq=3 ttl=64 time=0.208 ms
9198 bytes from 10.196.101.12: icmp_seq=4 ttl=64 time=0.155 ms

--- 10.196.101.12 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 47ms
rtt min/avg/max/mdev = 0.155/0.218/0.259/0.042 ms
root@PVE-01:~# ping 10.196.101.12 -M do -c 4 -s 9192
PING 10.196.101.12 (10.196.101.12) 9192(9220) bytes of data.
ping: local error: Message too long, mtu=9218
ping: local error: Message too long, mtu=9218
ping: local error: Message too long, mtu=9218
ping: local error: Message too long, mtu=9218

--- 10.196.101.12 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 46ms

root@PVE-01:~#

Package versions:
proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-2
pve-container: 3.0-16
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

This behavior could be improved? If additional information is require let me know.

Regards,

spirit · Jan 29, 2020

@Tekuno-Kage
can you provide your /etc/network/interfaces config ?

the 9216 vs 9218 could be from qinq 4bytes overhead.

spirit · Jan 29, 2020

also the message

interface 'vmbr100' - mtu 9216 is lower than 'vmbr0' - mtu 9218 (500)

seem to be strange, because is like you have a

Code:

vmbr100 (mtu 9216)
      bridge-ports vmbr0  (mtu 9218)

with your /etc/network/interfaces I can verify is something is not wrong in proxmox code verification

Tekuno-Kage · Feb 7, 2020

spirit said:
also the message

interface 'vmbr100' - mtu 9216 is lower than 'vmbr0' - mtu 9218 (500)

seem to be strange, because is like you have a

Code:

vmbr100 (mtu 9216) bridge-ports vmbr0 (mtu 9218)

with your /etc/network/interfaces I can verify is something is not wrong in proxmox code verification

Sorry for the delay....

Here is the requested file:

Code:

root@GUYNPR1-PVE-NAF1:~# more /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!


auto lo
iface lo inet loopback

auto dummy1
iface dummy1 inet static
        address 198.51.100.11
        netmask 32
        pre-up ip link add dummy1 type dummy
#Device ID

iface eno4 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF2 RmtInt:Eth1/21

iface eno3 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF1 RmtInt:Eth1/21

iface eno1 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF1 RmtInt:Eth1/1

iface eno2 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF2 RmtInt:Eth1/1

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9218
#RmtInt:Po101 Cmnt:10GbE_Links

auto bond1
iface bond1 inet manual
        bond-slaves eno3 eno4
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9216
#RmtInt:Po201 Cmnt:1GbE_Links

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9216
#10GbE Uplinks

auto vmbr0.22
iface vmbr0.22 inet static
        address  10.196.101.11
        netmask  24
        mtu 9000

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9216
#1GbE Uplinks

auto vmbr100
iface vmbr100 inet manual
        bridge-ports vmbr0.100
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
#Internet Bridge

root@GUYNPR1-PVE-NAF1:~#

The bond mtu of 9218 is the higher MTU, that I was able to validate against the physical switches. That is the reason I post the successful pings above.

Even so I end-up with a lower mtu for the vmbr0 an mtu of 9216 to make it consistent with the physical switch mtu (even that I leave the bond/interface mtu at 9218), smalles frames (vmbr0 sourced frames) could traverse traverse over the bond (biggest mtu).
For traffic source from the PVE Host itself I'm using 9000 MTU, from a QinQ switch (for some tenants) I allow the 9000 MTU, and for some other traffic attach to vmbr0 could go upto 9216.

Today (6Feb2020), I upgrade the environment and their are very cool feature that allow to modify the MTU on the GUI (That is great), but I still getting the same errors.

P.S.

I run FRRouting on the PVEs, with OSPF active. I use the install the version from the Proxmox repos...

spirit · Feb 7, 2020

Thanks, I'll test your conf (seem to be valid indeed), maybe it's a proxmox bug.
(and yes, I have added mtu to gui recently. more new options are coming soon too, like vlan management)

spirit · Feb 7, 2020

Ok, I'm able to reproduce.
I have found 2 bug:

indeed, the mtu check is reversed between bridge - bridge-ports
and also, another bug when the bridge-port is a tagged vlan interface.

I'll try to fix that today.

thanks for the report

spirit · Feb 7, 2020

Hi, can you test:
http://odisoweb1.odiso.net/libpve-common-perl_6.0-12_all.deb

?
It should be fixed

Tekuno-Kage · Feb 7, 2020

spirit said:
Hi, can you test:
http://odisoweb1.odiso.net/libpve-common-perl_6.0-12_all.deb

?
It should be fixed

I'm downloading to validate. Let you know. Thanks!!!

spirit · Feb 10, 2020

patches have been applied to proxmox git, so it should be available soon.
Don't hesitate to report if you still have bugs

Tekuno-Kage · Feb 10, 2020

Tekuno-Kage said:
I'm downloading to validate. Let you know. Thanks!!!

You got a fix! That work like a charm.

Is too much to ask, what are the changes? What is the commit on the git, so I can learn how to work on the git files and if possible directly contribute by summit git push request?

[SOLVED] Proxmox bridge MTU issue

New Member

Attachments

Proxmox Staff Member

New Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Distinguished Member

Renowned Member

Distinguished Member

Distinguished Member

Renowned Member

Distinguished Member

Distinguished Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member