[SOLVED] Proxmox bridge MTU issue

Jan 23, 2019
10
0
1
38
Hello,

I cannot add any new bridges to my cluster neither from the webGUI nor CLI.
The error messages I always get are like:
"interface 'vmbr270' - mtu 1500 is lower than 'eno1' - mtu 9000 (500)"
varying by vmbr* randomly.

Here eno1 has mtu 9000 and vmbr270 also has mtu 9000 as seen by "ip link".
So I don't understand what's the problem here?

Best regards
Jens
 

Attachments

  • proxmox.PNG
    proxmox.PNG
    5.5 KB · Views: 115

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,850
947
163
33
Vienna
does vmbr270 have mtu 9000 configured in /etc/network/interfaces?
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,841
1,035
164
adding a : `mtu 9000` line to the bridge stanza ought to do the trick:
Code:
auto vmbr1
iface vmbr1 inet manual
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    mtu 9000
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,841
1,035
164
Nice - glad to hear! Please mark the post as SOLVED - so that others know what to expect. Thanks!
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,841
1,035
164
Usually yes - since the guest OS normally also assumes a MTU of 1500
 
  • Like
Reactions: alatteri

spirit

Famous Member
Apr 2, 2010
5,622
608
133
www.odiso.com
Would it still be necessary to set MTU 9000 in the guest also, if you set it in the host bridge?


if you have mtu 9000 on bridge, and 1500 on guest -> 1500 (but ok)
if you have mtu 1500 on bridge, and 9000 on guest -> 1500 but with bad fragmentation

(I personnaly setup mtu 9000 on all my bridges and network switch, then setup mtu in guest between 1500-9000, depending on the application need)
 
  • Like
Reactions: alatteri

Tekuno-Kage

Active Member
Jun 1, 2016
34
6
28
42
I'm confronting similar message:
interface 'vmbr100' - mtu 9216 is lower than 'vmbr0' - mtu 9218 (500)

The configuration and implementation had no issues, everything is working if configure on shell, even can applying using the new Apply Configuration button. The issue probably is a configuration check just for the GUI. what is strange is that complains about an mtu size that fit on a bigger one, that mean that the generated traffic will be lower than the MTU Path.

The network solution is using QinQ therefore on the linux switch we are implementing this concept

L2 topology: PVE-01 (vmbr101 <-> (vmbr0.101|vmbr0)) <Trunk Link> Physical L2 Switch <Trunk Link> ((vmbr0|vmbr0.101) <-> vmbr101) PVE-02
In summary all traffic (including tagged frames on vmbr101 are encapsulated on vlan 101 of the vmbr0.

L1 Simplified Network Topology: PVE-01 <-> L2 Switch (MTU 9216) <-> PVE-02

Path Verification with Jumbo MTU Set:
root@PVE-01:~# ping 10.196.101.12 -M do -c 4 -s 9190
PING 10.196.101.12 (10.196.101.12) 9190(9218) bytes of data.
9198 bytes from 10.196.101.12: icmp_seq=1 ttl=64 time=0.259 ms
9198 bytes from 10.196.101.12: icmp_seq=2 ttl=64 time=0.251 ms
9198 bytes from 10.196.101.12: icmp_seq=3 ttl=64 time=0.208 ms
9198 bytes from 10.196.101.12: icmp_seq=4 ttl=64 time=0.155 ms

--- 10.196.101.12 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 47ms
rtt min/avg/max/mdev = 0.155/0.218/0.259/0.042 ms
root@PVE-01:~# ping 10.196.101.12 -M do -c 4 -s 9192
PING 10.196.101.12 (10.196.101.12) 9192(9220) bytes of data.
ping: local error: Message too long, mtu=9218
ping: local error: Message too long, mtu=9218
ping: local error: Message too long, mtu=9218
ping: local error: Message too long, mtu=9218

--- 10.196.101.12 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 46ms

root@PVE-01:~#

Package versions:
proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-2
pve-container: 3.0-16
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2
This behavior could be improved? If additional information is require let me know.

Regards,
 

spirit

Famous Member
Apr 2, 2010
5,622
608
133
www.odiso.com
also the message

interface 'vmbr100' - mtu 9216 is lower than 'vmbr0' - mtu 9218 (500)

seem to be strange, because is like you have a

Code:
vmbr100 (mtu 9216)
      bridge-ports vmbr0  (mtu 9218)

with your /etc/network/interfaces I can verify is something is not wrong in proxmox code verification
 

Tekuno-Kage

Active Member
Jun 1, 2016
34
6
28
42
also the message

interface 'vmbr100' - mtu 9216 is lower than 'vmbr0' - mtu 9218 (500)

seem to be strange, because is like you have a

Code:
vmbr100 (mtu 9216)
      bridge-ports vmbr0  (mtu 9218)

with your /etc/network/interfaces I can verify is something is not wrong in proxmox code verification

Sorry for the delay....

Here is the requested file:

Code:
root@GUYNPR1-PVE-NAF1:~# more /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!


auto lo
iface lo inet loopback

auto dummy1
iface dummy1 inet static
        address 198.51.100.11
        netmask 32
        pre-up ip link add dummy1 type dummy
#Device ID

iface eno4 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF2 RmtInt:Eth1/21

iface eno3 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF1 RmtInt:Eth1/21

iface eno1 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF1 RmtInt:Eth1/1

iface eno2 inet manual
#RmtEqpmnt:GUYNPR1-CS3548X-NAF2 RmtInt:Eth1/1

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9218
#RmtInt:Po101 Cmnt:10GbE_Links

auto bond1
iface bond1 inet manual
        bond-slaves eno3 eno4
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9216
#RmtInt:Po201 Cmnt:1GbE_Links

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9216
#10GbE Uplinks

auto vmbr0.22
iface vmbr0.22 inet static
        address  10.196.101.11
        netmask  24
        mtu 9000

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9216
#1GbE Uplinks

auto vmbr100
iface vmbr100 inet manual
        bridge-ports vmbr0.100
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
#Internet Bridge

root@GUYNPR1-PVE-NAF1:~#

The bond mtu of 9218 is the higher MTU, that I was able to validate against the physical switches. That is the reason I post the successful pings above.

Even so I end-up with a lower mtu for the vmbr0 an mtu of 9216 to make it consistent with the physical switch mtu (even that I leave the bond/interface mtu at 9218), smalles frames (vmbr0 sourced frames) could traverse traverse over the bond (biggest mtu).
For traffic source from the PVE Host itself I'm using 9000 MTU, from a QinQ switch (for some tenants) I allow the 9000 MTU, and for some other traffic attach to vmbr0 could go upto 9216.

Today (6Feb2020), I upgrade the environment and their are very cool feature that allow to modify the MTU on the GUI (That is great), but I still getting the same errors.

P.S.

I run FRRouting on the PVEs, with OSPF active. I use the install the version from the Proxmox repos...
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,622
608
133
www.odiso.com
Thanks, I'll test your conf (seem to be valid indeed), maybe it's a proxmox bug.
(and yes, I have added mtu to gui recently. more new options are coming soon too, like vlan management)
 

spirit

Famous Member
Apr 2, 2010
5,622
608
133
www.odiso.com
Ok, I'm able to reproduce.
I have found 2 bug:

indeed, the mtu check is reversed between bridge - bridge-ports
and also, another bug when the bridge-port is a tagged vlan interface.

I'll try to fix that today.

thanks for the report
 

Tekuno-Kage

Active Member
Jun 1, 2016
34
6
28
42
I'm downloading to validate. Let you know. Thanks!!!

You got a fix! That work like a charm.

Is too much to ask, what are the changes? What is the commit on the git, so I can learn how to work on the git files and if possible directly contribute by summit git push request?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!