[SOLVED] OVS - Problem with MTU 9000 on VMs assigned to vmbr0

K

KKill

Member
May 20, 2020
6
0
6
I'm investigating the issue what MTU over 1500 doesn't work for me from the (Windows) VM (works fine from the host, and I've set the MTU in adapter properties within the OS).
While looking into it, I've noticed that if VM is assigned to vmbr0 than fw* interfaces will have MTU 1500 but if I change it to vmbr1 it will be 9000. As part of testing I've changed vmbr0 to vmbr2 and when assigned to VM it also had MTU 9000. So this seems like the issue is just with vmbr0 regardless of the actual config.
Is this expected/intended?

Code: 
$cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

iface eno2 inet manual

iface ens3f0 inet manual

iface ens3f1 inet manual

auto enp175s0f0
iface enp175s0f0 inet manual

auto enp175s0f1
iface enp175s0f1 inet manual

auto vlan18
iface vlan18 inet static
        address 10.0.18.13/24
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_mtu 9000
        ovs_options tag=18

auto vlan17
iface vlan17 inet static
        address 10.0.17.13/24
        gateway 10.0.17.1
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_mtu 1500
        ovs_options tag=17
#mgmt

auto vlan3
iface vlan3 inet static
        address 10.0.3.113/24
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_mtu 9000
        ovs_options tag=3
#iSCSI

auto vlan19
iface vlan19 inet static
        address 10.0.19.13/24
        ovs_type OVSIntPort
        ovs_bridge vmbr1
        ovs_mtu 9000
        ovs_options tag=19
#cluster

auto bond0
iface bond0 inet manual
        ovs_bonds enp175s0f0 enp175s0f1
        ovs_type OVSBond
        ovs_bridge vmbr0
        ovs_mtu 9000
        ovs_options other_config:lacp-time=fast lacp=active bond_mode=balance-tcp vlan_mode=native-untagged
#10Gb

auto bond1
iface bond1 inet manual
        ovs_bonds eno1 eno2
        ovs_type OVSBond
        ovs_bridge vmbr1
        ovs_mtu 9000
        ovs_options vlan_mode=native-untagged bond_mode=active-backup
#1Gb

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports bond1 vlan19
        ovs_mtu 9000
#1Gb

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan18 vlan17 vlan3
        ovs_mtu 9000
#10Gb

Code: 
$ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether d0:94:66:87:90:4b brd ff:ff:ff:ff:ff:ff
    altname enp4s0f0
    inet6 fe80::d294:66ff:fe87:904b/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST> mtu 9000 qdisc noop master ovs-system state DOWN group default qlen 1000
    link/ether d0:94:66:87:90:4c brd ff:ff:ff:ff:ff:ff
    altname enp4s0f1
4: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether b4:96:91:34:c5:64 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b696:91ff:fe34:c564/64 scope link
       valid_lft forever preferred_lft forever
5: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether b4:96:91:34:c5:66 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b696:91ff:fe34:c566/64 scope link
       valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 26:26:17:44:8d:17 brd ff:ff:ff:ff:ff:ff
11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d0:94:66:87:90:4b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7437:28ff:feaf:b48/64 scope link
       valid_lft forever preferred_lft forever
12: vlan19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether c6:93:66:ad:0f:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.19.13/24 scope global vlan19
       valid_lft forever preferred_lft forever
    inet6 fe80::c493:66ff:fead:fd8/64 scope link
       valid_lft forever preferred_lft forever
14: bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 92:e5:04:f1:8b:ac brd ff:ff:ff:ff:ff:ff
    inet6 fe80::90e5:4ff:fef1:8bac/64 scope link
       valid_lft forever preferred_lft forever
15: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether d6:0d:0b:e4:17:de brd ff:ff:ff:ff:ff:ff
18: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master fwbr101i0 state UNKNOWN group default qlen 1000
    link/ether 22:62:10:4e:8a:9e brd ff:ff:ff:ff:ff:ff
21: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master fwbr102i0 state UNKNOWN group default qlen 1000
    link/ether 32:96:31:2f:6b:72 brd ff:ff:ff:ff:ff:ff
24: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master fwbr103i0 state UNKNOWN group default qlen 1000
    link/ether f6:33:38:85:ca:e3 brd ff:ff:ff:ff:ff:ff
28: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether b4:96:91:34:c5:64 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dcba:64ff:fee0:d445/64 scope link
       valid_lft forever preferred_lft forever
29: vlan18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether c6:64:bb:8b:f5:5c brd ff:ff:ff:ff:ff:ff
    inet 10.0.18.13/24 scope global vlan18
       valid_lft forever preferred_lft forever
    inet6 fe80::c464:bbff:fe8b:f55c/64 scope link
       valid_lft forever preferred_lft forever
30: vlan17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d2:d8:e8:5f:81:bb brd ff:ff:ff:ff:ff:ff
    inet 10.0.17.13/24 scope global vlan17
       valid_lft forever preferred_lft forever
    inet6 fe80::d0d8:e8ff:fe5f:81bb/64 scope link
       valid_lft forever preferred_lft forever
31: vlan3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ca:55:61:dd:76:96 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.113/24 scope global vlan3
       valid_lft forever preferred_lft forever
    inet6 fe80::c855:61ff:fedd:7696/64 scope link
       valid_lft forever preferred_lft forever
32: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 96:3c:51:0a:f0:a5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::943c:51ff:fe0a:f0a5/64 scope link
       valid_lft forever preferred_lft forever
33: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:57:5b:33:9d:56 brd ff:ff:ff:ff:ff:ff
34: fwln100o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether 0a:c0:bb:fc:bb:d8 brd ff:ff:ff:ff:ff:ff
35: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:68:41:81:71:31 brd ff:ff:ff:ff:ff:ff
36: fwln101o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UNKNOWN group default qlen 1000
    link/ether ba:4d:bd:e5:30:1b brd ff:ff:ff:ff:ff:ff
39: fwbr102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 12:c5:48:63:86:c2 brd ff:ff:ff:ff:ff:ff
40: fwln102o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UNKNOWN group default qlen 1000
    link/ether d6:8b:5d:8b:42:23 brd ff:ff:ff:ff:ff:ff
43: fwbr103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:f4:ac:0c:cf:42 brd ff:ff:ff:ff:ff:ff
44: fwln103o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr103i0 state UNKNOWN group default qlen 1000
    link/ether 96:d5:7a:b3:6d:a4 brd ff:ff:ff:ff:ff:ff

After moving the VM to vmbr1
Code: 
$ip a | grep 103
24: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master fwbr103i0 state UNKNOWN group default qlen 1000
41: fwbr103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
42: fwln103o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master fwbr103i0 state UNKNOWN group default qlen 1000

Code: 
$pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-3-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.4: 6.4-7
pve-kernel-libc-dev: 5.15.12-3
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-5-pve: 5.11.22-10+ocfs2+1
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
openvswitch-switch: 2.15.0+ds1-2
proxmox-backup-client: 2.1.3-1
proxmox-backup-file-restore: 2.1.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1
 
That's strange,
when the vm is started, proxmox copy the bridge or ovs switch mtu (looking at "/sys/class/net/vmbrX/mtu"), and copy it to the tap interfaces && fw* interfaces.


what is the result of "cat /sys/class/net/vmbr0/mtu" ?


Also, if you have changed vmbr0 mtu after vm start, I'll not change the mtu on fw* and tap interfaces.
Do you have tried to switch back from vmbr1 to vmbr0 ? still 1500 mtu ?
 
Last edited:
Code: 
$cat /sys/class/net/vmbr0/mtu
9000

vmbr MTU was not changed

>Do you have tried to switch back from vmbr1 to vmbr0 ? still 1500 mtu ?
Yes, it will change it back to 1500

I've also noticed this behavior on other servers on different clusters (but they're on PVEv7 as well)
 
if you do a stop/start of the vm on vmbr0, do you still have mtu 1500 ?

I'm currently doing some tests, I have found another bug, when you have 2 bridges with differents mtu, it seem than the mtu of the tap interface never change after first start.
I'll dig in the code to see exactly how it's works when you switch bridge.
 
I've tried it again - I've stopped the VM, verified all related interfaces were gone from "ip a", started the VM again and again getting MTU 1500 (VM is connected to the vmbr0)
Code: 
$ip a | grep 103
45: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master fwbr103i0 state UNKNOWN group default qlen 1000
46: fwbr103i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
47: fwln103o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr103i0 state UNKNOWN group default qlen 1000
 
ok, I have found multiple bugs (with ovs+firewall, and also update on mtu on switch).

I have made a patch:
could you try (if you use proxmox7): https://mutulin1.odiso.net/libpve-common-perl_7.1-3_all.deb ?

Code: 
wget https://mutulin1.odiso.net/libpve-common-perl_7.1-3_all.deb
dpkg -i libpve-common-perl_7.1-3_all.deb
systemctl restart pvedaemon
 
I've installed it and after changing the bridge of the vm to vmbr1 and back to vmbr0 mtu stays on 9000! Thank you very much for the fix!
Is this version safe for prod? Or should I wait till it will be delivered via standard repos?
 
KKill said:
I've installed it and after changing the bridge of the vm to vmbr1 and back to vmbr0 mtu stays on 9000! Thank you very much for the fix!
Is this version safe for prod? Or should I wait till it will be delivered via standard repos?
Click to expand...
ok thanks for the test.
I have send a patch to pve-devel mailing list, it should be available soon. (You can use it in production, changes are minor, but be careful that an update could override it, if my patch is not yet applied).
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back