Jumbo frames (MTU 9000), bond, vlans and bridges

Oct 2, 2018
33
1
11
34
Hi,
i'm trying to enable jumbo frames on our infrastructure in order to gain benefit for Ceph performance, unfortunately there is no way to make jumbo frames works in our situation.

When I set mtu 9000 on network configuration, interfaces are not going up with that MTU but with the default one (1500).

Here is our network configuration:

Bash:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual
    mtu 9000

iface eno2 inet manual
    mtu 9000

iface eno3 inet manual

iface eno4 inet manual

auto ens1f0
iface ens1f0 inet static
    address  10.65.0.10
    netmask  255.255.255.0
#Corosync Ring 1

auto ens1f1
iface ens1f1 inet static
    address  10.66.0.10
    netmask  255.255.255.0
#Corosync Ring 2

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode balance-alb
    mtu 9000
#Bond U - 10 Gb Network

iface bond0.1060 inet manual
    mtu 9000
#Bond T - Ceph Public Network

iface bond0.1061 inet static
    mtu 9000
#Bond T - Ceph Cluster Network

auto bond1
iface bond1 inet manual
    bond-slaves eno3 eno4
    bond-miimon 100
    bond-mode balance-alb
#Bond U - 1 Gb Network

iface bond1.1040 inet manual
#Bond T - DMZ Network

iface bond1.1051 inet manual
#Bond T - USB Network

auto vmbr1050
iface vmbr1050 inet static
    address  10.50.0.200
    netmask  255.255.255.0
    gateway  10.50.0.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    mtu 1500
#Bridge - Cluster Management Network

auto vmbr1060
iface vmbr1060 inet static
    address  10.60.0.10
    netmask  255.255.255.0
    bridge-ports bond0.1060
    bridge-stp off
    bridge-fd 0
    mtu 9000
#Bridge - Ceph Public Network

auto vmbr1061
iface vmbr1061 inet static
    address  10.61.0.10
    netmask  255.255.255.0
    bridge-ports bond0.1061
    bridge-stp off
    bridge-fd 0
    mtu 9000
#Bridge - Ceph Cluster Network

auto vmbr1055
iface vmbr1055 inet manual
    bridge-ports bond1
    bridge-stp off
    bridge-fd 0
#Bridge - VMs Network

auto vmbr1040
iface vmbr1040 inet manual
    bridge-ports bond1.1040
    bridge-stp off
    bridge-fd 0
#Bridge - DMZ Network

auto vmbr1051
iface vmbr1051 inet manual
    bridge-ports bond1.1051
    bridge-stp off
    bridge-fd 0
#Bridge - USB Network

Anyone can help me? Do you have an idea on what is the cause of that problem?

Many thanks
 
You dont need to set MTU on eno1 and eno2 if you have put it in the bond0. My network config looks the same mtu wise. How do you determine that mtu is not correctly set?

You could try to ping your ceph-ips with "ping -M do -S 8972 ip-address" to check if jumbo frames are working. Switch need to be set to highest possible mtu too otherwise it wont work.

Also check your interface eno1,eno2, bond0, bond0.vlan

Code:
root@pve01:~# cat /sys/class/net/bond0/mtu
9000

EDIT:

You also have bond0 on a vmbridge, why did you create vmbridges for the ceph networks? You could also put the ip directly on the bond without the need of a vmbridge. Im not sure seperation of the ceph network are useful if you use the same links (eno1, eno2).
 
Last edited:
I will try not to set it on the underlying interfaces. I'm checking MTU current size with "ip link show" command. Switch are already set with maximum MTU size.
 
I have another question. "auto" should be set on all interfaces? or only on particular one? This could be the cause with MTU not being set?
Also: Ip address should be set, as done, on bridge (vmbr) or should be set on bond (tagged with vlan)?

I tried the configuration with the two fixes above on a test server, this is the new config:

Bash:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto eno3
iface eno3 inet static
    address  10.65.0.12
    netmask  255.255.255.0
#Corosync Ring 1

auto eno4
iface eno4 inet static
    address  10.66.0.12
    netmask  255.255.255.0
#Corosync Ring 2

iface ens1f0 inet manual

iface ens1f1 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves ens1f0 ens1f1
    bond-miimon 100
    bond-mode balance-alb
    mtu 9000
#Bond U - 10 Gb Network

iface bond0.1060 inet manual
    mtu 9000
#Bond T - Ceph Public Network

iface bond0.1061 inet manual
    mtu 9000
#Bond T - Ceph Cluster Network

auto bond1
iface bond1 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode balance-alb
#Bond U - 1 Gb Network

iface bond1.1040 inet manual
#Bond T - DMZ Network

iface bond1.1051 inet manual
#Bond T - USB Network

auto vmbr1050
iface vmbr1050 inet static
    address  10.50.0.202
    netmask  255.255.255.0
    gateway  10.50.0.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    mtu 1500
#Bridge - Cluster Management Network

auto vmbr1060
iface vmbr1060 inet static
    address  10.60.0.12
    netmask  255.255.255.0
    bridge-ports bond0.1060
    bridge-stp off
    bridge-fd 0
    mtu 9000
#Bridge - Ceph Public Network

auto vmbr1061
iface vmbr1061 inet static
    address  10.61.0.12
    netmask  255.255.255.0
    bridge-ports bond0.1061
    bridge-stp off
    bridge-fd 0
    mtu 9000
#Bridge - Ceph Cluster Network

auto vmbr1055
iface vmbr1055 inet manual
    bridge-ports bond1
    bridge-stp off
    bridge-fd 0
#Bridge - VMs Network

auto vmbr1040
iface vmbr1040 inet manual
    bridge-ports bond1.1040
    bridge-stp off
    bridge-fd 0
#Bridge - DMZ Network

auto vmbr1051
iface vmbr1051 inet manual
    bridge-ports bond1.1051
    bridge-stp off
    bridge-fd 0
#Bridge - USB Network

And this is the output of "Ip link show"

Bash:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ec brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ed brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ee brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ef brd ff:ff:ff:ff:ff:ff
6: ens1f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:74 brd ff:ff:ff:ff:ff:ff
7: ens1f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:76 brd ff:ff:ff:ff:ff:ff
8: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether f8:ca:b8:ff:c3:a9 brd ff:ff:ff:ff:ff:ff
9: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1050 state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:74 brd ff:ff:ff:ff:ff:ff
10: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1055 state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ec brd ff:ff:ff:ff:ff:ff
11: vmbr1050: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:74 brd ff:ff:ff:ff:ff:ff
12: vmbr1060: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:74 brd ff:ff:ff:ff:ff:ff
13: bond0.1060@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1060 state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:74 brd ff:ff:ff:ff:ff:ff
14: vmbr1061: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:74 brd ff:ff:ff:ff:ff:ff
15: bond0.1061@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1061 state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:22:85:74 brd ff:ff:ff:ff:ff:ff
16: vmbr1055: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ec brd ff:ff:ff:ff:ff:ff
17: vmbr1040: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ec brd ff:ff:ff:ff:ff:ff
18: bond1.1040@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1040 state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ec brd ff:ff:ff:ff:ff:ff
19: vmbr1051: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ec brd ff:ff:ff:ff:ff:ff
20: bond1.1051@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1051 state UP mode DEFAULT group default qlen 1000
    link/ether 24:6e:96:9b:64:ec brd ff:ff:ff:ff:ff:ff

MTU is being set only on bridge but not on underlying bond and interfaces (ens1f0 e ens1f1)
 
Last edited:
MTU is being set only on bridge but not on underlying bond and interfaces (ens1f0 e ens1f1)

by default, bond inherit mtu from ens1f0 && ens1f1, but not the reverse way.
so better to set it everywhere to be sure. (parents interfaces always need to have equal or bigger mtu than chiild interfaces).

and set also "auto ...", or the config will not be applied.
 
You could try to ping your ceph-ips with "ping -M do -S 8972 ip-address" to check if jumbo frames are working. Switch need to be set to highest possible mtu too otherwise it wont work.
That has to be a lower-case "s" option there.

Code:
ping -M do -s 8972 192.168.1.5
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!