[SOLVED] Should MTU 9000 work on a bridge? [edit, yes it should]

T

Thomas Stone

Renowned Member
Proxmox Subscriber
Mar 8, 2016
75
6
73
Hello! I noticed I was getting horrible internet download speed on two of my Debian Containers (CT) on my Proxmox host while the host itself could download fine (connection is 1G down / 100M up fibre). Specifically, around 10 Megabits/sec download in the CT versus 800 Megabits/sec on the host (via iPerf3 to a 'nearby' VPS I control that gives pretty good speeds). Upload speed was not affected, was getting full 100 Mbps upload both CT and host. MTU of the bridge was set to 9000 (matching the hardware nic setting) and ip in the container showed MTU 9000 for the CT's nic.

After considerable fiddling around, I found that changing the CT to a different bridge (on a different NIC) with 1500 MTU solved the problem. The easy fix then was to change the bridge MTU to 1500 instead of 9000. The MTU of the NIC itslef (a 10-Gig Intel PCI card connected via Thunderbolt to the Intel NUC the proxmox host runs on) is still 9000.

The switch in question and all the computers on that LAN are set up for jumbo packets, MTU 9000.

The solution I arrived at is probably fine .. further testing will happen, but if it works then it works. But I did want to post to see if anyone had any thoughts of other things to try to fix it given that MTU of 9000 for the bridge seems like it is the correct setting. Thank you!
 
Last edited:
Jumbo packets or MTU 9000 are usually not related to internet connection download speeds. It’s more an option for large file transfers in local networks so running the vmbr on 1500 is the standard method. And keep in mind that MTUs usually are set to a value which includes the standard size plus vlan headers. So 1500 is 1496bytes + 4bytes for vlan. PPPoE usually 1492 (+8bytes overhead), WireGuard 1420, etc.

Regarding large MTU: 9000bytes payload + 14bytes (ethernet header) + 4bytes (crc) + 4bytes (optional for vlan). On the *switch* side you should always set a large MTU than 9000. For example NETGEAR (managed): 9216, Cisco Catalyst: 9216-9220, etc. Otherwise you‘ll get framedrops.
 
  • Like
Reactions: MarkusKo
Thank you, cwt, for the information. Indeed the reason I have MTU set to 9000 on this 10G LAN is to move large files as fast as possible (on the LAN). In this case, mostly between the Debian CT and a Mac workstation.

One thing that I may have observed incorrectly and said in my original post is that the proxmox host was not affected. However, now I realise that since it has multiple NICs it actually was using a different bridge (at 1500 MTU) for itself. I'd have to change the default route (gateway) to the affected vmbr at 9000 and test again.

It seems to me that I would not be hitting any MTU limits when downloading from internet, because for internet downloads my router is getting data coming to it from the internet server using its PPPoE MTU and those packets are getting sent to the MTU 9000 switch -- each packet should be more than 80% empty! The switch for the 10G computers is a 4 port Microtik. I could check if it allows MTU over 9000 if somehow this is a factor that I don't understand yet!
 
... ok, confirmed that this affects host networking same as CT networking. iperf3 download speed from internet server I control (pings at 30ms away) with MTU 9000 gives around 6.5 Mbits/s ... change bridge MTU to 1500 gives around 900 Mbits/s. Couldn't be much bigger difference!

I also confirmed the Mac on the 10G / 9000 MTU LAN (same switch) can download near a gigabit/sec so no problem there.

/etc/network/interfaces on Proxmox host (with vmbr2 set to 1500 instead of 9000 which is what I was trying before):

Code: 
auto lo
iface lo inet loopback

iface enp86s0 inet manual

iface enp87s0 inet manual

iface enp88s0 inet manual

iface ens3f0 inet manual
    mtu 9000

iface ens3f1 inet manual
    mtu 9000

auto vmbr0
iface vmbr0 inet static
    address 10.0.1.9/24
    gateway 10.0.1.1
    bridge-ports enp87s0
    bridge-stp off
    bridge-fd 0
#Home Lan Bridge

auto vmbr2
iface vmbr2 inet static
    address 172.16.16.9/24
    bridge-ports ens3f0
    bridge-stp off
    bridge-fd 0
    mtu 1500
#Studio Lan Bridge

auto vmbr1
iface vmbr1 inet static
    address 192.168.1.9/24
    bridge-ports enp88s0
    bridge-stp off
    bridge-fd 0
#Wifi Lan Bridge
 
Last edited:
To get large MTU working you‘ll need to set MTU 9000 on

- the parent interface of vmbrX
- the vmbrX itself
- within the LXC‘s config (add mtu=9000 at the end of -> net0: name=eth0,bridge=vmbr0,hwaddr=...,type=veth,mtu=9000)
- all involved switch ports and trunks (on the switch side better larger than 9000, 9216 for example)
- within the Mac‘s network card/interface

If one part of this chain doesn‘t „speak“ large MTU you‘ll run into drops.
 
  • Like
Reactions: Thomas Stone
Hello spirit. Thanks for chiming in. Maybe I was unclear, I am not trying to achieve jumbo frames across the internet. I am trying to run MTU 9000 on my 10G LAN (and that LAN's NIC in the router is also set to 9000). The router should take care of chanaging the packet size between 9000 and 1500 and vice-versa.

MTU 9000 is running fine from my Mac to the same 10G switch (same as the Proxmox machine).

So far it only fails to work when the vmbr is set to 9000. I guess my next test would be to bring up the pysical NIC (turning off the bridge temporarily) to see if the Proxmox host can work correctly that way.
 
UPDATE - Changing from Proxmox itself using the bridge to Proxmox using the NIC directly, I get same slow downloads from internet server (around 6 Megabits/s). Assuming I exectuted the networking change correctly, then I have falsely accused the bridge of being the source of the problem.

If I run iPerf between Proxmox and the router, I get full Gigabit speed (don't expect 10G because the NIC in the router is 1G). This is through the bridge or not, same.

Must be some NIC issue in Proxmox, or the switch port misbehaving (becuase the Mac computer on same switch and router NIC is fine).

On Proxmox:

Code: 
# ip -s link show vmbr2
7: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    RX:    bytes  packets errors dropped  missed   mcast          
    454565279403 31487372      0       0       0 2629155
    TX:    bytes  packets errors dropped carrier collsns          
    199279644430 22143544      0       0       0       0
root@pve1:~# ip -s link show ens3f0
4: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    RX:     bytes   packets errors dropped  missed   mcast          
    1018140202225 700782403      0       0   23823 2605047
    TX:     bytes   packets errors dropped carrier collsns          
     450780252598 418010864      0       0       0       0
    altname enp4s0f0

I'm not sure what other commands to show me more info. This looks ok to me...

Code: 
auto lo
iface lo inet loopback

iface enp86s0 inet manual

iface enp87s0 inet manual

iface enp88s0 inet manual

iface ens3f0 inet manual
    mtu 9000

iface ens3f1 inet manual
    mtu 9000

auto vmbr0
iface vmbr0 inet static
    address 10.0.1.9/24
    bridge-ports enp87s0
    bridge-stp off
    bridge-fd 0
#Home Lan Bridge

auto vmbr2
iface vmbr2 inet static
    address 172.16.16.9/24
    gateway 172.16.16.1
    bridge-ports ens3f0
    bridge-stp off
    bridge-fd 0
    mtu 9000
#Studio Lan Bridge

auto vmbr1
iface vmbr1 inet static
    address 192.168.1.9/24
    bridge-ports enp88s0
    bridge-stp off
    bridge-fd 0
#Wifi Lan Bridge
 
Last edited:
Ok... just to close the loop here. The switch seemed fine, rebooting it didn't fix anything. I also observed the same slow internet download behaviour on my Proxmox Backup Server (both it and my Proxmox VE box have 10G ethernet NICs and were attempting 9000 MTU).

My Mac workstation did not exhibit this problem.

With MTU 9000 I could get 9.9 Gbps with iperf3 on the LAN between Mac and either Proxmox box or between the two Proxmox boxes. With MTU 1500 I could get 9.4 Gbps. So my 'fix' is to give up on MTU 9000 / Jumbo Packets on this 10G LAN since the speed difference is something I will never notice in practice and it solves this strange issue.

I get the same speed from a container on the bridge as I do from the host so my implication that the bridge might be the problem was unfounded!
 
Last edited:
You must log in or register to reply here.
Top Bottom