working interface fails when added to bridge

[mikem12]

Hi All,

I have an Intel x540-T2 10gbe ethernet card that works perfectly with Proxmox 7.1-10. That is, until I try to create a Linux Bridge with it.

The interface (enp139s0f1) works at the pve host level as one would expect... ssh, nfs, etc. all work. If I take that interface down and create a Linux Bridge, specifying the interface name as the Bridge Port, assigning the same IP address, netmask, and MTU, things get really weird.

ifconfig shows the interface defined and up. I can ping across it, but that's it. ssh, nfs, https, ftp, all other protocols fail. When I tcpdump the traffic, I can see a packet or two every now and then hit the interface, but it's so slow and sporadic that connections just time out. Traffic fails in both directions... ie. ssh'ing in fails in the same way ssh'ing out does. There are no firewalls involved on the network and the proxmox firewall is disabled.

I've even watched the other interface to see if the traffic was hitting the wrong interface, but no. It's like the packets get lost or ignored simply by making it a bridge.

All in all, it's a pretty simple thing and I'm wondering if there's something more low-level that's broken? I have another bridge on a 1gig interface and I don't have any trouble out of it.

I am stumped and really could use some help. I'm new at proxmox, so if I've missed some details that would be helpful, please let me know and I'll try to get them.

If anyone is able to help, it would be greatly appreciated.

Thanks to you all in advance

 

[mikem12]

Friendly bump... anyone out there have any idea of where I might start looking to solve this one?

 

[spirit]

can you send your /etc/network/interfaces config file ?

 

[mikem12]

This is the current file. The 10gbe interface is not tied to a bridge right now since whenever I do, it breaks.

auto lo
iface lo inet loopback

iface eno2 inet manual
mtu 9000
#10.1.x.x Physical Interface

iface eno3 inet manual

iface eno4 inet manual

iface eno5 inet manual

iface enp144s0f0 inet manual

iface enp144s0f1 inet manual

iface enx42f2e929b549 inet manual

iface enp139s0f0 inet manual

auto enp139s0f1
iface enp139s0f1 inet static
address 10.10.1.101/24
mtu 15000
#10gbe

auto vmbr0
iface vmbr0 inet static
address 10.1.1.101/16
gateway 10.1.1.1
bridge-ports eno2
bridge-stp off
bridge-fd 0
mtu 9000

auto vmbr330
iface vmbr330 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Work 10.3.3.x Network

auto vmbr440
iface vmbr440 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Work 10.4.4.x Network

auto vmbr550
iface vmbr550 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Work 10.5.5.x Network


When I do make it a bridge, the entry looks like this:

auto vmbr1
iface vmbr1 inet static
address 10.10.1.101/24
bridge-ports enp139s0f1
bridge-stp off
bridge-fd 0
mtu 15000

 

[mikem12]

I think I've stumbled across a problem with proxmox on this one. Looking at ifconfig ouput, I just happened to notice the MTU size for the interface was different than that of the bridge (1500 vs 15000) ... even though the definition in /etc/network/interfaces stated otherwise.

In the proxmox GUI, I am able to set the MTU on the gbe interface to 15000, and can set the same when configuring the bridge, but I noticed with ifconfig that the interface MTU was stuck at 1500 and the bridge was at 15000. This explains why the initial TCP SYN - SYN/ACK - ACK traffic passed, but nothing else. It also explains why a basic ping worked... such small packets.

When I tried to set the MTU on the interface at the command line with "ifconfig enp139s0f1 mtu 15000" I got the error "SIOCSIFMTU: Invalid argument"

Using "ip -d link list" I was able to see the max mtu supported is 9710.

Once I set both sides of the connection to an MTU of 9000, traffic flowed without issue.

I see a few problems here...
1) The proxmox GUI doesn't respect the max MTU size for the interface being referenced. It will take the value, valid or not, and not produce any error. Behind the scenes, ifconfig is simply refusing to do it.
2) Proxmox allows the configuration of a bridge with an MTU that does not match the interface MTU it's tied to.
3) The 10GBE Intel driver is limited to 9K MTU whereas on Solaris it's 15K. I don't know if this is a Proxmox problem or a Linux problem.

What I don't understand is why this wasn't a problem with just the native interface (ie not a bridge), even with mismatched MTU's on either end of the connection. It didn't break until the bridge was added and the MTU between the bridge and the interface didn't match. Maybe something in the bridge mechanism handles MTU's differently when passing between internal structures?

Something definitely seems broken with the way proxmox is either configuring interfaces and bridges and/or it's an input-validation problem where proxmox isn't paying attention to what the underlying hardware can support and configuration keys between physical interfaces and virtual interfaces.

 

[thehammer]

Having a similar issue.

Intel X520-DA2

I can set vmbr1 to 9000 and its adapter port 1 to 9000 and traffic flows fine.

I set adapter port 0 to 9000. Everything is still fine.

When i set vmbr0 to 9000 everything goes down. Web Gui, SSH, etc.

Below is my config where everything stops working. Can't figure it out.

root@pve1a:/etc/network# cat interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface ens2f0 inet manual
mtu 9000

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface ens2f1 inet manual
mtu 9000

auto vmbr0
iface vmbr0 inet static
address 192.168.72.9/28
gateway 192.168.72.1
bridge-ports ens2f0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000

auto vmbr1
iface vmbr1 inet static
address 10.0.1.9/24
bridge-ports ens2f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000