Hi,
I am trying to troubleshoot a network issue in a three-node Proxmox VE cluster. The failure triggers whenever promiscuous mode is enabled on a Linux bridge (vmbr2) that uses an LACP bond (bond2) as its port. As soon as the bridge enters promiscuous mode, connectivity is lost on the Corosync network running over this bond. I have managed to reproduce this behavior manually, without the need to start any virtual machines.
I would like to know if anyone has encountered a similar behavior with Broadcom adapters, the bnxt_en driver, Linux bridge, and 802.3ad bonding, or if there is a known tweak to resolve this.
1. Hardware and software
All three nodes share the exact same hardware and network configuration.
2. Network configuration
The logical setup is identical across all three nodes. The relevant configuration from /etc/network/interfaces is shown below, using dummy IP addresses for privacy reasons.
auto nic3
iface nic3 inet manual
mtu 9000
auto nic5
iface nic5 inet manual
mtu 9000
auto bond2
iface bond2 inet manual
bond-slaves nic3 nic5
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 9000
auto bond2.721
iface bond2.721 inet static
address 192.0.2.1/29
mtu 1500
auto bond2.730
iface bond2.730 inet static
address 198.51.100.1/24
mtu 9000
auto vmbr2
iface vmbr2 inet manual
bridge-ports bond2
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000
In this configuration:
Both physical interfaces normally show up as UP at 25 Gb/s, acting as active, synchronized, and aggregated members of the same LACP aggregator.
3. Issue when starting a VM attached to vmbr2
When I create a virtual machine and connect its network interface to vmbr2, the issue occurs as soon as the VM boots.
The following message appears in the kernel logs:
vmbr2: entered promiscuous mode
Coinciding with the bridge entering promiscuous mode, connectivity on the Corosync link1 network drops completely. Interfaces nic3 and nic5 stop forwarding traffic properly, even though they still report a physical link state of UP and their status remains active within the bond.
I do not observe any physical link down events during this state.
4. Manual reproduction without starting any virtual machines
I can reproduce the failure without starting any VMs by manually forcing promiscuous mode on the bridge itself.
First, I verify that connectivity works when promiscuous mode is enabled only on the bond:
ip link set bond2 promisc on
ping -c 5 -W 1 192.0.2.2
In this case, the ping works fine.
Next, I disable promiscuous mode on the bond and enable it on the bridge:
ip link set bond2 promisc off
ip link set vmbr2 promisc on
ping -c 5 -W 1 192.0.2.2
At this point, connectivity is completely broken. The ping returns Destination Host Unreachable errors and receives no replies.
The kernel logs show:
vmbr2: entered promiscuous mode
To restore connectivity, I have to disable promiscuous mode on the bridge and administratively cycle both physical interfaces:
ip link set vmbr2 promisc off
ip link set nic3 down
ip link set nic3 up
ip link set nic5 down
ip link set nic5 up
After doing this, connectivity is fully restored.
The most critical takeaway here is that the failure can be triggered directly by enabling promiscuous mode on vmbr2, without any virtual machine running. Enabling promiscuous mode on bond2 alone does not replicate the issue in my testing.
5. Alternative test with a dedicated VLAN bridge
I also tested an alternative configuration to connect a virtual machine through a specific VLAN.
I created the VLAN interface bond2.67 and a matching bridge vmbr2.67 attached to it. This bridge does not have VLAN-aware mode enabled. The VM's virtual interface connects directly to vmbr2.67 without any extra VLAN tags inside the VM settings.
With this setup, the VM boots perfectly fine, and the connectivity loss observed with vmbr2 does not occur.
As mentioned in the configuration section, I also tested defining the Corosync and migration interfaces as separate bridges (vmbr2.721 and vmbr2.730), but this did not prevent the drop. Therefore, the issue seems strictly tied to the behavior of the VLAN-aware bridge (vmbr2) when it enters promiscuous mode, rather than just running a VM over bond2.
6. Questions for the community
I would appreciate any insights on the following points:
Thank you in advance for any guidance or experiences you can share!
I am trying to troubleshoot a network issue in a three-node Proxmox VE cluster. The failure triggers whenever promiscuous mode is enabled on a Linux bridge (vmbr2) that uses an LACP bond (bond2) as its port. As soon as the bridge enters promiscuous mode, connectivity is lost on the Corosync network running over this bond. I have managed to reproduce this behavior manually, without the need to start any virtual machines.
I would like to know if anyone has encountered a similar behavior with Broadcom adapters, the bnxt_en driver, Linux bridge, and 802.3ad bonding, or if there is a known tweak to resolve this.
1. Hardware and software
All three nodes share the exact same hardware and network configuration.
- Servers: Dell PowerEdge.
- Network Adapters: Broadcom BCM57504 NetXtreme-E and BCM57414.
- 25 Gb/s network interfaces via SFP28 transceivers.
- Switches: Extreme Networks 7520, configured with MLAG.
- Network driver: bnxt_en.
- Interface firmware: 238.1.168.0/pkg 38.11.70.00.
- Proxmox VE: kernel 7.0.14-23-pve.
- Bonding: IEEE 802.3ad (LACP).
- Transmit hash policy: layer3+4.
2. Network configuration
The logical setup is identical across all three nodes. The relevant configuration from /etc/network/interfaces is shown below, using dummy IP addresses for privacy reasons.
auto nic3
iface nic3 inet manual
mtu 9000
auto nic5
iface nic5 inet manual
mtu 9000
auto bond2
iface bond2 inet manual
bond-slaves nic3 nic5
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 9000
auto bond2.721
iface bond2.721 inet static
address 192.0.2.1/29
mtu 1500
auto bond2.730
iface bond2.730 inet static
address 198.51.100.1/24
mtu 9000
auto vmbr2
iface vmbr2 inet manual
bridge-ports bond2
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000
In this configuration:
- bond2 aggregates nic3 and nic5 via LACP.
- bond2.721 carries Corosync link1.
- bond2.730 is used for live migrations.
- vmbr2 is a VLAN-aware bridge using bond2 as its port, allowing VLANs 2-4094.
Both physical interfaces normally show up as UP at 25 Gb/s, acting as active, synchronized, and aggregated members of the same LACP aggregator.
3. Issue when starting a VM attached to vmbr2
When I create a virtual machine and connect its network interface to vmbr2, the issue occurs as soon as the VM boots.
The following message appears in the kernel logs:
vmbr2: entered promiscuous mode
Coinciding with the bridge entering promiscuous mode, connectivity on the Corosync link1 network drops completely. Interfaces nic3 and nic5 stop forwarding traffic properly, even though they still report a physical link state of UP and their status remains active within the bond.
I do not observe any physical link down events during this state.
4. Manual reproduction without starting any virtual machines
I can reproduce the failure without starting any VMs by manually forcing promiscuous mode on the bridge itself.
First, I verify that connectivity works when promiscuous mode is enabled only on the bond:
ip link set bond2 promisc on
ping -c 5 -W 1 192.0.2.2
In this case, the ping works fine.
Next, I disable promiscuous mode on the bond and enable it on the bridge:
ip link set bond2 promisc off
ip link set vmbr2 promisc on
ping -c 5 -W 1 192.0.2.2
At this point, connectivity is completely broken. The ping returns Destination Host Unreachable errors and receives no replies.
The kernel logs show:
vmbr2: entered promiscuous mode
To restore connectivity, I have to disable promiscuous mode on the bridge and administratively cycle both physical interfaces:
ip link set vmbr2 promisc off
ip link set nic3 down
ip link set nic3 up
ip link set nic5 down
ip link set nic5 up
After doing this, connectivity is fully restored.
The most critical takeaway here is that the failure can be triggered directly by enabling promiscuous mode on vmbr2, without any virtual machine running. Enabling promiscuous mode on bond2 alone does not replicate the issue in my testing.
5. Alternative test with a dedicated VLAN bridge
I also tested an alternative configuration to connect a virtual machine through a specific VLAN.
I created the VLAN interface bond2.67 and a matching bridge vmbr2.67 attached to it. This bridge does not have VLAN-aware mode enabled. The VM's virtual interface connects directly to vmbr2.67 without any extra VLAN tags inside the VM settings.
With this setup, the VM boots perfectly fine, and the connectivity loss observed with vmbr2 does not occur.
As mentioned in the configuration section, I also tested defining the Corosync and migration interfaces as separate bridges (vmbr2.721 and vmbr2.730), but this did not prevent the drop. Therefore, the issue seems strictly tied to the behavior of the VLAN-aware bridge (vmbr2) when it enters promiscuous mode, rather than just running a VM over bond2.
6. Questions for the community
I would appreciate any insights on the following points:
- Is there a known issue in the bnxt_en driver with Broadcom BCM57504 or BCM57414 cards when a Linux bridge enters promiscuous mode?
- Can enabling promiscuous mode on a VLAN-aware bridge cause problems with the adapter's RX filter programming, MAC address management, or frame reception over an LACP bond?
- Are there any known issues related to unicast filter overflow or Broadcom firmware RX mask updates that present these exact symptoms?
- Is there a specific driver, bonding, bridge, or firmware parameter that could prevent this behavior without breaking the single VLAN-aware bridge design into independent bridges for every VLAN?
- Would you recommend any particular Proxmox VE kernel version, bnxt_en driver version, or firmware version for this specific hardware?
Thank you in advance for any guidance or experiences you can share!
