NIC Entered Disabled State

[Devolve]

Hello

Every few hours when we action a reboot or stop/start the vmbr0 goes offline for ~15mins then comes back online, if we restart network it works as normal again and host responds but then we have to reboot VM also.

It is very random and getting annoying now

Mar 9 17:51:44 kernel: [ 4501.460886] fwbr1002i0: port 2(tap1002i0) entered disabled state
Mar 9 17:51:44 kernel: [ 4501.465811] fwbr1002i0: port 1(fwln1002i0) entered disabled state
Mar 9 17:51:44 kernel: [ 4501.466108] vmbr0: port 3(fwpr1002p0) entered disabled state
Mar 9 17:51:44 kernel: [ 4501.466403] device fwln1002i0 left promiscuous mode
Mar 9 17:51:44 kernel: [ 4501.466407] fwbr1002i0: port 1(fwln1002i0) entered disabled state
Mar 9 17:51:44 kernel: [ 4501.466784] device fwpr1002p0 left promiscuous mode
Mar 9 17:51:44 kernel: [ 4501.466788] vmbr0: port 3(fwpr1002p0) entered disabled state

NIC:
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet Multi Function (rev 10)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet Multi Function (rev 10)

VMs are KVM with default settings, Virtio Disk & Intel e1000 NIC

Latest NIC firmware, latest PVE

 

[ghusson]

Do you use ethernet bonding ?
Have you disabled igmp snooping on virtual bridges ?

 

[Devolve]

Do you use ethernet bonding ?
Have you disabled igmp snooping on virtual bridges ?

Sorry yes bonding is used. No special alteration to bond0/vmbr0 configs apart from standard setup

Should I try with "echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping"?

auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
    slaves eth0 eth1
    bond_miimon 100
    bond_mode 802.3ad
    bond_xmit_hash_policy layer3+4

auto vmbr0
iface vmbr0 inet static
    address X
    netmask  X
    gateway  X
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0
    post-up echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping

 

[ghusson]

Try disabling snooping.
And maybe try virtio network for the VMs ?

 

[Devolve]

Try disabling snooping.
And maybe try virtio network for the VMs ?

I actually switched from Virtio to e1000 after looking around this forum and some suggestion saying that worked for them but no luck here

Just happened again when I tried to change a VM NIC to Virtio again. Host node stops responding, a network restart makes it respond again.

Mar 10 12:48:19 system4 kernel: [48920.172298] vmbr0: port 4(fwpr1001p0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.172339] vmbr0: port 5(fwpr1002p0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.172356] vmbr0: port 3(tap101i0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.172368] vmbr0: port 2(tap100i0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.172380] vmbr0: port 1(bond0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.182108] device bond0 left promiscuous mode
Mar 10 12:48:19 system4 kernel: [48920.182115] device eth0 left promiscuous mode
Mar 10 12:48:19 system4 kernel: [48920.182181] device eth1 left promiscuous mode
Mar 10 12:48:19 system4 kernel: [48920.182288] vmbr0: port 1(bond0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.184929] IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready
Mar 10 12:48:19 system4 kernel: [48920.186206] device fwpr1001p0 left promiscuous mode
Mar 10 12:48:19 system4 kernel: [48920.186288] vmbr0: port 4(fwpr1001p0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.186506] device fwpr1002p0 left promiscuous mode
Mar 10 12:48:19 system4 kernel: [48920.186587] vmbr0: port 5(fwpr1002p0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.186808] vmbr0: port 3(tap101i0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.186997] vmbr0: port 2(tap100i0) entered disabled state
Mar 10 12:48:19 system4 kernel: [48920.297226] bond0: Removing slave eth0
Mar 10 12:48:19 system4 kernel: [48920.297431] bond0: Releasing active interface eth0
Mar 10 12:48:19 system4 kernel: [48920.297446] bond0: the permanent HWaddr of eth0 - xx:2a:72:xx:xx:xx - is still in use by bond0 - set the HWaddr of eth0 to a different address to avoid conflicts
Mar 10 12:48:19 system4 kernel: [48920.708729] bond0: Removing slave eth1
Mar 10 12:48:19 system4 kernel: [48920.708922] bond0: Removing an active aggregator
Mar 10 12:48:19 system4 kernel: [48920.708928] bond0: Releasing active interface eth1
Mar 10 12:48:20 system4 kernel: [48921.138595] bond0: Setting xmit hash policy to layer3+4 (1)
Mar 10 12:48:20 system4 kernel: [48921.138758] bond0: Setting MII monitoring interval to 100
Mar 10 12:48:20 system4 kernel: [48921.144180] bond0: Adding slave eth0
Mar 10 12:48:20 system4 kernel: [48921.688361] bnx2x 0000:01:00.2 eth0: using MSI-X  IRQs: sp 95  fp[0] 97 ... fp[7] 104
Mar 10 12:48:20 system4 kernel: [48921.763997] bnx2x 0000:01:00.2 eth0: NIC Link is Up, 1000 Mbps full duplex, Flow control: none
Mar 10 12:48:20 system4 kernel: [48921.841096] bond0: Enslaving eth0 as a backup interface with an up link
Mar 10 12:48:20 system4 kernel: [48921.845980] bond0: Adding slave eth1
Mar 10 12:48:21 system4 kernel: [48922.356088] bnx2x 0000:01:00.3 eth1: using MSI-X  IRQs: sp 106  fp[0] 108 ... fp[7] 115
Mar 10 12:48:21 system4 kernel: [48922.431725] bnx2x 0000:01:00.3 eth1: NIC Link is Up, 1000 Mbps full duplex, Flow control: none
Mar 10 12:48:21 system4 kernel: [48922.508829] bond0: Enslaving eth1 as a backup interface with an up link
Mar 10 12:48:21 system4 kernel: [48922.738370] device bond0 entered promiscuous mode
Mar 10 12:48:21 system4 kernel: [48922.738376] device eth0 entered promiscuous mode
Mar 10 12:48:21 system4 kernel: [48922.738432] device eth1 entered promiscuous mode
Mar 10 12:48:21 system4 kernel: [48922.744534] vmbr0: port 1(bond0) entered forwarding state
Mar 10 12:48:21 system4 kernel: [48922.744567] vmbr0: port 1(bond0) entered forwarding state

 

[Devolve]

Happened again today when I was changing firewall rules on a VM within proxmox, beyond production ready at the moment.

Looks like activity that touches the network causes it to break randomly but works most times

 

[ghusson]

did you tryed "echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping" ?
do you use multi queue on device ?
is there a debug mode in the ethernet card module ?

 

[Devolve]

did you tryed "echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping" ?
do you use multi queue on device ?
is there a debug mode in the ethernet card module ?

Yes I tried that no luck.

I am testing now with another OS + QEMU and there has been no issues using other panels. Drivers for bnx2x is same on Debian and CentOS so issue must be something related to proxmox somewhere.