Help setup Linux Bridge using Broadcom Limited NetXtreme II BCM57811

[Gurvinderpal S Narula]

Hello,
My system has 2 NIC. One is a Intel 1GBs interface and the other is a Broadcom Limited NetXtreme II BCM57811:

root@pve2:~# lspci | grep Ethernet
04:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57811 10-Gigabit Ethernet (rev 11)
09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection​

The corresponding interfaces are as follows:

root@pve2:~# ifconfig -a
enp4s0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether a0:42:3f:27:52:b4 txqueuelen 10000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 53 memory 0xdd800000-ddffffff
enp9s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether a0:42:3f:27:52:b6 txqueuelen 1000 (Ethernet)
RX packets 17221 bytes 1727360 (1.6 MiB)
RX errors 0 dropped 51 overruns 0 frame 0
TX packets 22724 bytes 2240631 (2.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xdfc40000-dfc60000​

I have a Linux Bridge Setup for my VMs:

auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp9s0 inet manual

auto vmbr1
iface vmbr1 inet static
address 172.16.75.2
netmask 255.255.0.0
gateway 172.16.255.254
bridge-ports enp9s0
bridge-stp off
bridge-fd 0​

Everything works fine (I'm able to ping from/to the host to other parts of my network, access the web gui etc), when the bridge is configured using the enp9s0 (the Intel NIC) as the underlining bridge port.

However, anytime I try to use the Broadcom NIC as the underlining bridge port (changing the bridge-ports values to enp4s0), I loose complete network connectivity. Neither can I ping other hosts nor can I ping / ssh into the system from other hosts.

I tried to do a fresh install and specify the Broadcom NIC for the default vmbr0 linux bridge during setup. But that did not help.

Can someone please help me understand what I'm doing wrong here?

I have updated my system to the latest binaries as of today morning:

root@pve2:~# pveversion -v
proxmox-ve: 5.4-1 (running kernel: 4.15.18-14-pve)
pve-manager: 5.4-5 (running version: 5.4-5/c6fdb264)
pve-kernel-4.15: 5.4-2
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-12-pve: 4.15.18-36
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-9
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-51
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-42
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-26
pve-cluster: 5.0-37
pve-container: 2.0-37
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-20
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-2
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-51
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2​

Would really appreciate it if someone and help resolve this issue.

Thanks in advance.

 

[Gurvinderpal S Narula]

Did some more homework on this issue - installed debian 9.9 on the same host and had some initial issues with the configuring the NIC . Figured out that that I need to install bnx2 firmware for the card. Installed firmware-bnx2_20161130-5_all.deb and firmware-bnx2x_20161130-5_all.deb and the card came alive.

Now I've reinstalled proxmox ve and tried to install the same files and I get an error :

dpkg: error processing archive ./firmware-bnx2_20161130-5_all.deb (--install):
 conflicting packages - not installing firmware-bnx2

Trying to install firmware-bnx2x_20161130-5_all.deb gives the same error.

So the issue seems to be non-free firmware related. Would appreciate it if someone can help me get these install on proxmox. I have never complied drivers before. I can try that as well if someone can point me in the right direction.

Unfortunately this is an on board NIC and I don't have any spare slots to put another 10G NIC. So I have to see what I can do to get this NIC up.

Thanks for looking and would really appreciate it if I can get some help.

 

[Gurvinderpal S Narula]

Bump - still trying to see if someone has a solution to this issue.
Update - So the NIC works under debian 9.9 - base install. NIC shows up - was able to establish connectivity etc. Did not have to take any special steps to get the NIC to work.

Installed Proxmox VE on top of debian - NIC / port continued to work without issue until I rebooted the box. On rebooting, the NIC went back to having no connectivity while still showing up as a usable port on Debian and Proxmov VE GUI.

 

[skaterdave]

Not sure if your issue still persists but I wanted to chime in as I have had a similar issue. I too have a Broadcom 57811 in a Dell r630 running Proxmox 6.0. I have a similar Linux vmbr setup, etc.

Though the host has connectivity, I cannot run any guest (Windows 2012/2019, Debian 10) with the virtio NetKVM drivers. While the e1000 and others work, they seem sluggish at times as initial ping returns are long (no hard evidence here but something just seems off)

I also tried adding "non-free" to the repo and apt-get install firmware-bnx2x with the exact same response from apt. My assumption here is that the firmware-pve package has dependencies tied to the rest of the Proxmox distro and can't simply be removed.

Also not sure if this is just Broadcom related as I tried the onboard Broadcom daughter card NIC with the same issue.

Because I simply have to make this system work, as it is an infrastructure VM host, I am purchasing an Intel 10g SFP+ card (refurb) just to see if that fixes things as there is no indication of anything else in either host logs or guest logs.

I'll report back once installed with the result.

 

[Gurvinderpal S Narula]

I could never get the Broadcom to work. I have put this on a back burner and am testing out SR-IOV to provide network connectivity to guest.

 

[skaterdave]

Definitely post the results as that was my next option too.

 

[Gurvinderpal S Narula]

So in my case I switched to using Mellanox ConnectX-3 and switched to Infiniband. So it’s quite a shift. I can post results if there is interest. I have it working and confirming stability before deploying across multiple vm.
But I did burn the one pcie slot that I had.

 

[skaterdave]

That is a shift indeed. Do post results as I'm sure it would help somebody out there on the webz, especially in light of possible Broadcom issues.

Once my Intel cards are in, I'll chime in as well.

 

[skaterdave]

Well, FWIW, no luck with the Intel cards either.

I can make it all work just fine in my test environment (Lenovo desktops running 5.4-3, RTL8168 NIC).