Help setup Linux Bridge using Broadcom Limited NetXtreme II BCM57811

Discussion in 'Proxmox VE: Networking and Firewall' started by Gurvinderpal S Narula, May 21, 2019.

  1. Gurvinderpal S Narula

    Joined:
    Mar 8, 2019
    Messages:
    5
    Likes Received:
    0
    Hello,
    My system has 2 NIC. One is a Intel 1GBs interface and the other is a Broadcom Limited NetXtreme II BCM57811:

    root@pve2:~# lspci | grep Ethernet
    04:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57811 10-Gigabit Ethernet (rev 11)
    09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

    The corresponding interfaces are as follows:

    root@pve2:~# ifconfig -a
    enp4s0: flags=4098<BROADCAST,MULTICAST> mtu 1500
    ether a0:42:3f:27:52:b4 txqueuelen 10000 (Ethernet)
    RX packets 0 bytes 0 (0.0 B)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 0 bytes 0 (0.0 B)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
    device interrupt 53 memory 0xdd800000-ddffffff
    enp9s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
    ether a0:42:3f:27:52:b6 txqueuelen 1000 (Ethernet)
    RX packets 17221 bytes 1727360 (1.6 MiB)
    RX errors 0 dropped 51 overruns 0 frame 0
    TX packets 22724 bytes 2240631 (2.1 MiB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
    device interrupt 16 memory 0xdfc40000-dfc60000

    I have a Linux Bridge Setup for my VMs:

    auto lo
    iface lo inet loopback

    iface enp4s0 inet manual

    iface enp9s0 inet manual

    auto vmbr1
    iface vmbr1 inet static
    address 172.16.75.2
    netmask 255.255.0.0
    gateway 172.16.255.254
    bridge-ports enp9s0
    bridge-stp off
    bridge-fd 0

    Everything works fine (I'm able to ping from/to the host to other parts of my network, access the web gui etc), when the bridge is configured using the enp9s0 (the Intel NIC) as the underlining bridge port.

    However, anytime I try to use the Broadcom NIC as the underlining bridge port (changing the bridge-ports values to enp4s0), I loose complete network connectivity. Neither can I ping other hosts nor can I ping / ssh into the system from other hosts.

    I tried to do a fresh install and specify the Broadcom NIC for the default vmbr0 linux bridge during setup. But that did not help.

    Can someone please help me understand what I'm doing wrong here?

    I have updated my system to the latest binaries as of today morning:

    root@pve2:~# pveversion -v
    proxmox-ve: 5.4-1 (running kernel: 4.15.18-14-pve)
    pve-manager: 5.4-5 (running version: 5.4-5/c6fdb264)
    pve-kernel-4.15: 5.4-2
    pve-kernel-4.15.18-14-pve: 4.15.18-39
    pve-kernel-4.15.18-12-pve: 4.15.18-36
    corosync: 2.4.4-pve1
    criu: 2.11.1-1~bpo90
    glusterfs-client: 3.8.8-1
    ksm-control-daemon: 1.2-2
    libjs-extjs: 6.0.1-2
    libpve-access-control: 5.1-9
    libpve-apiclient-perl: 2.0-5
    libpve-common-perl: 5.0-51
    libpve-guest-common-perl: 2.0-20
    libpve-http-server-perl: 2.0-13
    libpve-storage-perl: 5.0-42
    libqb0: 1.0.3-1~bpo9
    lvm2: 2.02.168-pve6
    lxc-pve: 3.1.0-3
    lxcfs: 3.0.3-pve1
    novnc-pve: 1.0.0-3
    proxmox-widget-toolkit: 1.0-26
    pve-cluster: 5.0-37
    pve-container: 2.0-37
    pve-docs: 5.4-2
    pve-edk2-firmware: 1.20190312-1
    pve-firewall: 3.0-20
    pve-firmware: 2.0-6
    pve-ha-manager: 2.0-9
    pve-i18n: 1.1-4
    pve-libspice-server1: 0.14.1-2
    pve-qemu-kvm: 3.0.1-2
    pve-xtermjs: 3.12.0-1
    qemu-server: 5.0-51
    smartmontools: 6.5+svn4324-1
    spiceterm: 3.0-5
    vncterm: 1.5-3
    zfsutils-linux: 0.7.13-pve1~bpo2

    Would really appreciate it if someone and help resolve this issue.

    Thanks in advance.
     
  2. Gurvinderpal S Narula

    Joined:
    Mar 8, 2019
    Messages:
    5
    Likes Received:
    0
    Did some more homework on this issue - installed debian 9.9 on the same host and had some initial issues with the configuring the NIC . Figured out that that I need to install bnx2 firmware for the card. Installed firmware-bnx2_20161130-5_all.deb and firmware-bnx2x_20161130-5_all.deb and the card came alive.

    Now I've reinstalled proxmox ve and tried to install the same files and I get an error :

    Code:
    dpkg: error processing archive ./firmware-bnx2_20161130-5_all.deb (--install):
     conflicting packages - not installing firmware-bnx2
    Trying to install firmware-bnx2x_20161130-5_all.deb gives the same error.

    So the issue seems to be non-free firmware related. Would appreciate it if someone can help me get these install on proxmox. I have never complied drivers before. I can try that as well if someone can point me in the right direction.

    Unfortunately this is an on board NIC and I don't have any spare slots to put another 10G NIC. So I have to see what I can do to get this NIC up.

    Thanks for looking and would really appreciate it if I can get some help.
     
  3. Gurvinderpal S Narula

    Joined:
    Mar 8, 2019
    Messages:
    5
    Likes Received:
    0
    Bump - still trying to see if someone has a solution to this issue.
    Update - So the NIC works under debian 9.9 - base install. NIC shows up - was able to establish connectivity etc. Did not have to take any special steps to get the NIC to work.

    Installed Proxmox VE on top of debian - NIC / port continued to work without issue until I rebooted the box. On rebooting, the NIC went back to having no connectivity while still showing up as a usable port on Debian and Proxmov VE GUI.
     
  4. skaterdave

    skaterdave New Member
    Proxmox Subscriber

    Joined:
    Aug 9, 2019
    Messages:
    4
    Likes Received:
    0
    Not sure if your issue still persists but I wanted to chime in as I have had a similar issue. I too have a Broadcom 57811 in a Dell r630 running Proxmox 6.0. I have a similar Linux vmbr setup, etc.

    Though the host has connectivity, I cannot run any guest (Windows 2012/2019, Debian 10) with the virtio NetKVM drivers. While the e1000 and others work, they seem sluggish at times as initial ping returns are long (no hard evidence here but something just seems off)

    I also tried adding "non-free" to the repo and apt-get install firmware-bnx2x with the exact same response from apt. My assumption here is that the firmware-pve package has dependencies tied to the rest of the Proxmox distro and can't simply be removed.

    Also not sure if this is just Broadcom related as I tried the onboard Broadcom daughter card NIC with the same issue.

    Because I simply have to make this system work, as it is an infrastructure VM host, I am purchasing an Intel 10g SFP+ card (refurb) just to see if that fixes things as there is no indication of anything else in either host logs or guest logs.

    I'll report back once installed with the result.
     
  5. Gurvinderpal S Narula

    Joined:
    Mar 8, 2019
    Messages:
    5
    Likes Received:
    0
    I could never get the Broadcom to work. I have put this on a back burner and am testing out SR-IOV to provide network connectivity to guest.
     
  6. skaterdave

    skaterdave New Member
    Proxmox Subscriber

    Joined:
    Aug 9, 2019
    Messages:
    4
    Likes Received:
    0
    Definitely post the results as that was my next option too.
     
  7. Gurvinderpal S Narula

    Joined:
    Mar 8, 2019
    Messages:
    5
    Likes Received:
    0
    So in my case I switched to using Mellanox ConnectX-3 and switched to Infiniband. So it’s quite a shift. I can post results if there is interest. I have it working and confirming stability before deploying across multiple vm.
    But I did burn the one pcie slot that I had.
     
  8. skaterdave

    skaterdave New Member
    Proxmox Subscriber

    Joined:
    Aug 9, 2019
    Messages:
    4
    Likes Received:
    0
    That is a shift indeed. Do post results as I'm sure it would help somebody out there on the webz, especially in light of possible Broadcom issues.

    Once my Intel cards are in, I'll chime in as well.
     
  9. skaterdave

    skaterdave New Member
    Proxmox Subscriber

    Joined:
    Aug 9, 2019
    Messages:
    4
    Likes Received:
    0
    Well, FWIW, no luck with the Intel cards either.

    I can make it all work just fine in my test environment (Lenovo desktops running 5.4-3, RTL8168 NIC).
     
    #9 skaterdave, Aug 19, 2019 at 19:05
    Last edited: Aug 19, 2019 at 21:27
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice