Proxmox on ASRock Rack B650D4U-2L2T/BCM Broadcom BCM57416 10g Nic Failure

[maires]

Creating a post for this even though there are a couple of threads floating around that touch on the subject

S

Thread 'Pve crashes because of BCM57416'

Jul 30, 2023

Hey everybody,
I have a problem and hope someone can help me solve it or point me in the right direction.
My setup: Mobo: Asrockrack X570D4U-2L2T/BCM
-- Bcm and bios updated to the latest drivers.
Cpu: AMD 4750G
Gpu: Nvidia GTX 1050Ti

I'm running pve8.0 After the host being up for some time I get the following error on my 10G broadcom Nics. (BCM57416)
After a reboot the Nics are down and not discovered by lspci or with ip link show.
If I disconnect the power from the host and after a few minutes reconnect and start the host the Nics are up again.
What do you think?

Error Message...

• SheridansNL
• Replies: 9
• Forum: Proxmox VE: Networking and Firewall

[TUTORIAL] Thread 'Broadcom NICs down after PVE 8.2 (Kernel 6.8)'

Apr 30, 2024

We had some issues with some broadcom nics going down after update to 6.8

Workaround: NICs go up if you do a service networking restart
FIX: Update Broadcom Firmware to latest firmware and blacklist their "beautiful" infiniband-driver

This will update ALL YOUR Broadcom-Network Cards to their latest firmware (live) (but reboot needed after it):

apt install unzip
cat << 'EOF' > bcm-nic-update.sh
wget https://www.thomas-krenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip
unzip bnxtnvm.zip
chmod +x bnxtnvm
for i in $(./bnxtnvm listdev...

• jsterr
• broadcom kernel 6.8 network p225g p425g pve 8.2
• Replies: 65
• Forum: Proxmox VE: Installation and configuration

• 

Il try to keep this concise. I believe this issue is related to proxmox latest install where people were having to upgrade
their broadcom nic firmware.

- I do not think this is overheating. The NIC does not get over 50C-60C and there is a fan pointed directly at it. If it is overheating at this point the board would need to be returned.

+ I think that the most likely culprit is the firmware needs to be updated. But most of the people that have had this issue with broadcom nics also seem to have a supermicro board not the asrock one that I am using.

- I have not managed to update the firmware
+ Used https://www.thomas-krenn.com/de/wiki/Installation_NICCLI_unter_Proxmox_VE to get NICCLI Installed
+ Get an error that PCIE ID does not match. Mine is 14e4:16d8 but for the life of me I cannot find the corresponding firmware update...


root@proxmox01:/opt/niccli# niccli -i 1 install /root/BCM957416A4160C.pkg
/opt/niccli/niccli.x86_64: /lib/x86_64-linux-gnu/libnl-3.so.200: no version info rmation available (required by /opt/niccli/niccli.x86_64)

-------------------------------------------------------------------------------
NIC CLI v231.0.162.1 - Broadcom Inc. (c) 2024 (Bld-94.52.34.117.16.0)
-------------------------------------------------------------------------------

WARNING : Don't perform power cycle or reboot the system while firmware update
is in progress as the device may become inoperable.

PCI ID(s) mismatch. Package installation failed
you may use the following to recover if you're sure that .pkg is for this board:
you may use: install -rescue <CONFIG>.pkg
you may use: install -rescue -force <FW>.pkg
ERROR: Package update is not SUCCESSFUL!


EXIT CODE : 0B003137
DESCRIPTION : Package PCI Id mismatch.
Command install failed.


+ I narrowed the chip down to https://docs.broadcom.com/docs/BCM957416A4160C_FW_231.1.162.1 however this is for a seperate PCIE card and mine is built onto the motherboard.
+ I can use the NICCLI Force command to force the firmware but not sure if I should and not sure how to check the firmware for compatibility with my device or do a general search and find a different firmware.
+ I did update the bnxt_en driver

version: 1.10.3-231.0.162.0
srcversion: C7F36D975DBC0F137A69D80
vermagic: 6.8.12-5-pve SMP preempt mod_unload modversions

I have an open ticket with asrock but I am not really holding out any hope to get a productive response from them.

 

[maires]

I am also open to looking at bios settings. They are default at the moment but the Broadcom nics have the ability to have SR-IOV on at one point i had them configured like that but now they are back to the second image below where SR-IOV is disabled.

 

[tsueri]

Hi

I have the exact same board in a 1U4LW-B650/2L2T RPSU. Just had to restart one of my servers (unplug the server completly). Did you get any advice from support yet?

 

[FallingSnow]

I have an Asrock Rack ROMED8-2T. I've gotten SR-IOV to work before using bxntnvm application but reinstalled my OS and am back to trying to get it working. While I was at it I figured I'd update the firmware. After running into those "Package PCI Id mismatch." errors while using broadcom's new niccli I needed to figure out what package I should actually be looking for.

Run lspci | grep net to find the slot your network card is in. I had a dual port so my first one was "42:00.0". Then use the following to get more specific information about the device.

$ lspci -vvvv -s 42:00.0
42:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E Dual-port 10GBASE-T Ethernet OCP 3.0 Adapter (BCM957416N4160C)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        ...

I was interested in the subsystem part, specifically the "BCM957416N4160C" part. If you search that on broadcom's website you end up with a bunch of firmware packages. I grabbed the one with the latest date. That got me past the PCI mismatch error, but install still failed without an error code so... BTW if you got to this part, make sure you are connected on a port different than the controller you are updating, for example directly via a monitor and mouse or a SSH'ed via a USB to Ethernet dongle, using the other port on the same controller wont work.

To get around the install failing I used the install command's -recovery option. And that worked!

Now to figure out how to enable SR-IOV again

 

[nonamenonumber]

I have an Asrock Rack ROMED8-2T. I've gotten SR-IOV to work before using bxntnvm application but reinstalled my OS and am back to trying to get it working. While I was at it I figured I'd update the firmware. After running into those "Package PCI Id mismatch." errors while using broadcom's new niccli I needed to figure out what package I should actually be looking for.

Run lspci | grep net to find the slot your network card is in. I had a dual port so my first one was "42:00.0". Then use the following to get more specific information about the device.

$ lspci -vvvv -s 42:00.0
42:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
        Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E Dual-port 10GBASE-T Ethernet OCP 3.0 Adapter (BCM957416N4160C)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        ...

I was interested in the subsystem part, specifically the "BCM957416N4160C" part. If you search that on broadcom's website you end up with a bunch of firmware packages. I grabbed the one with the latest date. That got me past the PCI mismatch error, but install still failed without an error code so... BTW if you got to this part, make sure you are connected on a port different than the controller you are updating, for example directly via a monitor and mouse or a SSH'ed via a USB to Ethernet dongle, using the other port on the same controller wont work.

To get around the install failing I used the install command's -recovery option. And that worked!

Now to figure out how to enable SR-IOV again

Is there a reason not to use the online mode to update the firmware? Seemed to work for me. My ROMED8-2T/BCM came with 221.x firmware, ran the below command, it found the right package and flashed fine as far as i can tell?

niccli list
/opt/niccli/niccli.x86_64: /lib/x86_64-linux-gnu/libnl-3.so.200: no version information available (required by /opt/niccli/niccli.x86_64)

-------------------------------------------------------------------------------
NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0)
-------------------------------------------------------------------------------

     BoardId(Rev)    MAC Address        FwVersion    PCIAddr        Type   Mode
  1) BCM57416(B1)    9C:6B:00:78:15:AF  233.0.151.0  0000:42:00.0   NIC    PCI  
  2) BCM57416(B1)    9C:6B:00:78:15:B0  233.0.151.0  0000:42:00.1   NIC    PCI  



root@x:~# niccli -i 1 install -online

Did you figure out SR-IOV? I've enabled it on one of the NICs in the BIOS, done the sriov_numvfs update but don't see any additional devices in lspci on prox. Seems i'm a bit slow and didn't start the interface before requesting VFs... all working