Proxmox on ASRock Rack B650D4U-2L2T/BCM Broadcom BCM57416 10g Nic Failure

[maires]

Creating a post for this even though there are a couple of threads floating around that touch on the subject

S

Thread 'Pve crashes because of BCM57416'

Jul 30, 2023

Hey everybody,
I have a problem and hope someone can help me solve it or point me in the right direction.
My setup: Mobo: Asrockrack X570D4U-2L2T/BCM
-- Bcm and bios updated to the latest drivers.
Cpu: AMD 4750G
Gpu: Nvidia GTX 1050Ti

I'm running pve8.0 After the host being up for some time I get the following error on my 10G broadcom Nics. (BCM57416)
After a reboot the Nics are down and not discovered by lspci or with ip link show.
If I disconnect the power from the host and after a few minutes reconnect and start the host the Nics are up again.
What do you think?

Error Message...

• SheridansNL
• Replies: 8
• Forum: Proxmox VE: Networking and Firewall

[TUTORIAL] Thread 'Broadcom NICs down after PVE 8.2 (Kernel 6.8)'

Apr 30, 2024

We had some issues with some broadcom nics going down after update to 6.8

Workaround: NICs go up if you do a service networking restart
FIX: Update Broadcom Firmware to latest firmware and blacklist their "beautiful" infiniband-driver

This will update ALL YOUR Broadcom-Network Cards to their latest firmware (live) (but reboot needed after it):

apt install unzip
cat << 'EOF' > bcm-nic-update.sh
wget https://www.thomas-krenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip
unzip bnxtnvm.zip
chmod +x bnxtnvm
for i in $(./bnxtnvm listdev...

• jsterr
• broadcom kernel 6.8 network p225g p425g pve 8.2
• Replies: 60
• Forum: Proxmox VE: Installation and configuration

• 

Il try to keep this concise. I believe this issue is related to proxmox latest install where people were having to upgrade
their broadcom nic firmware.

- I do not think this is overheating. The NIC does not get over 50C-60C and there is a fan pointed directly at it. If it is overheating at this point the board would need to be returned.

+ I think that the most likely culprit is the firmware needs to be updated. But most of the people that have had this issue with broadcom nics also seem to have a supermicro board not the asrock one that I am using.

- I have not managed to update the firmware
+ Used https://www.thomas-krenn.com/de/wiki/Installation_NICCLI_unter_Proxmox_VE to get NICCLI Installed
+ Get an error that PCIE ID does not match. Mine is 14e4:16d8 but for the life of me I cannot find the corresponding firmware update...


root@proxmox01:/opt/niccli# niccli -i 1 install /root/BCM957416A4160C.pkg
/opt/niccli/niccli.x86_64: /lib/x86_64-linux-gnu/libnl-3.so.200: no version info rmation available (required by /opt/niccli/niccli.x86_64)

-------------------------------------------------------------------------------
NIC CLI v231.0.162.1 - Broadcom Inc. (c) 2024 (Bld-94.52.34.117.16.0)
-------------------------------------------------------------------------------

WARNING : Don't perform power cycle or reboot the system while firmware update
is in progress as the device may become inoperable.

PCI ID(s) mismatch. Package installation failed
you may use the following to recover if you're sure that .pkg is for this board:
you may use: install -rescue <CONFIG>.pkg
you may use: install -rescue -force <FW>.pkg
ERROR: Package update is not SUCCESSFUL!


EXIT CODE : 0B003137
DESCRIPTION : Package PCI Id mismatch.
Command install failed.


+ I narrowed the chip down to https://docs.broadcom.com/docs/BCM957416A4160C_FW_231.1.162.1 however this is for a seperate PCIE card and mine is built onto the motherboard.
+ I can use the NICCLI Force command to force the firmware but not sure if I should and not sure how to check the firmware for compatibility with my device or do a general search and find a different firmware.
+ I did update the bnxt_en driver

version: 1.10.3-231.0.162.0
srcversion: C7F36D975DBC0F137A69D80
vermagic: 6.8.12-5-pve SMP preempt mod_unload modversions

I have an open ticket with asrock but I am not really holding out any hope to get a productive response from them.

 

[maires]

I am also open to looking at bios settings. They are default at the moment but the Broadcom nics have the ability to have SR-IOV on at one point i had them configured like that but now they are back to the second image below where SR-IOV is disabled.