[TUTORIAL] Broadcom NICs down after PVE 8.2 (Kernel 6.8)

[jtt]

The goal right now is getting a working network connection through these NICs and confirming they are not DOA from the factory (which does not seem to be the case). I've reinstalled proxmox 8.2 and attached some images of the output of ip a, /etc/network/interfaces, and systemctl restart networking (some input on the infiniband errors might be helpful also). You can also see the FW_HANG, which I was able to resolve with this:

echo "blacklist bnxt_re" >> /etc/modprobe.d/blacklist-bnxt_re.conf
update-initramfs -u

Still thinking updating to the latest build 226.0.145.0 would fix it, but not able to establish any connection

 

[msc]

Did you connect/try both network ports?
On my Supermicro H13SSL-NT the left/upper port is the one with the higher number. For most of my other mainboards it's the other way around.
As jsterr said after "service networking restart" the network should work with the old firmware. At least it did on my H13SSL-NT.

For firmware Updates/configuration changes I ignored the niccli tool and just used the old bnxtnvm tool.
For my H13SSL-NT I could download a newer Firmware from Thomas Krenn, which also included the bnxtnvm tool: https://www.thomas-krenn.com/en/download.html?manufacturer=5&category=82&product=25829
(Though no firmware/bnxtnvm tool is available for your H12SSL-NT-O there. Maybe you can find it elsewhere or get it from Supermicro.)
Then I just used the included FWUpdate.sh script (which uses bnxtnvm) to update the firmware.
The Infiniband Feature Status can be shown/disabled with (you will have to change the dev name):

root@xxx:~/SSWNT5C_NUP# ./bnxtnvm -dev=enp193s0f0np0 getoption=support_rdma:0
support_rdma = Enabled
 
root@xxx:~/SSWNT5C_NUP# ./bnxtnvm -dev=enp193s0f0np0 setoption=support_rdma:0#0
support_rdma is set successfully
Please reboot the system to apply the configuration

 

[jtt]

Well.. after continued testing I realized that I had the cable plugged into a 1 gig connection on my switch. Replacing the connection with a 10 gig port solved all of my issues. Excuse me while I put on my clown shoes... Thanks again to everyone who gave their two cents.

 

[sherminator]

I would like to share another issue which in our case was solved by firmware updating Broadcom NICs.
We have two servers with P425G NICs and SFP28 25G transceivers, the "300m" edition. The OM4 path between these two servers is longer than 300 meters. So the expected behaviour is that they negogiate down to 10G. But this didn't happen. Instead they remained at 25G which resulted in random and weird connection issues.
After updating both NICs to the latest firmware, everything went fine to a stable 10G connection.
Many thanks to @jsterr for pointing out how to do the update procedere - worked like a charm!

 

[SkepticNerdGuy]

Success!

(disclaimer: I'm running Fedora 40 [6.8.8-300.fc40.x86_64] on a Supermicro H11SSW-NT but had the same issues as this and other posts here so I ended up here)

I did a hex dump of my original firmware and saw that at the end there was mention of "p210tep", which is very close to the p210tp firmware on Broadcom's site (direct link: https://docs.broadcom.com/docs/BCM957416A4160C_FW_229.1.123.0). Checking the file from that link also shows "p210tep" at the end, so I felt pretty comfortable in trying this:

niccli -i 1 install -rescue -force BCM957416A4160C.pkg (this will flash both NICs)

I don't need to blacklist bnxt_re anymore, but I still do need to disable RDMA or I get this error:

UBSAN: shift-out-of-bounds in ./include/linux/log2.h:57:13
shift exponent 64 is too large for 64-bit type 'long unsigned int'

This is mentioned in other places on this forum and appears to be a driver problem. I don't need RDMA anyway, so no big deal.

(disable RDMA for easy reference):

niccli -i 1 nvm -setoption support_rdma -scope 0 -value 0
niccli -i 2 nvm -setoption support_rdma -scope 0 -value 0

All Broadcom firmware is listed here: https://www.broadcom.com/support/do...itching,+and+PHYs&pn=&pa=&po=Broadcom&dk=&pl=

lshw:

  *-network:0
       description: Ethernet interface
       product: BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0
       bus info: pci@0000:05:00.0
       logical name: eno1np0
       version: 01
       serial: 00:25:90:5f:99:ec
       size: 10Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical tp 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bnxt_en driverversion=6.8.8-300.fc40.x86_64 duplex=full firmware=229.0.141.0/pkg 229.1.123.0 latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=10Gbit/s
       resources: irq:250 memory:edf10000-edf1ffff memory:ede00000-edefffff memory:edf22000-edf23fff memory:ef980000-ef9fffff
  *-network:1
       description: Ethernet interface
       product: BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0.1
       bus info: pci@0000:05:00.1
       logical name: eno2np1
       version: 01
       serial: 00:25:90:5f:99:ec
       size: 10Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical tp 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bnxt_en driverversion=6.8.8-300.fc40.x86_64 duplex=full firmware=229.0.141.0/pkg 229.1.123.0 latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=10Gbit/s
       resources: irq:267 memory:edf00000-edf0ffff memory:edd00000-eddfffff memory:edf20000-edf21fff memory:ef900000-ef97ffff

Great Success!

Thanks for your post, it really helped fix my problem. Like @darki73 and @jtt I also have a Supermicro H12SSL-NT-O with dual BCM57416 onboard NICs.
I am very new at this and do this as a hobby and just rebuilt my Homelab server with this board. I was a bit disappointed that i had to restart network services every time I rebooted to get the system working, but your post with a few other resources definitely helped out.

@jsterr's post on this thread lead me to https://www.thomas-krenn.com/de/wiki/Installation_NICCLI_unter_Proxmox_VE
I also looked at Broadcom's guide as well here: https://techdocs.broadcom.com/us/en...talling-the-niccli-configuration-utility.html

I was able to just WGET the utility and driver package directly from the broadcom site you linked.
wget https://docs.broadcom.com/docs-and-downloads/ethernet-network-adapters/NXE/BRCM_229.1.123.0/NVRAM_Images/BCM957416A4160C.pkg
wget https://docs.broadcom.com/docs-and-downloads/ethernet-network-adapters/NXE/BRCM_229.1.123.0/niccli/Linux/niccli-229.0.150.0_linux.zip

Then run the command to update the driver:
niccli -i 1 install -rescue -force /root/BCM957416A4160C.pkg

And for solidarity I also disabled RDMA

niccli -i 1 nvm -setoption support_rdma -scope 0 -value 0
niccli -i 2 nvm -setoption support_rdma -scope 0 -value 0

Now the network comes up on boot without issue and there was no need to blacklist bnxt_re.

Thanks!

 

[ness1602]

I've upgraded the firmware of the card to the 226.0.145.0/pkg 226.1.107.1 , but the problem remains.
So,blacklisting it is.

 

[sherminator]

The Firmwareupdate needs a reboot to get active!

I would like to share my today's observations with this:

I just updated some P425G and after a server reboot everything looks as expected:

Active Package version      : 230.1.116.0
Package version on NVM      : 230.1.116.0
Firmware version            : 230.0.156.0

But on a P225G a reboot doesn't seem to be enough, looks like this:

Active Package version      : 216.0.333.11
Package version on NVM      : 230.1.116.0
Firmware version            : 214.4.91.1

Only after shutting the server down and start it again, everything looks fine:

Active Package version      : 230.1.116.0
Package version on NVM      : 230.1.116.0
Firmware version            : 230.0.157.0

 

[Jospeh Huber]

@jsterr many thx for your post!
It has saved my a.. tonight
I had the issue after an upgrade to the latest proxmox 8 on a ASUS RS720A-E12-RS24U server with broadcom adapters.

 

[elkan76]

Hi, I have a HPE Proliant DL360 Gen10, it comes with the Broadcom BCM57416 OCP3, with fresh install of PVE 8.1-1 and 8.2-1. The network not works. I'm following the thread, but when I try to do the NICCLI installation from Thomas Krenn page, I can not download de pve-headers, and then I can´t resolve the firmware installation. I'm stuck to make work two 10Gb LAN ports.
Thanks for any suggestion.

 

[jsterr]

@elkan76 do you have any errors messages? Im not sure if the tool works on dell/hpe hardware, as they often have their own non-standard firmware.

 

[supermicro_server]

Hi, I have a HPE Proliant DL360 Gen10, it comes with the Broadcom BCM57416 OCP3, with fresh install of PVE 8.1-1 and 8.2-1. The network not works. I'm following the thread, but when I try to do the NICCLI installation from Thomas Krenn page, I can not download de pve-headers, and then I can´t resolve the firmware installation. I'm stuck to make work two 10Gb LAN ports.
Thanks for any suggestion.

Hi,
I share to you my commands that I used to upgrade the firmware in my case.
The process got me a Success message but the network cards didn't work well at 25Gbps.

I've tried to install drivers downloaded by Broadcom site.

https://docs.broadcom.com/docs/NXE_Linux_Installer-230.1.116.0

We are using proxmox with the Kernel Linux 6.8.8-2

I've followed the guide and it seams that the installation process is successful but, after shutdown and reboot, the network card doesn't respond anymore

Here the list of commands:

tar -xf bcm_230.1.116.0c.tar.gz

cd /bcm_230.1.116.0c/utils/niccli/linux_x86_64/sliff/dkms#
apt install ./sliff-dkms_230.0.163.0_all.deb

cd /bcm_230.1.116.0c/utils/niccli/linux_x86_64
dpkg -i niccli_230.0.163.0-1_x86_64.deb

cat /bcm_230.1.116.0c/readme.txt

Since we are using AOC-A25G-b2SM, I can suppose that tha correct .pkg file is BCM957414N4140C.

niccli -i 1 install -rescue -force /bcm_230.1.116.0c/board_sku_files/BCM957414N4140C.pkg

 

[elkan76]

@elkan76 do you have any errors messages? Im not sure if the tool works on dell/hpe hardware, as they often have their own non-standard firmware.

Hi, @jsterr after a googling night on the forum, I found some points to do before install niccli:

1. Install PVE with another working NIC to access repositories.
2. Add the no-subscription and ceph-reef repositories.
3. Install proxmox-headers-$(uname -r) and dkms
4. After that, install sliff and niccli.

When I execute, NICCLI see the card, the firmware it was installed with no errors, but the network card still is not working. Maybe as you said, HPE is more "special" than others .


Well, we will buy a Intel dual port to go out of this little problem. This server works fine in the others aspects.
Thanks for the answers.

 

[elkan76]

Hi, @jsterr after a googling night on the forum, I found some points to do before install niccli:

1. Install PVE with another working NIC to access repositories.
2. Add the no-subscription and ceph-reef repositories.
3. Install proxmox-headers-$(uname -r) and dkms
4. After that, install sliff and niccli.

When I execute, NICCLI see the card, the firmware it was installed with no errors, but the network card still is not working. Maybe as you said, HPE is more "special" than others .
View attachment 71916

Well, we will buy a Intel dual port to go out of this little problem. This server works fine in the others aspects.
Thanks for the answers.

After a couple of days with one PCIe GigaLAN Intel, the server was working fine, but some day seems to be offline. When I go to the rack, the NIC is linking. I had a hunch, and plug the Broadcom interfaces. They start to link and then I configure bonding and replace the eno1 and works!!

87:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
87:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)

Now I start to test the performance, but I have doubts. Some Windows Server VM, can not find the Domain Controller.
Well, Broadcom 57416 works on Proxmox 8.2.4
Thanks to all.

 

[DJFriar]

After a couple of days with one PCIe GigaLAN Intel, the server was working fine, but some day seems to be offline. When I go to the rack, the NIC is linking. I had a hunch, and plug the Broadcom interfaces. They start to link and then I configure bonding and replace the eno1 and works!!

87:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
87:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)

Now I start to test the performance, but I have doubts. Some Windows Server VM, can not find the Domain Controller.
Well, Broadcom 57416 works on Proxmox 8.2.4
Thanks to all.

I have a Broadcom 57416 on my Proxmox Backup Server 3.2-3 and am experiencing the no networking at boot issue... where you able to fix it using one of the solutions in this thread? I'm running a Supermicro server with the X13SCH-SYS motherboard, if that helps.

 

[elkan76]

I have a Broadcom 57416 on my Proxmox Backup Server 3.2-3 and am experiencing the no networking at boot issue... where you able to fix it using one of the solutions in this thread? I'm running a Supermicro server with the X13SCH-SYS motherboard, if that helps.

I follow the instructions on this thread, but I add some considerations:

1. You MUST have at least one interface with intel chip or other fully compatible to download the no subscriptions and ceph repositories.
2. Activate the repositories download the proxmox headers.
3. Install sliff and niccli downloaded from Thomas Krenn website.
4. Reboot the system and update.
5. For security, I disable no subscription and ceph repositories.
6. Test the Broadcom NIC, if works, uninstall the other NIC.

Thats all I can say tell us if works for you.

 

[ocmt58z1]

@jsterr I am experiencing the same problem with the Supermicro N12SSL-NT.
I tried to apply the update, but I get this error:



What can I do?

Running it with the listdev flag shows this:



I want to point out that I do have these 10G ports connected to a 1G port on my router - I would assume that they are backwards compatbile..? (I bought this mainboard with the 10Gig ports to be able to use 10Gig in the future).

So, what are my options? I can't update this firmware, apparently? Do I have to buy a 1G Network card and disable everything else, or what should I do?

 

[jsterr]

@jsterr I am experiencing the same problem with the Supermicro N12SSL-NT.
I tried to apply the update, but I get this error:

View attachment 73467

What can I do?

Running it with the listdev flag shows this:

View attachment 73468

I want to point out that I do have these 10G ports connected to a 1G port on my router - I would assume that they are backwards compatbile..? (I bought this mainboard with the 10Gig ports to be able to use 10Gig in the future).

So, what are my options? I can't update this firmware, apparently? Do I have to buy a 1G Network card and disable everything else, or what should I do?

Afaik my guide only works for external not onboard nics. NT = 10GBit onBoard so you need to reach out to supermicro to get the correct firmware. As we do not have N12 in our portfolio I cant help you with getting and testing firmware via Thomas Krenn.

 

[ocmt58z1]

Afaik my guide only works for external not onboard nics. NT = 10GBit onBoard so you need to reach out to supermicro to get the correct firmware. As we do not have N12 in our portfolio I cant help you with getting and testing firmware via Thomas Krenn.

Not sure if I follow. I am not entirely sure what to tell Supermicro.
The dual onboard 10Gig NICs can't be updated with bnxtnvm - but how do I know what firmware is adequate for these NICs.

Do you mind ellaborating?

 

[billy999]

@jsterr I am experiencing the same problem with the Supermicro N12SSL-NT

Im unable to find that Mainboard, can you link the product page?

 

[ocmt58z1]

Im unable to find that Mainboard, can you link the product page?

https://www.supermicro.com/en/products/motherboard/H12SSL-NT