[TUTORIAL] Broadcom NICs down after PVE 8.2 (Kernel 6.8)

[jtt]

The goal right now is getting a working network connection through these NICs and confirming they are not DOA from the factory (which does not seem to be the case). I've reinstalled proxmox 8.2 and attached some images of the output of ip a, /etc/network/interfaces, and systemctl restart networking (some input on the infiniband errors might be helpful also). You can also see the FW_HANG, which I was able to resolve with this:

echo "blacklist bnxt_re" >> /etc/modprobe.d/blacklist-bnxt_re.conf
update-initramfs -u

Still thinking updating to the latest build 226.0.145.0 would fix it, but not able to establish any connection

 

[msc]

Did you connect/try both network ports?
On my Supermicro H13SSL-NT the left/upper port is the one with the higher number. For most of my other mainboards it's the other way around.
As jsterr said after "service networking restart" the network should work with the old firmware. At least it did on my H13SSL-NT.

For firmware Updates/configuration changes I ignored the niccli tool and just used the old bnxtnvm tool.
For my H13SSL-NT I could download a newer Firmware from Thomas Krenn, which also included the bnxtnvm tool: https://www.thomas-krenn.com/en/download.html?manufacturer=5&category=82&product=25829
(Though no firmware/bnxtnvm tool is available for your H12SSL-NT-O there. Maybe you can find it elsewhere or get it from Supermicro.)
Then I just used the included FWUpdate.sh script (which uses bnxtnvm) to update the firmware.
The Infiniband Feature Status can be shown/disabled with (you will have to change the dev name):

root@xxx:~/SSWNT5C_NUP# ./bnxtnvm -dev=enp193s0f0np0 getoption=support_rdma:0
support_rdma = Enabled
 
root@xxx:~/SSWNT5C_NUP# ./bnxtnvm -dev=enp193s0f0np0 setoption=support_rdma:0#0
support_rdma is set successfully
Please reboot the system to apply the configuration

 

[jtt]

Well.. after continued testing I realized that I had the cable plugged into a 1 gig connection on my switch. Replacing the connection with a 10 gig port solved all of my issues. Excuse me while I put on my clown shoes... Thanks again to everyone who gave their two cents.

 

[sherminator]

I would like to share another issue which in our case was solved by firmware updating Broadcom NICs.
We have two servers with P425G NICs and SFP28 25G transceivers, the "300m" edition. The OM4 path between these two servers is longer than 300 meters. So the expected behaviour is that they negogiate down to 10G. But this didn't happen. Instead they remained at 25G which resulted in random and weird connection issues.
After updating both NICs to the latest firmware, everything went fine to a stable 10G connection.
Many thanks to @jsterr for pointing out how to do the update procedere - worked like a charm!