[TUTORIAL] Broadcom NICs down after PVE 8.2 (Kernel 6.8)

[bcnx]

Hi,
quickly mentioning that we had the same problem on a recent acquired Supermicro server. We asked their support for the latest firmware and got file aoc-a25g-b2s\ fw\ 228.0.116.0.zip. Mind you, after upgrade version 227 is on the NICs.
We then installed niccli_231.0.162.1-1_x86_64.deb.
We changed the upgrade script to:

#!/bin/bash

# Remove bnxt_en/re driver
rmmod bnxt_re >/dev/null 2>&1
rmmod bnxt_en >/dev/null 2>&1

niccli -i 1 install A25G227_transition.pkg --yes
sleep 3

niccli -i 1 reset
sleep 3

niccli -i 1 install AOC-A25G-b2S.pkg --yes
sleep 3

niccli -i 1 reset
sleep 3
# chmod +x FWUpdate_custom.sh

That fixed the problem, no further blacklisting required,

BC

 

[Ewout]

Hi

i hope some of you might be able too help me, i am trying to get a (AS -2015SV-WTNRT) system up and running, this system has a embed,
BCM57416. i installed proxmox 8.3 originally, during the install the network seemed to be recognized a green circle around the interface during the GUI install. After the install the system booted up and the network stayed down even if i tried and reboot the network using

service networking restart

, the network didn't come up inside of proxmox but the lights on the server where still blinking.

thinking it might be firmware issue, i tried this method, i also got the error

UBSAN: shift-out-of-bounds in ./include/linux/log2.h:57:13
shift exponent 64 is too large for 64-bit type 'long unsigned int'

so i also ran the commands

niccli -i 1 nvm -setoption support_rdma -scope 0 -value 0
niccli -i 2 nvm -setoption support_rdma -scope 0 -value 0

this didn't seem to fix the issue, and also when i rebooted my server and looked at the network lights again it seems to have stopped blinking.
I don't know if i have bricked the card because i still see it in the bios and it still shows up when running niccli.
I also tried installing 8.1 after this, but it doesn't seem to be any different appart from the network interface it not being green during the gui install.
i also tried installing the newest firmware from the broadcom site. Problably a dumb thing to have done, but the card already wasn't blinking anymore.
The only thing i can think of doing wrong is that i have the 10gb port connected too a 1gb connection and i don't know it the BCM57416 supports this. because i saw a post about @jtt

should i maybe try running the tools provided by thomas kren?
or what should i try next, i currently don't have a seperate network card on hand, and the network switch i ordered with 10gb was DOA.


Any recommendation on what too try next are appreciated.


update i installed the firmware from the H13SSL-NT (this board matches closly with the board i have) from thomas krenns website, this brought back the led's in the back, also the bios now shows connected, still can't get the link too work. Also i have contacted my reseller, for a firmware version.

I installed the orginall firmware again which is version 226,
when running proxmox i still can't get my interface too work.
the dmesg shows the following when restarting the service

Entered allmulticast mode
entered promiscuous mode
NIC Link is down
Nic Link is Up, 100 Mbps full duplex, Flow control: None
FEC autoneg off encoding: None
vmbr0: port (eno1np0) entered blocking state
vmbr0: port (eno1np0) entered forwarding state

VMBR0 shows up, but can't ping 1.1.1.1 or 8.8.8.8, Get a Destination Host Unreachable

i fixed it by using proxmox 8.1 for now, and connecting too a 10gb network.

 

[stepanov1975]

Disabled RDMA in bios solved the problem for Supermicro H12SSL-NT motherboard

 

[NexusDC]

@stepanov1975 - I've got the same Supermicro H12SSL-NT - where did you disable RDMA at? I'm not seeing it. I was on BIOS 2.7 so I flashed to 3.0 but I'm still not seeing it. I'm sure I'm just missing it... Thanks!

Edited: Nevermind - I found it. It was under the Network card itself which now shows up at the bottom of Advanced. Reloading Proxmox 8.3 - hopefully it will actually work...!

Update: I reloaded Proxmox 8.3 after just disabling RDMA for both 10G NICs - the network came up immediately on startup and there are no errors when starting up or shutting down (this is a first...!). Thanks to @stepanov1975 for pointing me in the right direction!

 

[hostmost]

@stepanov1975 - I've got the same Supermicro H12SSL-NT - where did you disable RDMA at? I'm not seeing it. I was on BIOS 2.7 so I flashed to 3.0 but I'm still not seeing it. I'm sure I'm just missing it... Thanks!

Edited: Nevermind - I found it. It was under the Network card itself which now shows up at the bottom of Advanced. Reloading Proxmox 8.3 - hopefully it will actually work...!

Update: I reloaded Proxmox 8.3 after just disabling RDMA for both 10G NICs - the network came up immediately on startup and there are no errors when starting up or shutting down (this is a first...!). Thanks to @stepanov1975 for pointing me in the right direction!

Thank you!

That solved my issue!!!