[TUTORIAL] Broadcom NICs down after PVE 8.2 (Kernel 6.8)

[ocmt58z1]

I am currently trying to follow the steps described by @SkepticNerdGuy who apparently has the same motherboard and the same problem.
According to the guide he linked I need to install the pve headers to get niccli to work - e.g.

apt install pve-headers-$(uname -r)

However, It seems like these headers don't exist in my case. My kernel is 6.8.4-2-pve, and when I try to run

apt install pve-headers-6.8.4-2-pve

I am being told no package was found..

 

[billy999]

https://www.supermicro.com/en/products/motherboard/H12SSL-NT

Ive got the H12SSL-CT one which has the same onboard NIC and I dont have this issue as they dont use the bnxt_re but the bnxt_en driver. But I did try updating before installing the new kernel and it failed with the same error. What does lshw say about your NIC configuration?
Mine contains this: driver=bnxt_en driverversion=6.8.12-1-pve ... firmware=214.4.9.10/pkg 214.0.286.18

I am currently trying to follow the steps described by @SkepticNerdGuy who apparently has the same motherboard and the same problem.
According to the guide he linked I need to install the pve headers to get niccli to work - e.g.

apt install pve-headers-$(uname -r)

However, It seems like these headers don't exist in my case. My kernel is 6.8.4-2-pve, and when I try to run

apt install pve-headers-6.8.4-2-pve

I am being told no package was found..

Have you tried the 6.8.4-3 kernel?

 

[ocmt58z1]

@billy999 Thanks for trying to help me out, appreciate it.
In my case it shows:

configuration: autonegotiation=on broadcast=yes depth=32 driver=bnxt_en driverversion=6.8.4-2-pve duplex=full firmware=226.0.145.0/pkg 226.1.107.1 latency=0 link=yes mode=1280x1024 multicast=yes port=twisted pair speed=1Gbit/s visual=truecolor xres=1280 yres=1024
                configuration: autonegotiation=on broadcast=yes driver=bnxt_en driverversion=6.8.4-2-pve firmware=226.0.145.0/pkg 226.1.107.1 latency=0 link=no multicast=yes port=twisted pair

Regarding the kernel, is there even a newer one available? Or would I have to manually build and install it? There are no updates available for me.

 

[billy999]

@billy999 Thanks for trying to help me out, appreciate it.
In my case it shows:

configuration: autonegotiation=on broadcast=yes depth=32 driver=bnxt_en driverversion=6.8.4-2-pve duplex=full firmware=226.0.145.0/pkg 226.1.107.1 latency=0 link=yes mode=1280x1024 multicast=yes port=twisted pair speed=1Gbit/s visual=truecolor xres=1280 yres=1024
                configuration: autonegotiation=on broadcast=yes driver=bnxt_en driverversion=6.8.4-2-pve firmware=226.0.145.0/pkg 226.1.107.1 latency=0 link=no multicast=yes port=twisted pair

Regarding the kernel, is there even a newer one available? Or would I have to manually build and install it? There are no updates available for me.

Im on the no-sub repo, the newest is 6.8.12-1 on there.

The NIC issue doesnt seem to appear when the fw is much older like mine. Either way, the Thomas Krenn bnxtnvm tool doesnt for work flashing a new one, you should try using the official Broadcom niccli utility. I didnt try niccli since I planned to fix it retroactively but the issue never came up for me (I also had my server running on 6.8.4-2 for quite some time).

 

[ocmt58z1]

@billy999 Nice, I forgot about that. I switched to the no-sub repo and I am now also on 6.8.12-1. Also got the headers installed.
This means niccli now works, and it lists my NICs:



I will now attempt to do the firmware upgrade described by @SkepticNerdGuy and see how it goes...

 

[ocmt58z1]

@billy999 Okay, I flashed firmware 229.0.141.0, and the good news its - it works. The network interface of the proxmox system is now up and the vmbr0 gets the IP I configured, as to be expected.

However, I noticed that on reboot, proxmox still gets stuck for a good minute, and eventually systemd-networkd-wait-online throws an error. Here are the details:



Not sure what causes this or how to fix it, but at least the system now gets an IP..

 

[billy999]

@billy999 Okay, I flashed firmware 229.0.141.0, and the good news its - it works. The network interface of the proxmox system is now up and the vmbr0 gets the IP I configured, as to be expected.

However, I noticed that on reboot, proxmox still gets stuck for a good minute, and eventually systemd-networkd-wait-online throws an error. Here are the details:

View attachment 73479

Not sure what causes this or how to fix it, but at least the system now gets an IP..

Its not able to get an IP while booting, the service is there to wait for one. Usually happens when auto vmbr0 is removed from the config

 

[ocmt58z1]

Its not able to get an IP while booting, the service is there to wait for one. Usually happens when auto vmbr0 is removed from the config

Hmm, auto vmbr0 is present in my config.

 

[billy999]

Hmm, auto vmbr0 is present in my config.

View attachment 73481

I just noticed that Proxmox/Debian shouldnt have this service enabled by default. AFAIK only ubuntu does. Maybe check your systemd-analyze critical-chain

 

[ocmt58z1]

I just noticed that Proxmox/Debian shouldnt have this service enabled by default. AFAIK only ubuntu does. Maybe check your systemd-analyze critical-chain

Did that, as you can see, yea, the systemd-networkd-wait-online took over 2 minutes...


Should I just disable the service, or do you have a better idea to troubleshoot this?

 

[billy999]

Did that, as you can see, yea, the systemd-networkd-wait-online took over 2 minutes...
View attachment 73482

Should I just disable the service, or do you have a better idea to troubleshoot this?

Yes you can disable it and check if the network still connects properly after boot.