Proxmox VE 9.2.21 - BCM57412 bnxt_en network failure after upgrade to kernel 7.0.14-20-pve

[ManuS]

Hi,
After upgrade on my test cluster using no-subscription repository, broadcom network card failed to load

Environment:

- Proxmox VE 9.2.21

- No-subscription repository

- Dell XC750-14 CORE

- Broadcom BCM57412 NetXtreme-E 10Gb

- Firmware 236.1.167.0

- Driver bnxt_en



Problem:

After upgrading to kernel 7.0.14-20-pve and rebooting, the management network becomes unavailable.



Observed errors:
bnxt_init_nic err: fffffff3
nic open fail (rc: fffffff3)
NETDEV WATCHDOG: transmit queue timed out

The server retains its network configuration, VLANs, bridges and routes are present, but cannot reach its default gateway.

Troubleshooting:
- Bond members detected
- Interfaces present
- IP configuration intact
- Gateway unreachable
- No configuration changes made

Workaround:
Booting back on kernel 7.0.14-19-pve immediately restores normal operation.


This issue appears to be related to the bnxt_en driver and/or BCM57412 firmware compatibility with kernel 7.0.14-20-pve.

Thank you for your help
Regards

 

[Stoiko Ivanov]

could you share the journal from a boot where the issue happens, and one where the NICs work?

else - I did not find it right away - but there seems to be a newer firmware version available for the NIC - https://dl.dell.com/FOLDER14995749M/2/Release_user_NX1-20260921.txt
via https://www.dell.com/support/product-details/de-at/product/poweredge-r750xa/drivers
maybe the newer version contains the fix for the issue.

 

[JensF]

Kernel 7.0.14-20 isn't in the no-sub-repo. You're using the test-repo. Right?

 

[ManuS]

Thank you for your reply.

Yes, right, I'm using the test repo for this cluster.

I identified the affected and working kernels:

Working kernel:
7.0.14-19-pve (boot 0)

Failing kernel:
7.0.14-20-pve (boot -1)

The issue is fully reproducible:
- booting 7.0.14-20-pve results in network loss
- booting 7.0.14-19-pve immediately restores connectivity

I attached:
- kernel journal from the failing boot (7.0.14-20-pve)
- kernel journal from the working boot (7.0.14-19-pve)
- filtered bnxt_en related messages

Regarding the firmware, the issue was reproduced using firmware version 236.1.167.0. I have not yet updated the firmware, so both tests were performed with the same firmware and identical configuration.

 

[Stoiko Ivanov]

Hm - the failing boot does not only fail with the NIC - I also see disk-errors in the journal:

Sep 29 13:33:51 proxmox10 kernel: I/O error, dev dm-7, sector 2214873984 op 0x1WRITE) flags 0x8800 phys_seg 1 prio class 2
Sep 29 13:33:51 proxmox10 kernel: I/O error, dev dm-7, sector 2254438784 op 0x0READ) flags 0x0 phys_seg 16 prio class 2
Sep 29 13:33:51 proxmox10 kernel: I/O error, dev dm-7, sector 2238660344 op 0x0READ) flags 0x0 phys_seg 4 prio class 2
Sep 29 13:33:51 proxmox10 kernel: I/O error, dev dm-7, sector 2215200952 op 0x1WRITE) flags 0x8800 phys_seg 6 prio class 2
Sep 29 13:33:51 proxmox10 kernel: I/O error, dev dm-7, sector 2216218288 op 0x1WRITE) flags 0x8800 phys_seg 11 prio class 2
Sep 29 13:33:51 proxmox10 kernel: I/O error, dev dm-7, sector 2223398624 op 0x1WRITE) flags 0x8800 phys_seg 3 prio class 2

on a hunch - could you check the BIOS for SR-IOV global and I/OAT DMA - see:
https://pve.proxmox.com/wiki/Roadmap#9.1-known-issues
(the issue was reported with older Dell Servers, but we never could pin it to a certain modell-number or firmware/bios version)

comparing the logs between -20 and -19 shows the following lines which might point to an issue in that area (or iommu) as first substantial deviation:

DMAR: DRHD: handling fault status reg 2

apart from the boot issue with the new kernel - I'd check out if you might have some misconfiguration or a loop in your network:

proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)
proxmox10 kernel: vmbr1: received packet on bond0 with own address as source address (addr:84:16:0c:71:db:30, vlan:1)