Hello everyone,
I'm facing an intermittent issue with the integrated 10GbE ports on a Dell PowerEdge R630 running Proxmox VE and I'd appreciate any insight from people who have dealt with the same hardware.
Hardware
Symptom
On some boots, the bnxt_en driver fails to probe the BCM57416 10G ports (PCI 01:00.0 / 01:00.1) with error -16, and the interfaces (eno2np1 etc.) do not appear in the system. The 1GbE ports on the same rNDC keep working, although they show repeated link flaps in the iDRAC logs.
Behaviour observed
Questions
Thanks in advance!
I'm facing an intermittent issue with the integrated 10GbE ports on a Dell PowerEdge R630 running Proxmox VE and I'd appreciate any insight from people who have dealt with the same hardware.
Hardware
- Dell PowerEdge R630, iDRAC 8 Enterprise
- rNDC: Broadcom 5720/57416 combo (2x 10GBase-T + 2x 1GbE on the same module)
- PVE version: pve-manager/9.2.21, kernel: 7.0.14-19-pve
- BIOS: 2.19.0, iDRAC firmware: 2.86.86.86
Symptom
On some boots, the bnxt_en driver fails to probe the BCM57416 10G ports (PCI 01:00.0 / 01:00.1) with error -16, and the interfaces (eno2np1 etc.) do not appear in the system. The 1GbE ports on the same rNDC keep working, although they show repeated link flaps in the iDRAC logs.
Behaviour observed
- The failure is non-deterministic: it appears after some reboots and not others.
- In the failing state, the 10G controller is missing from the iDRAC Firmware Inventory (racadm swinventory).
- Physically reseating the rNDC fixed it once; a prolonged AC power removal fixed it another time, but the failure came back after a subsequent reboot.
- A full power cycle did not resolve it on another occasion.
- Applying the Dell Update Package for Broadcom NetXtreme-E (22.21.06.80) fails with "Update Package is not compatible with your system configuration", consistent with the 10G component not being enumerated at that moment.
- iDRAC Lifecycle Log: no memory ECC errors. Recurring events of the riser cable/interconnect being disconnected and reconnected on its own, and iDRAC alternating between the correct controller name ("Broadcom Adv. Dual 10GBASE-T Ethernet") and a generic "NIC" across consecutive reboots.
- Working boot for reference: both functions enumerate, link negotiated at PCIe x8 8.0 GT/s (63.008 Gb/s available).
Questions
- Has anyone seen bnxt_en error -16 on R630 rNDC modules, and was it hardware (rNDC/riser) or something addressable via firmware/kernel parameters?
- Is there a known interaction between the BCM57416 and specific kernel/driver versions in PVE 8/9?
- Any further diagnostics worth running (e.g. PCIe AER logs, lspci -vvv while in the failing state) before opening a replacement with Dell?
Thanks in advance!