LACP no UP with Broadcom BCM57416 NetXtreme-E 10GBase-T Ethernet

[Diego Mendoza]

Dear,

I have 2 network cards of the type "Broadcom BCM57416 NetXtreme-E 10GBase-T Ethernet".
I am using Proxmox 6, I made the recommended settings for LACP operation, the following happens:
1.- When the server finishes starting up, the LACP protocol does not work and consequently there is no network connectivity on the cards.
2.- When I turn off and on the switch ports, the LACP protocol works correctly.

I want to highlight that on the same server I have 2 1Gb network cards with the LACP configuration and it works perfectly.

Your help please.

In the log, when trying to build BOND with LACP the following happens:

Aug 15 11:33:18 prxfndr01 kernel: [ 13.702247] bnxt_en 0000:18:00.0 eno1np0: EEE is not active
Aug 15 11:33:18 prxfndr01 kernel: [ 13.702250] bnxt_en 0000:18:00.0 eno1np0: FEC autoneg off encodings: None
Aug 15 11:33:18 prxfndr01 kernel: [ 13.702341] bnxt_en 0000:18:00.1 eno2np1: NIC Link is Down
Aug 15 11:33:18 prxfndr01 kernel: [ 13.702731] bond1: Enslaving eno2np1 as a backup interface with a down link
Aug 15 11:33:18 prxfndr01 kernel: [ 13.714758] bond1: link status up again after 0 ms for interface eno1np0

 

[Stoiko Ivanov]

* hmm - have you tried updating all firmwares to the latest available version (NIC, BIOS/UEFI ,System)?
* which kernel are you running? (`pveversion -v`)

a quick search suggest that it might be related to older firmware - e.g.:
https://www.dell.com/community/Powe...en-module-crashes-on-14G-servers/td-p/6031769
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1814095

Hope this helps!

 

[Diego Mendoza]

Dear, the version of Proxmox that I am using is 6.
This is the data of the Proxmox packages.

~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.0-4 (running version: 6.0-4/2a719255)
pve-kernel-5.0: 6.0-5
pve-kernel-helper: 6.0-5
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph: 14.2.1-pve2
ceph-fuse: 14.2.1-pve2
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-2
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-5
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-3
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-5
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-5
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

 

[Stoiko Ivanov]

* hmm - have you tried updating all firmwares to the latest available version (NIC, BIOS/UEFI ,System)?

pve-kernel-5.0: 6.0-5

There is a new kernel which hit the enterprise repository recently - you could try that (in addition to updating the firmwares)

 

[Diego Mendoza]

Dear,

Thanks for responding, tell them that the firmware version of the 10Gb card is 21.40.20.00

 

[Stoiko Ivanov]

quick googling suggest that there is at least one newer firmware around - https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverid=6dr4k

Please update (along with all other components BIOS/UEFI/etc.)
Please reboot to use the latest kernel in your repository - pve-kernel-5.0.18-1-pve (pve-enterprise), pve-kernel-5.0.21-1-pve (pve-no-subscription)

Thanks!