Link detected: no on 4 nodes with 3 different NIC.

K

kez

Member
Mar 26, 2023
75
11
13
Hi,

Can someone point me in the right direction?

I have 4 nodes with Proxmox VE. I have 3 different 10G NICs. None of them seem to work on Proxmox, but they do work on a Live CD on other Linux distributions.

Example:

NIC port LED = YES
OS Link detected = No
Switch port LED = YES

Code: 
Node 3 Link

root@box1-node3:~# ethtool ens1f0np0
Settings for ens1f0np0:
    Supported ports: [ FIBRE ]
    Supported link modes:   10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  10000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: g
    Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: no

Node 3 NIC

root@box1-node3:~# lspci | grep -i eth
37:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
37:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
37:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
37:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

Node 3 NIC driver

root@box1-node3:~# ethtool -i ens1f0np0
driver: i40e
version: 6.8.12-11-pve
firmware-version: 6.01 0x8000351b 255.65535.255
expansion-rom-version:
bus-info: 0000:37:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Node 4 Link

root@box1-node4:~# ethtool enp55s0f1
Settings for ens1f1:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  10000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 1
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000000 (0)

    Link detected: no

Node 4 NIC

root@box1-node4:~# lspci -nn | grep -i ether
37:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
37:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
37:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
37:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)

Node 4 NIC driver

root@box1-node4:~# ethtool -i enp55s0f0
driver: bnx2x
version: 6.8.12-11-pve
firmware-version: bc 7.6.55
expansion-rom-version:
bus-info: 0000:37:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Thanks for any advice, not sure where to go from here. It's also the same on node 1 & 2 with Emulex OneConnect NIC.
 
Proxmox Support Forum
Thread: [Broadcom NICs down after PVE 8.2 (Kernel 6.8)]
https://forum.proxmox.com/threads/broadcom-nics-down-after-pve-8-2-kernel-6-8.146185/

We had similar issues in the past. Users reported that Broadcom NICs (including BCM57416 and BCM57840, using the bnx2x or bnxt_en drivers) stopped working after upgrading to Proxmox VE 8.2 with kernel 6.8.x. The symptoms included:
  • Link detected: no
  • NIC LEDs and switch port lights on, but no link
  • Works fine in older kernels or Live CDs
  • Mostly affects OEM versions (Supermicro, Dell, HPE)

Update Broadcom firmware to at least version v229.x using the bnxtnvm utility provided by Thomas-Krenn:
Bash: 
wget https://www.thomas-krenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip
unzip bnxtnvm.zip
chmod +x bnxtnvm
./bnxtnvm listdev
./bnxtnvm -dev=<interface> install -online -y

Blacklist the bnxt_re (Infiniband) driver, which causes kernel issues even if unused:

Bash: 
echo "blacklist bnxt_re" > /etc/modprobe.d/blacklist-bnxt_re.conf
update-initramfs -u
reboot
 
You must log in or register to reply here.
Top Bottom