Link detected: no on 4 nodes with 3 different NIC.

K

kez

Member
Mar 26, 2023
85
12
13
Hi,

Can someone point me in the right direction?

I have 4 nodes with Proxmox VE. I have 3 different 10G NICs. None of them seem to work on Proxmox, but they do work on a Live CD on other Linux distributions.

Example:

NIC port LED = YES
OS Link detected = No
Switch port LED = YES

Code: 
Node 3 Link

root@box1-node3:~# ethtool ens1f0np0
Settings for ens1f0np0:
    Supported ports: [ FIBRE ]
    Supported link modes:   10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  10000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: g
    Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: no

Node 3 NIC

root@box1-node3:~# lspci | grep -i eth
37:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
37:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
37:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
37:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)

Node 3 NIC driver

root@box1-node3:~# ethtool -i ens1f0np0
driver: i40e
version: 6.8.12-11-pve
firmware-version: 6.01 0x8000351b 255.65535.255
expansion-rom-version:
bus-info: 0000:37:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Node 4 Link

root@box1-node4:~# ethtool enp55s0f1
Settings for ens1f1:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  10000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 1
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000000 (0)

    Link detected: no

Node 4 NIC

root@box1-node4:~# lspci -nn | grep -i ether
37:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
37:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
37:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
37:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)

Node 4 NIC driver

root@box1-node4:~# ethtool -i enp55s0f0
driver: bnx2x
version: 6.8.12-11-pve
firmware-version: bc 7.6.55
expansion-rom-version:
bus-info: 0000:37:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Thanks for any advice, not sure where to go from here. It's also the same on node 1 & 2 with Emulex OneConnect NIC.
 
Proxmox Support Forum
Thread: [Broadcom NICs down after PVE 8.2 (Kernel 6.8)]
https://forum.proxmox.com/threads/broadcom-nics-down-after-pve-8-2-kernel-6-8.146185/

We had similar issues in the past. Users reported that Broadcom NICs (including BCM57416 and BCM57840, using the bnx2x or bnxt_en drivers) stopped working after upgrading to Proxmox VE 8.2 with kernel 6.8.x. The symptoms included:
  • Link detected: no
  • NIC LEDs and switch port lights on, but no link
  • Works fine in older kernels or Live CDs
  • Mostly affects OEM versions (Supermicro, Dell, HPE)

Update Broadcom firmware to at least version v229.x using the bnxtnvm utility provided by Thomas-Krenn:
Bash: 
wget https://www.thomas-krenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip
unzip bnxtnvm.zip
chmod +x bnxtnvm
./bnxtnvm listdev
./bnxtnvm -dev=<interface> install -online -y

Blacklist the bnxt_re (Infiniband) driver, which causes kernel issues even if unused:

Bash: 
echo "blacklist bnxt_re" > /etc/modprobe.d/blacklist-bnxt_re.conf
update-initramfs -u
reboot
 
lo0ip said:
Proxmox Support Forum
Thread: [Broadcom NICs down after PVE 8.2 (Kernel 6.8)]
https://forum.proxmox.com/threads/broadcom-nics-down-after-pve-8-2-kernel-6-8.146185/

We had similar issues in the past. Users reported that Broadcom NICs (including BCM57416 and BCM57840, using the bnx2x or bnxt_en drivers) stopped working after upgrading to Proxmox VE 8.2 with kernel 6.8.x. The symptoms included:
  • Link detected: no
  • NIC LEDs and switch port lights on, but no link
  • Works fine in older kernels or Live CDs
  • Mostly affects OEM versions (Supermicro, Dell, HPE)

Update Broadcom firmware to at least version v229.x using the bnxtnvm utility provided by Thomas-Krenn:
Bash: 
wget https://www.thomas-krenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip
unzip bnxtnvm.zip
chmod +x bnxtnvm
./bnxtnvm listdev
./bnxtnvm -dev=<interface> install -online -y

Blacklist the bnxt_re (Infiniband) driver, which causes kernel issues even if unused:

Bash: 
echo "blacklist bnxt_re" > /etc/modprobe.d/blacklist-bnxt_re.conf
update-initramfs -u
reboot
Click to expand...
Thank you :)
 
I have something similar still..

I'm running Proxmox v9.0.10. And did a kernel upgrade to 6.14.11-2, after which only one of the two SFP+ ports works now.. I tried to go back to the previous kernel.. but also no luck...

I tried to flash to the latest Broadcom firmware for my Broadcom P210p dual 10GBE network card (see link), also no luck.

I'm actually using the niccli tool not the bnxtnvm tool.. does that matter? What i the difference between these tools..?

Anyway.. I upgraded to the latest firmware which is v234.1.124.0 at the moment, using:

1758462260172.png

Code: 
niccli --dev 1 fw --update -f ./BCM957412A4120AC.pkg

Then always do a reboot after a firmware update of course.

So I got enp47s0f0np0 interface working without issues.. But when I try to use the 2nd port (enp47s0f1np1), the link will not come up at all.. Nothing. No green led on the device..

Also when I want to use that 2nd port network interface (enp47s0f1np1) in /etc/network/interfaces and assign it to the default vmbr0 interface.. Restarting the networking service under Debian.. Rebooting the server.. Nothing works.

Yes, I use the same DAC (Direct Attach Copper) cable - it's not a cable issue. So I try to just use the other port, physically switching the DAC cable to the other port. Edit the network interface settings. Restarting services / restarting the whole server..

It's really weird. So I can only use one of the two ports now. The other port just doesn't want to come up anymore. I tried dmesg as well, no errors.

Log when booting successfully on enp47s0f0np0 (interface comes up.. everything works as expected):

Bash: 
# dmesg --follow | grep bnx
[    1.517921] bnxt_en 0000:2f:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog
[    1.666793] bnxt_en 0000:2f:00.0 eth1: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 7ffff20000, node addr 6c:92:cf:6d:19:80
[    1.666802] bnxt_en 0000:2f:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    1.667707] bnxt_en 0000:2f:00.1 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog
[    1.693778] bnxt_en 0000:2f:00.1 eth2: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 7ffff10000, node addr 6c:92:cf:6d:19:81
[    1.693785] bnxt_en 0000:2f:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    2.003776] bnxt_en 0000:2f:00.0 enp47s0f0np0: renamed from eth1
[    2.004245] bnxt_en 0000:2f:00.1 enp47s0f1np1: renamed from eth2
[    6.601829] bnxt_en 0000:2f:00.1 enp47s0f1np1: entered allmulticast mode
[    6.601873] bnxt_en 0000:2f:00.1 enp47s0f1np1: entered promiscuous mode
[  133.872040] bnxt_en 0000:2f:00.1 enp47s0f1np1: left allmulticast mode
[  133.872047] bnxt_en 0000:2f:00.1 enp47s0f1np1: left promiscuous mode
[  134.442814] bnxt_en 0000:2f:00.0 enp47s0f0np0: entered allmulticast mode
[  134.442865] bnxt_en 0000:2f:00.0 enp47s0f0np0: entered promiscuous mode
[  136.561179] bnxt_en 0000:2f:00.0 enp47s0f0np0: NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
[  136.561187] bnxt_en 0000:2f:00.0 enp47s0f0np0: FEC autoneg off encoding: None

I switched the cable to the other port (physically on my network card). And changed the /etc/network/interfaces file to use the other interface, look so:
Code: 
iface enp47s0f1np1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.43/24
        gateway 192.168.1.1
        bridge-ports enp47s0f1np1
        bridge-stp off
        bridge-fd 0

And reboot the full server.

And and now it works.. of course.. sure... :rolleyes:

Bash: 
][root@pve ~]# dmesg --follow | grep bnx
[    1.498589] bnxt_en 0000:2f:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog
[    1.649872] bnxt_en 0000:2f:00.0 eth1: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 7ffff20000, node addr 6c:92:cf:6d:19:80
[    1.649880] bnxt_en 0000:2f:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    1.650809] bnxt_en 0000:2f:00.1 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog
[    1.686193] bnxt_en 0000:2f:00.1 eth2: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 7ffff10000, node addr 6c:92:cf:6d:19:81
[    1.686200] bnxt_en 0000:2f:00.1: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    2.009831] bnxt_en 0000:2f:00.0 enp47s0f0np0: renamed from eth1
[    2.010690] bnxt_en 0000:2f:00.1 enp47s0f1np1: renamed from eth2
[    6.651816] bnxt_en 0000:2f:00.1 enp47s0f1np1: entered allmulticast mode
[    6.651863] bnxt_en 0000:2f:00.1 enp47s0f1np1: entered promiscuous mode
[    8.797879] bnxt_en 0000:2f:00.1 enp47s0f1np1: NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none
[    8.797883] bnxt_en 0000:2f:00.1 enp47s0f1np1: FEC autoneg off encoding: None
 
Last edited:
Just to follow-up. OK so no clue why it now works again.. I might have been a BIOS issue who knows. Or the network card got overheated when in the BIOS... And goes crazy.

I have no idea frankly. If somebody else got these kind of weird issues with their Broadcom card, just let me / us know.
 
You must log in or register to reply here.
Top Bottom