Mellanox ConnectX-3 Interface Not Working on Proxmox

[tarantsevm]

Hello,

I am currently experiencing an issue with my Mellanox ConnectX-3 network card on my Proxmox setup. Specifically, one of the interfaces is not functioning correctly.

Here are the details of my setup:

Proxmox version: 8.3.0 Linux
Network card model: Mellanox ConnectX-3 MCX354A-FCB_A2-A5
Driver version: 2.42.5000
Kernel version: 6.8.12-4-pve

Issue Description:​

• The card is detected by the system, but one of the interfaces does not appear to work properly.
• I have tried different interface states on both the server and the switch, as well as using different cables, but none of these actions had any effect.
• The problematic interface indicates that there is no physical link.

Logs and Outputs:​

Here are some outputs for reference:

Spoiler: mlxfwmanager --query

Device #1:
----------

Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: /dev/mst/mt4099_pci_cr0
Port1 MAC: f45214759bb1
Port2 MAC: f45214759bb2
Versions: Current Available
FW 2.42.5000 N/A

Spoiler: mlxconfig -d /dev/mst/mt4099_pciconf0 q

Device #1:
----------

Device type: ConnectX3
Device: /dev/mst/mt4099_pciconf0

Configurations: Next Boot
SRIOV_EN False(0)
NUM_OF_VFS 8
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 False(0)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 None(0)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 False(0)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 None(0)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)

Spoiler: dmesg output:

root@hv-pve-81:~# dmesg | grep mlx
[ 2.138229] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 2.138698] mlx4_core: Initializing 0000:21:00.0
[ 2.139040] mlx4_core 0000:21:00.0: enabling device (0000 -> 0002)
[ 8.524239] mlx4_core 0000:21:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[ 8.537714] mlx4_core 0000:21:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 8.827444] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 8.828469] mlx4_en 0000:21:00.0: Activating port:1
[ 8.846406] mlx4_en: 0000:21:00.0: Port 1: Using 32 TX rings
[ 8.850106] mlx4_en: 0000:21:00.0: Port 1: Using 16 RX rings
[ 8.858429] mlx4_en: 0000:21:00.0: Port 1: Initializing port
[ 8.863310] mlx4_en 0000:21:00.0: registered PHC clock
[ 8.871860] mlx4_en 0000:21:00.0: Activating port:2
[ 8.881042] mlx4_en: 0000:21:00.0: Port 2: Using 32 TX rings
[ 8.883070] mlx4_en: 0000:21:00.0: Port 2: Using 16 RX rings
[ 8.890448] mlx4_en: 0000:21:00.0: Port 2: Initializing port
[ 8.903587] <mlx4_ib> mlx4_ib_probe: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[ 8.906584] <mlx4_ib> mlx4_ib_probe: counter index 2 for port 1 allocated 1
[ 8.907022] <mlx4_ib> mlx4_ib_probe: counter index 3 for port 2 allocated 1
[ 8.921083] mlx4_core 0000:21:00.0 ens8: renamed from eth0
[ 8.944088] mlx4_core 0000:21:00.0 ens8d1: renamed from eth1
[ 11.143218] mlx4_en: ens8: Link Up
[ 13.923153] mlx4_core 0000:21:00.0 ens8: entered allmulticast mode
[ 13.926096] mlx4_core 0000:21:00.0 ens8: entered promiscuous mode
[ 13.975742] mlx4_en: ens8: Steering Mode 1
[ 14.003418] mlx4_en: ens8: Link Up

Spoiler: ip link or ifconfig output:

root@hv-pve-81:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp65s0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a0:36:bc:cc:a4:a8 brd ff:ff:ff:ff:ff:ff
3: enx3ab3991a05c9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 3a:b3:99:1a:05:c9 brd ff:ff:ff:ff:ff:ff
4: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether f4:52:14:75:9b:b1 brd ff:ff:ff:ff:ff:ff
altname enp33s0
5: ens8d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
link/ether f4:52:14:75:9b:b2 brd ff:ff:ff:ff:ff:ff
altname enp33s0d1
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether f4:52:14:75:9b:b1 brd ff:ff:ff:ff:ff:ff
inet 10.100.106.81/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::f652:14ff:fe75:9bb1/64 scope link
valid_lft forever preferred_lft forever
7: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether f4:52:14:75:9b:b2 brd ff:ff:ff:ff:ff:ff
inet 10.20.20.81/24 scope global vmbr1
valid_lft forever preferred_lft forever

Spoiler: /etc/network/interfaces configuration:

root@hv-pve-81:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface ens8 inet manual

iface enp65s0np0 inet manual

iface enx3ab3991a05c9 inet manual

iface ens8d1 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.100.106.81/24
gateway 10.100.106.1
bridge-ports ens8
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 10.20.20.81/24
bridge-ports ens8d1
bridge-stp off
bridge-fd 0

source /etc/network/interfaces.d/*

I would greatly appreciate any suggestions or guidance on how to resolve this issue. Thank you in advance for your help!

 

[tarantsevm]

In addition, I disabled everything related to InfiniBand, but the issue persists.

Spoiler: dmesg | grep mlx4

root@hv-pve-81:~# dmesg | grep mlx4
ethtool ens8d1
[ 2.239448] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 2.240395] mlx4_core: Initializing 0000:21:00.0
[ 2.240763] mlx4_core 0000:21:00.0: enabling device (0000 -> 0002)
[ 8.658938] mlx4_core 0000:21:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[ 8.671750] mlx4_core 0000:21:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[ 8.962778] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 8.963729] mlx4_en 0000:21:00.0: Activating port:1
[ 8.985831] mlx4_en: 0000:21:00.0: Port 1: Using 32 TX rings
[ 8.986177] mlx4_en: 0000:21:00.0: Port 1: Using 16 RX rings
[ 8.989859] mlx4_en: 0000:21:00.0: Port 1: Initializing port
[ 9.001100] mlx4_en 0000:21:00.0: registered PHC clock
[ 9.012457] mlx4_en 0000:21:00.0: Activating port:2
[ 9.015600] mlx4_en: 0000:21:00.0: Port 2: Using 32 TX rings
[ 9.016296] mlx4_en: 0000:21:00.0: Port 2: Using 16 RX rings
[ 9.016982] mlx4_en: 0000:21:00.0: Port 2: Initializing port
[ 9.034113] mlx4_core 0000:21:00.0 ens8d1: renamed from eth1
[ 9.045138] mlx4_core 0000:21:00.0 ens8: renamed from eth0
[ 10.981270] mlx4_en: ens8: Link Up
[ 13.807156] mlx4_core 0000:21:00.0 ens8: entered allmulticast mode
[ 13.807901] mlx4_core 0000:21:00.0 ens8: entered promiscuous mode
[ 13.832327] mlx4_en: ens8: Steering Mode 1
[ 13.858870] mlx4_en: ens8: Link Up
[ 13.914162] mlx4_core 0000:21:00.0 ens8d1: entered allmulticast mode
[ 13.915033] mlx4_core 0000:21:00.0 ens8d1: entered promiscuous mode
[ 13.939215] mlx4_en: ens8d1: Steering Mode 1
[ 13.964965] mlx4_en: ens8d1: Link Down