I'm having issues w/ my Proxmox installation w/ the installed ConnectX-3 card. I believe I have Proxmox configured properly, the ports of the card show up in Proxmox, but the card shows NO-CARRIER, shows no activity lights on the server or switch side ports, and is not passing any packets whatsoever. We're able to manage the instance over its onboard interface, but the ConnectX-3 card just isn't showing any activity whatsoever.
I have Promox 7.3 VE installed on a Dell XL R730 server w/ a 2-port QSFP Mellanox MT27500 Family card using configured using NVIDA MFT to be on Ethernet mode. The card is currently wired up to a Juniper switch, the connecting of which has been configured as a trunk port.
Both ports on the card show NO-CARRER (only enp5s0d1 is physically wired, but even if we swap there's no change):
The card is detected as an OS device:
Modules are loaded:
The card has both ports set to ETH:
I've set SRIOV passthrough to enabled in BIOS.
We're able to see the card in the GUI interface, and create a bridge on it (which is shown in the output of ip a above as vmbr1). The card is good, as we were previously using it for another application, and connecting it up to one of our Infiniband switches shows it light up. We've tested the cable we're using with it, which checks out.
I attempted upgrading the Mellanox firmware on one server
Additionally, I attempted installing Proxmox on a newer server w/ a ConnectX-5 card, and that one came up just fine without issue. We're kinda scratching our heads on this one.
It feels like a driver issue to me, but everything appears to work... except for the actual connectivity, as we're seeing zero packets cross over.
Any idea what needs to be done to resolve this? Thank you.
I have Promox 7.3 VE installed on a Dell XL R730 server w/ a 2-port QSFP Mellanox MT27500 Family card using configured using NVIDA MFT to be on Ethernet mode. The card is currently wired up to a Juniper switch, the connecting of which has been configured as a trunk port.
Both ports on the card show NO-CARRER (only enp5s0d1 is physically wired, but even if we swap there's no change):
Code:
root@pmox1:~# ip a
6: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e4:1d:2d:bc:f8:51 brd ff:ff:ff:ff:ff:ff
7: enp5s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
link/ether e4:1d:2d:bc:f8:52 brd ff:ff:ff:ff:ff:ffThe card is detected as an OS device:
Code:
root@pmox1:~# lspci | grep Mella
05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]Modules are loaded:
Code:
root@pmox1:~# lsmod | grep mlx
mlx4_ib 245760 0
ib_uverbs 167936 1 mlx4_ib
ib_core 389120 6 rdma_cm,mlx4_ib,iw_cm,ib_iser,ib_uverbs,ib_cm
mlx4_en 155648 0
mlx4_core 401408 2 mlx4_ib,mlx4_enThe card has both ports set to ETH:
Code:
root@pmox1:~# mlxconfig query
Device #1:
----------
Device type: ConnectX3
Device: 0000:05:00.0
Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 8
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 True(1)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)I've set SRIOV passthrough to enabled in BIOS.
We're able to see the card in the GUI interface, and create a bridge on it (which is shown in the output of ip a above as vmbr1). The card is good, as we were previously using it for another application, and connecting it up to one of our Infiniband switches shows it light up. We've tested the cable we're using with it, which checks out.
I attempted upgrading the Mellanox firmware on one server
Additionally, I attempted installing Proxmox on a newer server w/ a ConnectX-5 card, and that one came up just fine without issue. We're kinda scratching our heads on this one.
It feels like a driver issue to me, but everything appears to work... except for the actual connectivity, as we're seeing zero packets cross over.
Any idea what needs to be done to resolve this? Thank you.
Last edited: