ConnectX-3 QSFP Ethernet Interface Shows No-Carrier

[lwilbert]

I'm having issues w/ my Proxmox installation w/ the installed ConnectX-3 card. I believe I have Proxmox configured properly, the ports of the card show up in Proxmox, but the card shows NO-CARRIER, shows no activity lights on the server or switch side ports, and is not passing any packets whatsoever. We're able to manage the instance over its onboard interface, but the ConnectX-3 card just isn't showing any activity whatsoever.

I have Promox 7.3 VE installed on a Dell XL R730 server w/ a 2-port QSFP Mellanox MT27500 Family card using configured using NVIDA MFT to be on Ethernet mode. The card is currently wired up to a Juniper switch, the connecting of which has been configured as a trunk port.

Both ports on the card show NO-CARRER (only enp5s0d1 is physically wired, but even if we swap there's no change):

root@pmox1:~# ip a
6: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether e4:1d:2d:bc:f8:51 brd ff:ff:ff:ff:ff:ff
7: enp5s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
    link/ether e4:1d:2d:bc:f8:52 brd ff:ff:ff:ff:ff:ff

The card is detected as an OS device:

root@pmox1:~# lspci | grep Mella
05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

Modules are loaded:

root@pmox1:~# lsmod | grep mlx
mlx4_ib               245760  0
ib_uverbs             167936  1 mlx4_ib
ib_core               389120  6 rdma_cm,mlx4_ib,iw_cm,ib_iser,ib_uverbs,ib_cm
mlx4_en               155648  0
mlx4_core             401408  2 mlx4_ib,mlx4_en

The card has both ports set to ETH:

root@pmox1:~# mlxconfig query

Device #1:
----------

Device type:    ConnectX3     
Device:         0000:05:00.0  

Configurations:                                      Next Boot
         SRIOV_EN                                    True(1)       
         NUM_OF_VFS                                  8             
         LINK_TYPE_P1                                ETH(2)        
         LINK_TYPE_P2                                ETH(2)        
         LOG_BAR_SIZE                                3             
         BOOT_PKEY_P1                                0             
         BOOT_PKEY_P2                                0             
         BOOT_OPTION_ROM_EN_P1                       True(1)       
         BOOT_VLAN_EN_P1                             False(0)      
         BOOT_RETRY_CNT_P1                           0             
         LEGACY_BOOT_PROTOCOL_P1                     PXE(1)        
         BOOT_VLAN_P1                                1             
         BOOT_OPTION_ROM_EN_P2                       True(1)       
         BOOT_VLAN_EN_P2                             False(0)      
         BOOT_RETRY_CNT_P2                           0             
         LEGACY_BOOT_PROTOCOL_P2                     PXE(1)        
         BOOT_VLAN_P2                                1             
         IP_VER_P1                                   IPv4(0)       
         IP_VER_P2                                   IPv4(0)       
         CQ_TIMESTAMP                                True(1)

I've set SRIOV passthrough to enabled in BIOS.

We're able to see the card in the GUI interface, and create a bridge on it (which is shown in the output of ip a above as vmbr1). The card is good, as we were previously using it for another application, and connecting it up to one of our Infiniband switches shows it light up. We've tested the cable we're using with it, which checks out.

I attempted upgrading the Mellanox firmware on one server

Additionally, I attempted installing Proxmox on a newer server w/ a ConnectX-5 card, and that one came up just fine without issue. We're kinda scratching our heads on this one.

It feels like a driver issue to me, but everything appears to work... except for the actual connectivity, as we're seeing zero packets cross over.
Any idea what needs to be done to resolve this? Thank you.

 

[lwilbert]

Update:

I installed a newer ConnectX-4 card into the system, and the links activated just fine. So it seems to only be when these ConnectX-3 cards are in use that Proxmox has this issue. We do wish to use the ConnectX-3 cards, as the other ones we have for other uses.

 

[proxwolfe]

Same issue here with CX3-Pro card and PVE8.2

Has anyone got those to work?