Problem installing Mellanox ConnectX-3 CX311A 10g NIC

G

geozapata

Member
Mar 17, 2022
5
0
6
46
Good afternno all...

I have been trying to install and configure a Mellanox ConnectX-3 CX311A 10bge nic on a Proxmox v8.2.4 server running on an HP Elite 800 G4. The Proxmox GUI does not show the card but dmesg and lspci recognize it.

What I've done so far:

Physically install card into full length PCIE slot

Bash: 
root@elitelab:~# lspci | grep -i mellanoX
01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

root@elitelab:~# dmesg | grep -i mellanoX
[    1.088170] mlx4_core: Mellanox ConnectX core driver v4.0-0
[   10.556130] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[   10.608794] <mlx4_ib> mlx4_ib_probe: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0

Following this post https://forum.proxmox.com/threads/pve-6-3-infiniband-error.86682/
I have added 'acpi=off pci=realloc=off' in /etc/default/grub

Any ideas?
 
Good news card is working correctly and detected. Please make sure you set the card in Ethernet mode with Mellanox/Nvidia tools. Once that is done, you will likely need manually add/activate the card in Proxmox, with ifup or through the gui.
 
I downloaded and installed mft-4.28.0-92-x86_64-deb.tgz from the MFT Download Center.


Here is the output when I the commands:

Bash: 
root@elitelab:~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
Unloading MST PCI configuration module (unused) - Success


root@elitelab:~# mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module is not loaded

PCI Devices:
------------

    No devices were found.


Still struggling
 
Seems like a difficult card to get into Ethernet mode, might be time to upgrade or if you have two cards use Infiniband when they connect to each other directly.

As a last resort I would create a few VMs that are suppose to support the card and do a Passthrough of the card into the VM to see if it works. Ubuntu, Windows Server, FreeBSD..
If none of them see it as Ethernet card either, time to move on buying a card where you can activate Ethernet mode succesfully.
 
If you need 10GbE with fiber, and you have a Mellanox compatible Transciever already, then maybe a ConnectX-4 from ebay
https://www.ebay.com/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=connectx-4+10+GbE&_sacat=0
Just make sure it says "10 GbE" in the description.

If you have the option of 10GbE via RJ45 twisted pair Ethernet cable, then for 10 GbE I have this card and it works well. I put an extra fan over it keep temps OK, since it arrived without the fan on the heatsink, pictured. Heatsink was included.
https://www.amazon.com/dp/B08FBBX8MX?ref=ppx_yo2ov_dt_b_product_details&th=1
 
geozapata said:
I downloaded and installed mft-4.28.0-92-x86_64-deb.tgz from the MFT Download Center.


Here is the output when I the commands:

Bash: 
root@elitelab:~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
Unloading MST PCI configuration module (unused) - Success


root@elitelab:~# mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module is not loaded

PCI Devices:
------------

    No devices were found.


Still struggling
Click to expand...
The only version of mft that supports the older Connectx-3 cards is mft-4.22.1-417-x86_64-deb
Refer to this guide https://blog.joeplaa.com/mellanox-connectx-3-on-proxmox-8-2/
 
You must log in or register to reply here.
Top Bottom