Mellanox ConnectX-3 - no mlx4_en, only IB

[jimibob]

Hello all,

I placed a mellanox ConnectX-3 into my Proxmox 7.2 post install. I cannot see it in the web GUI network window, nor is it in /etc/interfaces. Seems like the mlx4_en driver is missing from what I can tell (it is set to Ethernet mode). Could someone help me?

root@proxmox:~# lshw -class network -businfo
Bus info          Device      Class          Description
========================================================
pci@0000:04:00.0              network        MT27500 Family [ConnectX-3]
pci@0000:02:00.0  eno3        network        NetXtreme BCM5720 2-port Gigabit Etherne
pci@0000:02:00.1  eno4        network        NetXtreme BCM5720 2-port Gigabit Etherne
pci@0000:01:00.0  eno1        network        NetXtreme BCM5720 2-port Gigabit Etherne
pci@0000:01:00.1  eno2        network        NetXtreme BCM5720 2-port Gigabit Etherne
                  vmbr0       network        Ethernet interface
                  tap500i0    network        Ethernet interface
                  fwbr500i0   network        Ethernet interface
                  fwpr500p0   network        Ethernet interface
                  fwln500i0   network        Ethernet interface
root@proxmox:~# dmesg | grep mlx
[    2.873839] mlx4_core: Mellanox ConnectX core driver v4.0-0
[    2.873875] mlx4_core: Initializing 0000:04:00.0
[    8.887676] mlx4_core 0000:04:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[    8.887945] mlx4_core 0000:04:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[    9.037615] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[    9.038690] <mlx4_ib> mlx4_ib_add: counter index 0 for port 1 allocated 0
root@proxmox:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.200.75/24
        gateway 192.168.200.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        up /bin/ip link set vmbr0 promisc on

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual
root@proxmox:~# lspci | grep Mellanox
04:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

 

[jimibob]

Solved: Some how the card had reverted back to IB mode.

 

[kenneth104]

You should configure manually in CLI, i'm show you some code ..

apt update && apt upgrade -y
apt install opensm htop nload pve-headers -y
modprobe ib_ipoib
echo "ib_ipoib" >> /etc/modules-load.d/infiniband.conf
systemctl enable opensm
systemctl restart opensm

##  add in /etc/network/interfaces

auto ibp4s0
iface ibp4s0 inet static
        address  10.10.10.1
        netmask  255.255.255.0
        pre-up modprobe ib_ipoib
        pre-up modprobe ib_umad
        pre-up echo connected > /sys/class/net/ibp4s0/mode
        mtu 65520