[SOLVED] Mellanox ConnectX-3 Pro (MT27520) not showing under PVE 8

[Ertu]

Hi,

Just bought this card to do some speed test between nodes and vms and validate them for production use.
Dell 19RNV Mellanox CX322A
But in network tab, i can't see it :/

I know the card is recognized by my server

root@proxmox2:~# lspci |grep -i mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

But not active under proxmox
root@proxmox2:~# ip -br link show
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eno1 UP d4:ae:52:ea:be:9b <BROADCAST,MULTICAST,UP,LOWER_UP>
eno2 DOWN d4:ae:52:ea:be:9d <BROADCAST,MULTICAST>
eno3 DOWN d4:ae:52:ea:be:9f <BROADCAST,MULTICAST>
eno4 DOWN d4:ae:52:ea:be:a1 <BROADCAST,MULTICAST>
enp9s0f0 UP a0:36:9f:6d:06:dc <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP>
enp9s0f1 UP a0:36:9f:6d:06:dc <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP>
vmbr0 UP d4:ae:52:ea:be:9b <BROADCAST,MULTICAST,UP,LOWER_UP>
bond0 UP a0:36:9f:6d:06:dc <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP>
vmbr3 UP a0:36:9f:6d:06:dc <BROADCAST,MULTICAST,UP,LOWER_UP>

Do i miss something ? Maybe shoul i install missing driver ?
Can you please help me with this ?
Thanks

 

[cwt]

What about

enp9s0f0
enp9s0f1

Is this another NIC? If not this should be your Mellanox card.

 

[Ertu]

Hi,

Unfortunately no, this is Intel X520 adapter

root@proxmox2:~# lspci -k | grep Intel
09:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) Subsystem: Intel Corporation 10GbE 2P X520 Adapter
09:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) Subsystem: Intel Corporation 10GbE 2P X520 Adapter

 

[Ertu]

I have nothing starting with enp6sf0 or enp6sf1 for the Mellanox adapter

 

[Ertu]

I'am really new to linux and Proxmox
What does this message mean's
root@proxmox2:~/mft-4.25.0-62-x86_64-deb# dmesg | grep mlx
[ 4.053108] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 4.053147] mlx4_core: Initializing 0000:06:00.0
[ 4.053273] mlx4_core 0000:06:00.0: Missing UAR, aborting

What is UAR ??

 

[uk_user]

The following may be related:

https://forum.proxmox.com/threads/pve-6-3-infiniband-error.86682/
https://forums.developer.nvidia.com/t/mlx4-core-missing-uar-aborting/207322

 

[Ertu]

Thanks will read that

 

[Ertu]

Ok i have done modification in grub default, now my node doens't start anymore
Trying to figure out if the command in grub default "pci=realloc=off" hangs my server or the "ACPI=off"

 

[Ertu]

Ok the ACPI=off hangs my server.
Now i can see the 2 network cards.

Thanks for your support.

 

[Ertu]

Testing this nic cards and it's Awesome with host to linux ct
i got 9.4 Gb/sec, amazing
from ct to ct same speed

but and that's the point, from or to a win 11 vm, only have 3 Gb/s, balloning is off
I already search for a solution to get better speed as i plan to migrate from esxi to proxmox my servers, but speed is really important.
Does anybody found a way to fix this issue ?

 

[Ertu]

Here is my conf file for Windows 11 vm

root@proxmox2:~# cat /etc/pve/qemu-server/102.conf
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 8
cpu: host,flags=+aes
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-8.0
memory: 8192
meta: creation-qemu=8.0.2,ctime=1694604289
name: WIN11TEST
net0: virtio=E6B:F6:00:4D:99,bridge=vmbr1
numa: 0
ostype: win11
scsi0: local-lvm:vm-102-disk-1,cache=writeback,discard=on,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=d2696760-f2a6-41b0-99dd-433a9d78c7c1
sockets: 1
tpmstate0: local-lvm:vm-102-disk-2,size=4M,version=v2.0
vga: qxl,memory=32
vmgenid: 67920eb2-6a6c-4757-8348-b6a7271763cd

 

[Ertu]

 

[Ertu]

best i have achieved
i lost another Gb/s from my PC with 10Gb nic and connected to a mikrotik switch (3 Gb/s)

 

[Ertu]

From CT to CT

root@TESTFIBRE:~# iperf3 -c 10.1.212.12
Connecting to host 10.1.212.12, port 5201
[ 5] local 10.1.212.11 port 55064 connected to 10.1.212.12 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.14 GBytes 9.78 Gbits/sec 0 468 KBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 9.84 Gbits/sec 0 468 KBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 0 468 KBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 9.85 Gbits/sec 0 468 KBytes
[ 5] 4.00-5.00 sec 1.14 GBytes 9.78 Gbits/sec 0 468 KBytes
[ 5] 5.00-6.00 sec 1.14 GBytes 9.78 Gbits/sec 0 468 KBytes
[ 5] 6.00-7.00 sec 1.13 GBytes 9.67 Gbits/sec 0 468 KBytes
[ 5] 7.00-8.00 sec 1.13 GBytes 9.73 Gbits/sec 0 468 KBytes
[ 5] 8.00-9.00 sec 1.14 GBytes 9.80 Gbits/sec 0 468 KBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 9.85 Gbits/sec 0 468 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.4 GBytes 9.80 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.4 GBytes 9.80 Gbits/sec receiver