Hi,
I can't get my CX-6 LX to work with SR_IOV and Proxmox 9.1.1.
SR_IOV is enabled in the motherboard BIOS and in the card's firmware, and I'm using this UDEV rule.
PCIe pass-through works without problems on this system, tested with the iGPU and with a U2 NVMe.
I'm using mlx5_core, is it better to install the OFED drivers from Nvidia?
Here is the error message I receive when starting the VM.
I installed Ubuntu Server 25.10 on a second SSD to check if I have the same problem with it, but SR-IOV works with Ubuntu Server.
I've also tried it with this script, same result.
tutorial-proxmox-8-mellanox-infiniband-and-sr-iov
Here is some relevant information.
The motherboard BIOS and nic firmware are up to date and kernel version is 6.17.2-1-pve.
Configuration of the VM.
Everything looks good to me
I can't get my CX-6 LX to work with SR_IOV and Proxmox 9.1.1.
SR_IOV is enabled in the motherboard BIOS and in the card's firmware, and I'm using this UDEV rule.
Code:
ACTION=="add", SUBSYSTEM=="net", ATTRS{vendor}=="0x15b3", ATTRS{device}=="0x101f", ATTR{device/sriov_drivers_autoprobe}="0", ATTR{device/sriov_numvfs}="4"PCIe pass-through works without problems on this system, tested with the iGPU and with a U2 NVMe.
I'm using mlx5_core, is it better to install the OFED drivers from Nvidia?
Here is the error message I receive when starting the VM.
Code:
Nov 21 01:24:36 tvpve pvedaemon[1601]: <root@pam> starting task UPID:tvpve:00011F53:00151E69:691FB144:qmstart:101:root@pam:
Nov 21 01:24:36 tvpve pvedaemon[73555]: start VM 101: UPID:tvpve:00011F53:00151E69:691FB144:qmstart:101:root@pam:
Nov 21 01:24:36 tvpve kernel: vfio-pci 0000:02:00.0: resetting
Nov 21 01:24:36 tvpve kernel: vfio-pci 0000:02:00.0: reset done
Nov 21 01:24:36 tvpve pvedaemon[73555]: error writing '0000:01:00.2' to '/sys/bus/pci/drivers/vfio-pci/bind': No such device
Nov 21 01:24:36 tvpve pvedaemon[73555]: Cannot bind 0000:01:00.2 to vfio
Nov 21 01:24:36 tvpve pvedaemon[1601]: <root@pam> end task UPID:tvpve:00011F53:00151E69:691FB144:qmstart:101:root@pam: Cannot bind 0000:01:00.2 to vfioI installed Ubuntu Server 25.10 on a second SSD to check if I have the same problem with it, but SR-IOV works with Ubuntu Server.
I've also tried it with this script, same result.
tutorial-proxmox-8-mellanox-infiniband-and-sr-iov
Here is some relevant information.
The motherboard BIOS and nic firmware are up to date and kernel version is 6.17.2-1-pve.
Configuration of the VM.
Code:
root@tvpve:~# cat /etc/pve/nodes/tvpve/qemu-server/101.conf
balloon: 0
bios: seabios
boot: order=scsi0
cores: 4
cpu: host,flags=+pdpe1gb
hostpci0: 0000:02:00.0,pcie=1
hostpci1: 0000:01:00.2
ide2: none,media=cdrom
machine: q35
memory: 12288
meta: creation-qemu=10.1.2,ctime=1763332613
name: SW-VSAN
numa: 0
ostype: l26
scsi0: local-zfs:vm-101-disk-1,iothread=1,size=30G
scsihw: virtio-scsi-single
smbios1: uuid=ab2723d5-d6fc-4d25-9513-8cf10c17c00c
sockets: 1
unused0: local-zfs:vm-101-disk-0
vmgenid: d401b045-f704-447c-a092-4101a7314f13Everything looks good to me
Code:
4: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether b8:3f:d2:c9:44:c2 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
altname enxb83fd2c944c2
5: enp1s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether b8:3f:d2:c9:44:c3 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
01:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
01:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
01:00.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
01:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
01:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
01:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
01:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
01:00.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
01:01.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
01:01.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
│ 0x020000 │ 0x101f │ 0000:01:00.0 │ 13 │ 0x15b3 │ MT2894 Family [ConnectX-6 Lx] │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101f │ 0000:01:00.1 │ 14 │ 0x15b3 │ MT2894 Family [ConnectX-6 Lx] │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101e │ 0000:01:00.2 │ 30 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101e │ 0000:01:00.3 │ 32 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101e │ 0000:01:00.4 │ 34 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101e │ 0000:01:00.5 │ 36 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
│ 0x020000 │ 0x101e │ 0000:01:00.6 │ 31 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101e │ 0000:01:00.7 │ 33 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101e │ 0000:01:01.0 │ 35 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
│ 0x020000 │ 0x101e │ 0000:01:01.1 │ 37 │ 0x15b3 │ ConnectX Family mlx5Gen Virtual Function │
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────┼─
Code:
root@tvpve:~# mstconfig -e -d 01:00.0 q
Device #1:
----------
Device type: ConnectX6LX
Name: MCX631102AS-ADA_Ax
Description: ConnectX-6 Lx EN adapter card; 25GbE; Dual-port SFP28; PCIe 4.0 x8; Secure Boot; No Crypto;
Device: 01:00.0
Configurations: Default Current Next Boot
LARGE_MTU_TWEAK_64 False(0) False(0) False(0)
AES_XTS_TWEAK_INC_64 False(0) False(0) False(0)
CRYPTO_POLICY UNRESTRICTED(1) UNRESTRICTED(1) UNRESTRICTED(1)
FLEX_PARSER_PROFILE_ENABLE 0 0 0
PROG_PARSE_GRAPH False(0) False(0) False(0)
FLEX_IPV4_OVER_VXLAN_PORT 0 0 0
SWITCH_COMPT_FEATURE_MASK 0x0(0) 0x0(0) 0x0(0)
PF_BAR2_SIZE 0 0 0
PF_NUM_OF_VF_VALID False(0) False(0) False(0)
NON_PREFETCHABLE_PF_BAR False(0) False(0) False(0)
VF_VPD_ENABLE False(0) False(0) False(0)
PF_NUM_PF_MSIX_VALID False(0) False(0) False(0)
PER_PF_NUM_SF False(0) False(0) False(0)
STRICT_VF_MSIX_NUM False(0) False(0) False(0)
VF_NODNIC_ENABLE False(0) False(0) False(0)
NUM_PF_MSIX_VALID True(1) True(1) True(1)
* NUM_OF_VFS 8 4 4
NUM_OF_PF 2 2 2
PF_BAR2_ENABLE False(0) False(0) False(0)
SRIOV_EN True(1) True(1) True(1)
Code:
root@tvpve:~# /usr/bin/mstflint -d 01:00.0 verify
FS4 failsafe image
/0x00000018-0x0000001f (0x000008)/ (HW_POINTERS) - OK
/0x00000020-0x00000027 (0x000008)/ (HW_POINTERS) - OK
/0x00000028-0x0000002f (0x000008)/ (HW_POINTERS) - OK
/0x00000030-0x00000037 (0x000008)/ (HW_POINTERS) - OK
/0x00000038-0x0000003f (0x000008)/ (HW_POINTERS) - OK
/0x00000040-0x00000047 (0x000008)/ (HW_POINTERS) - OK
/0x00000048-0x0000004f (0x000008)/ (HW_POINTERS) - OK
/0x00000050-0x00000057 (0x000008)/ (HW_POINTERS) - OK
/0x00000058-0x0000005f (0x000008)/ (HW_POINTERS) - OK
/0x00000060-0x00000067 (0x000008)/ (HW_POINTERS) - OK
/0x00000068-0x0000006f (0x000008)/ (HW_POINTERS) - OK
/0x00000070-0x00000077 (0x000008)/ (HW_POINTERS) - OK
/0x00000078-0x0000007f (0x000008)/ (HW_POINTERS) - OK
/0x00000080-0x00000087 (0x000008)/ (HW_POINTERS) - OK
/0x00000088-0x0000008f (0x000008)/ (HW_POINTERS) - OK
/0x00000090-0x00000097 (0x000008)/ (HW_POINTERS) - OK
/0x00000500-0x0000053f (0x000040)/ (TOOLS_AREA) - OK
/0x00001000-0x00004b5f (0x003b60)/ (BOOT2) - OK
/0x00005000-0x0000501f (0x000020)/ (ITOC_HEADER) - OK
/0x00007000-0x00024ad3 (0x01dad4)/ (IRON_PREP_CODE) - OK
/0x00024b00-0x00024bff (0x000100)/ (RESET_INFO) - OK
/0x00025300-0x0056bd7f (0x546a80)/ (MAIN_CODE) - OK
/0x0056bd80-0x0057e21b (0x01249c)/ (PCIE_LINK_CODE) - OK
/0x0057e280-0x0057edcf (0x000b50)/ (POST_IRON_BOOT_CODE) - OK
/0x0057ee00-0x005a393f (0x024b40)/ (PCI_CODE) - OK
/0x005a3980-0x005a76cb (0x003d4c)/ (UPGRADE_CODE) - OK
/0x005a7700-0x005d12ff (0x029c00)/ (ACE_CODE) - OK
/0x005d1300-0x005e9c0f (0x018910)/ (PHY_UC_CODE) - OK
/0x005e9c80-0x005fcedf (0x013260)/ (PCIE_PHY_UC_CODE) - OK
/0x005fcf00-0x005ff513 (0x002614)/ (CCIR_INFRA_CODE) - OK
/0x005ff580-0x006030a3 (0x003b24)/ (CCIR_ALGO_CODE) - OK
/0x00603100-0x006034ff (0x000400)/ (IMAGE_INFO) - OK
/0x00603500-0x00603fff (0x000b00)/ (FW_MAIN_CFG) - OK
/0x00604000-0x0060453f (0x000540)/ (FW_BOOT_CFG) - OK
/0x00604580-0x0060577f (0x001200)/ (HW_MAIN_CFG) - OK
/0x00605780-0x00605d7f (0x000600)/ (HW_BOOT_CFG) - OK
/0x00605d80-0x0060627f (0x000500)/ (PHY_UC_CONSTS) - OK
/0x00606280-0x006063bf (0x000140)/ (IMAGE_SIGNATURE_256) - CRC IGNORED
/0x00606400-0x00606cff (0x000900)/ (PUBLIC_KEYS_2048) - OK
/0x00606d00-0x00606d8f (0x000090)/ (FORBIDDEN_VERSIONS) - OK
/0x00606e00-0x0060703f (0x000240)/ (IMAGE_SIGNATURE_512) - CRC IGNORED
/0x00607080-0x0060817f (0x001100)/ (PUBLIC_KEYS_4096) - OK
/0x00608180-0x0060927f (0x001100)/ (RSA_PUBLIC_KEY) - OK
/0x00609280-0x0060987f (0x000600)/ (RSA_4096_SIGNATURES) - CRC IGNORED
/0x00609880-0x006cfaa7 (0x0c6228)/ (ROM_CODE) - OK
/0x006cfb00-0x006d036b (0x00086c)/ (DBG_FW_INI) - OK
/0x006d0380-0x006d0387 (0x000008)/ (DBG_FW_PARAMS) - OK
/0x006d0400-0x006e43ff (0x014000)/ (CRDUMP_MASK_DATA) - OK
-I- FW image verification succeeded. Image is bootable.
Last edited: