Hi,
I have ubuntu 22.04 guest os where i am trying to enable use dpdk for mellanox network cards on guest os.
for achieving it i enabled pci passthrough for the mellanox cards, and they are visible on guest os. however when i try to load vfio-pci driver for these cards im getting an error -22, which chatgpt tells that its regarding iommu error. but i guess i've done everything for iommu to work properly.
im confused, any help is appreciated.
thanks
Proxmox OS:
cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done | grep -i mell
BOOT_IMAGE=/boot/vmlinuz-6.8.12-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
IOMMU group 3 81:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 4 81:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 54 02:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 55 02:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 5 82:00.0 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU group 6 82:00.1 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU group 7 83:00.0 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU group 8 83:00.1 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
Guest OS:
sudo ./dpdk-devbind.py --bind=vfio-pci 0000:00:10.0
Error: bind failed for 0000:00:10.0 - Cannot bind to driver vfio-pci: [Errno 22] Invalid argument
./dpdk-devbind.py -s
Network devices using kernel driver
===================================
0000:00:12.0 'Virtio network device 1000' if=ens18 drv=virtio-pci unused=virtio_pci,vfio-pci,uio_pci_generic *Active*
0000:00:1b.0 'MT27800 Family [ConnectX-5] 1017' if=ens27np0 drv=mlx5_core unused=vfio-pci,uio_pci_generic
0000:00:1c.0 'MT27800 Family [ConnectX-5] 1017' if=ens28np1 drv=mlx5_core unused=vfio-pci,uio_pci_generic
Other Network devices
=====================
0000:00:10.0 'MT27800 Family [ConnectX-5] 1017' unused=mlx5_core,vfio-pci,uio_pci_generic
0000:00:11.0 'MT27800 Family [ConnectX-5] 1017' unused=mlx5_core,vfio-pci,uio_pci_generic
dmesg | grep -i vfio
[ 288.917804] VFIO - User Level meta-driver version: 0.3
[ 293.746867] vfio-pci: probe of 0000:00:10.0 failed with error -22
[ 293.746967] vfio-pci: probe of 0000:00:10.0 failed with error -22
[ 343.506982] vfio-pci: probe of 0000:00:11.0 failed with error -22
I have ubuntu 22.04 guest os where i am trying to enable use dpdk for mellanox network cards on guest os.
for achieving it i enabled pci passthrough for the mellanox cards, and they are visible on guest os. however when i try to load vfio-pci driver for these cards im getting an error -22, which chatgpt tells that its regarding iommu error. but i guess i've done everything for iommu to work properly.
im confused, any help is appreciated.
thanks
Proxmox OS:
cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done | grep -i mell
BOOT_IMAGE=/boot/vmlinuz-6.8.12-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt
IOMMU group 3 81:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 4 81:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 54 02:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 55 02:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
IOMMU group 5 82:00.0 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU group 6 82:00.1 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU group 7 83:00.0 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
IOMMU group 8 83:00.1 Ethernet controller [0200]: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [15b3:1019]
Guest OS:
sudo ./dpdk-devbind.py --bind=vfio-pci 0000:00:10.0
Error: bind failed for 0000:00:10.0 - Cannot bind to driver vfio-pci: [Errno 22] Invalid argument
./dpdk-devbind.py -s
Network devices using kernel driver
===================================
0000:00:12.0 'Virtio network device 1000' if=ens18 drv=virtio-pci unused=virtio_pci,vfio-pci,uio_pci_generic *Active*
0000:00:1b.0 'MT27800 Family [ConnectX-5] 1017' if=ens27np0 drv=mlx5_core unused=vfio-pci,uio_pci_generic
0000:00:1c.0 'MT27800 Family [ConnectX-5] 1017' if=ens28np1 drv=mlx5_core unused=vfio-pci,uio_pci_generic
Other Network devices
=====================
0000:00:10.0 'MT27800 Family [ConnectX-5] 1017' unused=mlx5_core,vfio-pci,uio_pci_generic
0000:00:11.0 'MT27800 Family [ConnectX-5] 1017' unused=mlx5_core,vfio-pci,uio_pci_generic
dmesg | grep -i vfio
[ 288.917804] VFIO - User Level meta-driver version: 0.3
[ 293.746867] vfio-pci: probe of 0000:00:10.0 failed with error -22
[ 293.746967] vfio-pci: probe of 0000:00:10.0 failed with error -22
[ 343.506982] vfio-pci: probe of 0000:00:11.0 failed with error -22