[SOLVED] Passthrough two PCI devices

[yarii]

I tried to passthrough PCI device alone:


I tried to passthrough another PCI device also alone:



Then I tried to Passthrough together two devices:




And I've got:
kvm: -device vfio-pci,host=0000:81:00.0,id=hostpci1.0,bus=pci.0,addr=0x11.0,multifunction=on: vfio 0000:81:00.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

What can I do about it?

 

[oguz]

hi,

have you tried blacklisting the kernel modules for those devices on your PVE host? it could be that your host is still loading the drivers for those and that's why you receive device busy.

you can check the used kernel modules like this: lspci -v -s 0000:88:00 (and same for the other one with 81:00).

after figuring out the kernel modules, you can blacklist them:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf # would blacklist 'radeon' module, change this with the one you find

and do a reboot.

 

[yarii]

@oguz did You read one mail infromation from my post about that VM is starting when only one device (no matter whitch) is set for passthrough?

In my /etc/modprobe.d/ I've got:
-rw-r--r-- 1 root root 16 Jun 1 11:36 ixgbe.conf
-rw-r--r-- 1 root root 20 Jun 1 11:36 mlx4_core.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_en.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_ib.conf

 

[oguz]

In my /etc/modprobe.d/ I've got:
-rw-r--r-- 1 root root 16 Jun 1 11:36 ixgbe.conf
-rw-r--r-- 1 root root 20 Jun 1 11:36 mlx4_core.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_en.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_ib.conf

do these blacklist the correct drivers? (just to be sure)

VM is starting when only one device (no matter whitch) is set for passthrough?

is that the only VM where you do this passthrough?

can you post the configuration file qm config 900

also could you check the output of find /sys/kernel/iommu_groups/ -type l to verify iommu isolation?

 

[yarii]

do these blacklist the correct drivers? (just to be sure)

Yes - I checked it twice. Name of module is like blacklist module inside conf file.

I attached my lspci for more info.

Most important thing:
1. Kernel driver in use: vfio-pci
Kernel modules: ixgbe

2. Kernel driver in use: vfio-pci
Kernel modules: mlx4_core

So that means that is working?

can you post the configuration file qm config 900

qm config 900
agent: 1
boot: order=scsi0
cores: 8
hostpci0: 0000:88:00
hostpci1: 0000:81:00
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=6.1.0,ctime=1639579480
name: rtx-bgp
net0: virtio=42:BE:A5:99:E4:24,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-zfs:vm-900-disk-0,size=8G
smbios1: uuid=4fe83f6d-ac32-4088-8ccc-da1906952d5b
sockets: 2
vmgenid: 35b3d286-aca9-4f21-96b5-83462e5a5e07

also could you check the output of find /sys/kernel/iommu_groups/ -type l to verify iommu isolation?

/sys/kernel/iommu_groups/108/devices/0000:ff:0c.6
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.4
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.2
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.0
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.7
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.5
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.3
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.1
/sys/kernel/iommu_groups/55/devices/0000:7f:14.0
/sys/kernel/iommu_groups/136/devices/0000:ff:1f.0
/sys/kernel/iommu_groups/136/devices/0000:ff:1f.2
/sys/kernel/iommu_groups/83/devices/0000:80:03.2
/sys/kernel/iommu_groups/17/devices/0000:00:04.6
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.6
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.4
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.2
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.0
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.7
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.5
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.3
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.1
/sys/kernel/iommu_groups/126/devices/0000:ff:16.1
/sys/kernel/iommu_groups/73/devices/0000:7f:1f.2
/sys/kernel/iommu_groups/73/devices/0000:7f:1f.0
/sys/kernel/iommu_groups/35/devices/0000:02:00.2
/sys/kernel/iommu_groups/35/devices/0000:02:00.0
/sys/kernel/iommu_groups/35/devices/0000:02:00.3
/sys/kernel/iommu_groups/35/devices/0000:02:00.1
/sys/kernel/iommu_groups/7/devices/0000:00:03.0
/sys/kernel/iommu_groups/116/devices/0000:ff:13.3
/sys/kernel/iommu_groups/63/devices/0000:7f:16.1
/sys/kernel/iommu_groups/91/devices/0000:80:04.6
/sys/kernel/iommu_groups/25/devices/0000:00:14.0
/sys/kernel/iommu_groups/106/devices/0000:ff:09.6
/sys/kernel/iommu_groups/53/devices/0000:7f:13.3
/sys/kernel/iommu_groups/134/devices/0000:ff:17.7
/sys/kernel/iommu_groups/134/devices/0000:ff:17.5
/sys/kernel/iommu_groups/134/devices/0000:ff:17.6
/sys/kernel/iommu_groups/134/devices/0000:ff:17.4
/sys/kernel/iommu_groups/81/devices/0000:80:03.0
/sys/kernel/iommu_groups/15/devices/0000:00:04.4
/sys/kernel/iommu_groups/43/devices/0000:7f:09.6
/sys/kernel/iommu_groups/124/devices/0000:ff:15.1
/sys/kernel/iommu_groups/71/devices/0000:7f:17.7
/sys/kernel/iommu_groups/71/devices/0000:7f:17.5
/sys/kernel/iommu_groups/71/devices/0000:7f:17.6
/sys/kernel/iommu_groups/71/devices/0000:7f:17.4
/sys/kernel/iommu_groups/33/devices/0000:00:1f.0
/sys/kernel/iommu_groups/33/devices/0000:00:1f.3
/sys/kernel/iommu_groups/5/devices/0000:00:02.2
/sys/kernel/iommu_groups/114/devices/0000:ff:13.1
/sys/kernel/iommu_groups/61/devices/0000:7f:15.1
/sys/kernel/iommu_groups/23/devices/0000:00:05.6
/sys/kernel/iommu_groups/104/devices/0000:ff:09.2
/sys/kernel/iommu_groups/104/devices/0000:ff:09.0
/sys/kernel/iommu_groups/51/devices/0000:7f:13.1
/sys/kernel/iommu_groups/132/devices/0000:ff:17.2
/sys/kernel/iommu_groups/13/devices/0000:00:04.2
/sys/kernel/iommu_groups/41/devices/0000:7f:09.2
/sys/kernel/iommu_groups/41/devices/0000:7f:09.0
/sys/kernel/iommu_groups/122/devices/0000:ff:14.7
/sys/kernel/iommu_groups/122/devices/0000:ff:14.5
/sys/kernel/iommu_groups/122/devices/0000:ff:14.6
/sys/kernel/iommu_groups/122/devices/0000:ff:14.4
/sys/kernel/iommu_groups/98/devices/0000:81:00.0
/sys/kernel/iommu_groups/31/devices/0000:00:1c.7
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/112/devices/0000:ff:12.6
/sys/kernel/iommu_groups/112/devices/0000:ff:12.4
/sys/kernel/iommu_groups/112/devices/0000:ff:12.2
/sys/kernel/iommu_groups/112/devices/0000:ff:12.0
/sys/kernel/iommu_groups/112/devices/0000:ff:12.5
/sys/kernel/iommu_groups/112/devices/0000:ff:12.1
/sys/kernel/iommu_groups/88/devices/0000:80:04.3
/sys/kernel/iommu_groups/21/devices/0000:00:05.2
/sys/kernel/iommu_groups/102/devices/0000:ff:08.3
/sys/kernel/iommu_groups/130/devices/0000:ff:17.0
/sys/kernel/iommu_groups/78/devices/0000:80:02.1
/sys/kernel/iommu_groups/11/devices/0000:00:04.0
/sys/kernel/iommu_groups/120/devices/0000:ff:14.2
/sys/kernel/iommu_groups/68/devices/0000:7f:17.1
/sys/kernel/iommu_groups/96/devices/0000:80:05.4
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.6
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.4
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.2
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.0
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.5
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.3
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.1
/sys/kernel/iommu_groups/58/devices/0000:7f:14.3
/sys/kernel/iommu_groups/86/devices/0000:80:04.1
/sys/kernel/iommu_groups/100/devices/0000:88:00.0
/sys/kernel/iommu_groups/48/devices/0000:7f:10.0
/sys/kernel/iommu_groups/48/devices/0000:7f:10.7
/sys/kernel/iommu_groups/48/devices/0000:7f:10.5
/sys/kernel/iommu_groups/48/devices/0000:7f:10.1
/sys/kernel/iommu_groups/48/devices/0000:7f:10.6
/sys/kernel/iommu_groups/129/devices/0000:ff:16.7
/sys/kernel/iommu_groups/129/devices/0000:ff:16.6
/sys/kernel/iommu_groups/76/devices/0000:80:01.1
/sys/kernel/iommu_groups/38/devices/0000:7f:08.0
/sys/kernel/iommu_groups/38/devices/0000:7f:08.2
/sys/kernel/iommu_groups/119/devices/0000:ff:14.1
/sys/kernel/iommu_groups/66/devices/0000:7f:16.6
/sys/kernel/iommu_groups/66/devices/0000:7f:16.7
/sys/kernel/iommu_groups/94/devices/0000:80:05.1
/sys/kernel/iommu_groups/28/devices/0000:00:1c.2
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.5
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.3
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.1
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.4
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.2
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.0
/sys/kernel/iommu_groups/56/devices/0000:7f:14.1
/sys/kernel/iommu_groups/84/devices/0000:80:03.3
/sys/kernel/iommu_groups/18/devices/0000:00:04.7
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.0
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.5
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.3
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.1
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.4
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.2
/sys/kernel/iommu_groups/127/devices/0000:ff:16.2
/sys/kernel/iommu_groups/74/devices/0000:80:00.0
/sys/kernel/iommu_groups/36/devices/0000:03:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:03.1
/sys/kernel/iommu_groups/117/devices/0000:ff:13.7
/sys/kernel/iommu_groups/117/devices/0000:ff:13.6
/sys/kernel/iommu_groups/64/devices/0000:7f:16.2
/sys/kernel/iommu_groups/92/devices/0000:80:04.7
/sys/kernel/iommu_groups/26/devices/0000:00:1a.0
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.1
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.2
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.0
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.3
/sys/kernel/iommu_groups/54/devices/0000:7f:13.7
/sys/kernel/iommu_groups/54/devices/0000:7f:13.6
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.4
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.2
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.0
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.3
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.1
/sys/kernel/iommu_groups/82/devices/0000:80:03.1
/sys/kernel/iommu_groups/16/devices/0000:00:04.5
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.3
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.1
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.2
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.0
/sys/kernel/iommu_groups/125/devices/0000:ff:16.0
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.4
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.2
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.0
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.3
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.1
/sys/kernel/iommu_groups/34/devices/0000:01:00.4
/sys/kernel/iommu_groups/34/devices/0000:01:00.2
/sys/kernel/iommu_groups/34/devices/0000:01:00.0
/sys/kernel/iommu_groups/34/devices/0000:01:00.1
/sys/kernel/iommu_groups/6/devices/0000:00:02.3
/sys/kernel/iommu_groups/115/devices/0000:ff:13.2
/sys/kernel/iommu_groups/62/devices/0000:7f:16.0
/sys/kernel/iommu_groups/90/devices/0000:80:04.5
/sys/kernel/iommu_groups/24/devices/0000:00:11.0
/sys/kernel/iommu_groups/105/devices/0000:ff:09.3
/sys/kernel/iommu_groups/52/devices/0000:7f:13.2
/sys/kernel/iommu_groups/133/devices/0000:ff:17.3
/sys/kernel/iommu_groups/80/devices/0000:80:02.3
/sys/kernel/iommu_groups/14/devices/0000:00:04.3
/sys/kernel/iommu_groups/42/devices/0000:7f:09.3
/sys/kernel/iommu_groups/123/devices/0000:ff:15.0
/sys/kernel/iommu_groups/70/devices/0000:7f:17.3
/sys/kernel/iommu_groups/99/devices/0000:81:00.1
/sys/kernel/iommu_groups/32/devices/0000:00:1d.0
/sys/kernel/iommu_groups/4/devices/0000:00:02.1
/sys/kernel/iommu_groups/113/devices/0000:ff:13.0
/sys/kernel/iommu_groups/60/devices/0000:7f:15.0
/sys/kernel/iommu_groups/89/devices/0000:80:04.4
/sys/kernel/iommu_groups/22/devices/0000:00:05.4
/sys/kernel/iommu_groups/103/devices/0000:ff:08.6
/sys/kernel/iommu_groups/50/devices/0000:7f:13.0
/sys/kernel/iommu_groups/131/devices/0000:ff:17.1
/sys/kernel/iommu_groups/79/devices/0000:80:02.2
/sys/kernel/iommu_groups/12/devices/0000:00:04.1
/sys/kernel/iommu_groups/40/devices/0000:7f:08.6
/sys/kernel/iommu_groups/121/devices/0000:ff:14.3
/sys/kernel/iommu_groups/69/devices/0000:7f:17.2
/sys/kernel/iommu_groups/97/devices/0000:80:05.6
/sys/kernel/iommu_groups/30/devices/0000:00:1c.6
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/111/devices/0000:ff:10.0
/sys/kernel/iommu_groups/111/devices/0000:ff:10.7
/sys/kernel/iommu_groups/111/devices/0000:ff:10.5
/sys/kernel/iommu_groups/111/devices/0000:ff:10.1
/sys/kernel/iommu_groups/111/devices/0000:ff:10.6
/sys/kernel/iommu_groups/59/devices/0000:7f:14.7
/sys/kernel/iommu_groups/59/devices/0000:7f:14.5
/sys/kernel/iommu_groups/59/devices/0000:7f:14.6
/sys/kernel/iommu_groups/59/devices/0000:7f:14.4
/sys/kernel/iommu_groups/87/devices/0000:80:04.2
/sys/kernel/iommu_groups/20/devices/0000:00:05.1
/sys/kernel/iommu_groups/101/devices/0000:ff:08.2
/sys/kernel/iommu_groups/101/devices/0000:ff:08.0
/sys/kernel/iommu_groups/49/devices/0000:7f:12.1
/sys/kernel/iommu_groups/49/devices/0000:7f:12.6
/sys/kernel/iommu_groups/49/devices/0000:7f:12.4
/sys/kernel/iommu_groups/49/devices/0000:7f:12.2
/sys/kernel/iommu_groups/49/devices/0000:7f:12.0
/sys/kernel/iommu_groups/49/devices/0000:7f:12.5
/sys/kernel/iommu_groups/77/devices/0000:80:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:03.3
/sys/kernel/iommu_groups/39/devices/0000:7f:08.3
/sys/kernel/iommu_groups/67/devices/0000:7f:17.0
/sys/kernel/iommu_groups/95/devices/0000:80:05.2
/sys/kernel/iommu_groups/29/devices/0000:00:1c.4
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/57/devices/0000:7f:14.2
/sys/kernel/iommu_groups/85/devices/0000:80:04.0
/sys/kernel/iommu_groups/19/devices/0000:00:05.0
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.6
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.4
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.2
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.0
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.5
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.3
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.1
/sys/kernel/iommu_groups/128/devices/0000:ff:16.3
/sys/kernel/iommu_groups/75/devices/0000:80:01.0
/sys/kernel/iommu_groups/37/devices/0000:08:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:03.2
/sys/kernel/iommu_groups/118/devices/0000:ff:14.0
/sys/kernel/iommu_groups/65/devices/0000:7f:16.3
/sys/kernel/iommu_groups/93/devices/0000:80:05.0
/sys/kernel/iommu_groups/27/devices/0000:00:1c.0

 

[oguz]

Most important thing:
1. Kernel driver in use: vfio-pci
Kernel modules: ixgbe

2. Kernel driver in use: vfio-pci
Kernel modules: mlx4_core

So that means that is working?

yes should be fine, as you can already passthrough them one by one.

your IOMMU groups also look fine (the cards are in different groups) so i'm not sure what it could be...
have you tried with different physical slots as well?

do you have any BIOS updates available for your motherboard?

 

[yarii]

have you tried with different physical slots as well?

No - I will try.

do you have any BIOS updates available for your motherboard?

No. All of the available updates are applied.

 

[yarii]

The other thing I tried is add another virtual machine and passthrough second device at the same time.

It doesn't worked.

Do this matter if the devices got the same IRQ's:

88:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3] Physical Slot: 4 IOMMU group: 100 Interrupt: pin A routed to IRQ 16


08:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3] Physical Slot: 3 IOMMU group: 37 Interrupt: pin A routed to IRQ 16

?

Is it possible to change IRQ's for them?

 

[yarii]

When I am trying to start VM with passtrough two network devices there are errors in DMESG:

root@rtx-proxmox:~# dmesg | grep 'Flags mismatch'
[   99.938242] genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:81:00.0)) vs. 00000000 (vfio-intx(0000:08:00.0))
[  302.258175] genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:81:00.0)) vs. 00000000 (vfio-intx(0000:08:00.0))
[  383.502167] genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:81:00.0)) vs. 00000000 (vfio-intx(0000:08:00.0))

 

[yarii]

I tried to juggle cards with different PCI-E - it didn't help.

When I turn on second network device (no matter whitch) it gives me an error:
genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:88:00.0)) vs. 00000000 (vfio-intx(0000:05:00.0))

It looks like some race condition bug when I trying to use two network devices in VM with passthrough.

 

[yarii]

This issue is only valid when there is a 2x Mellanox ConnectX-3 cards.
I changed the cards to ConnectX-4 - all works like a charm.