[SOLVED] Passthrough two PCI devices

Y

yarii

Renowned Member
Mar 24, 2014
145
8
83
I tried to passthrough PCI device alone:
1654174380071.png

I tried to passthrough another PCI device also alone:
1654174469663.png


Then I tried to Passthrough together two devices:
1654174494368.png



And I've got:
kvm: -device vfio-pci,host=0000:81:00.0,id=hostpci1.0,bus=pci.0,addr=0x11.0,multifunction=on: vfio 0000:81:00.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

What can I do about it?
 
hi,

have you tried blacklisting the kernel modules for those devices on your PVE host? it could be that your host is still loading the drivers for those and that's why you receive device busy.

you can check the used kernel modules like this: lspci -v -s 0000:88:00 (and same for the other one with 81:00).

after figuring out the kernel modules, you can blacklist them:
Code: 
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf # would blacklist 'radeon' module, change this with the one you find

and do a reboot.
 
@oguz did You read one mail infromation from my post about that VM is starting when only one device (no matter whitch) is set for passthrough?

In my /etc/modprobe.d/ I've got:
-rw-r--r-- 1 root root 16 Jun 1 11:36 ixgbe.conf
-rw-r--r-- 1 root root 20 Jun 1 11:36 mlx4_core.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_en.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_ib.conf
 
yarii said:
In my /etc/modprobe.d/ I've got:
-rw-r--r-- 1 root root 16 Jun 1 11:36 ixgbe.conf
-rw-r--r-- 1 root root 20 Jun 1 11:36 mlx4_core.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_en.conf
-rw-r--r-- 1 root root 18 Jun 1 11:36 mlx4_ib.conf
Click to expand...
do these blacklist the correct drivers? (just to be sure)

yarii said:
VM is starting when only one device (no matter whitch) is set for passthrough?
Click to expand...
is that the only VM where you do this passthrough?

can you post the configuration file qm config 900

also could you check the output of find /sys/kernel/iommu_groups/ -type l to verify iommu isolation?
 
do these blacklist the correct drivers? (just to be sure)
Click to expand...

Yes - I checked it twice. Name of module is like blacklist module inside conf file.

I attached my lspci for more info.

Most important thing:
1. Kernel driver in use: vfio-pci
Kernel modules: ixgbe

2. Kernel driver in use: vfio-pci
Kernel modules: mlx4_core

So that means that is working?

can you post the configuration file qm config 900
Click to expand...

Code: 
qm config 900
agent: 1
boot: order=scsi0
cores: 8
hostpci0: 0000:88:00
hostpci1: 0000:81:00
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=6.1.0,ctime=1639579480
name: rtx-bgp
net0: virtio=42:BE:A5:99:E4:24,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-zfs:vm-900-disk-0,size=8G
smbios1: uuid=4fe83f6d-ac32-4088-8ccc-da1906952d5b
sockets: 2
vmgenid: 35b3d286-aca9-4f21-96b5-83462e5a5e07

also could you check the output of find /sys/kernel/iommu_groups/ -type l to verify iommu isolation?
Click to expand...

Code: 
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.6
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.4
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.2
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.0
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.7
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.5
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.3
/sys/kernel/iommu_groups/108/devices/0000:ff:0c.1
/sys/kernel/iommu_groups/55/devices/0000:7f:14.0
/sys/kernel/iommu_groups/136/devices/0000:ff:1f.0
/sys/kernel/iommu_groups/136/devices/0000:ff:1f.2
/sys/kernel/iommu_groups/83/devices/0000:80:03.2
/sys/kernel/iommu_groups/17/devices/0000:00:04.6
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.6
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.4
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.2
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.0
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.7
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.5
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.3
/sys/kernel/iommu_groups/45/devices/0000:7f:0c.1
/sys/kernel/iommu_groups/126/devices/0000:ff:16.1
/sys/kernel/iommu_groups/73/devices/0000:7f:1f.2
/sys/kernel/iommu_groups/73/devices/0000:7f:1f.0
/sys/kernel/iommu_groups/35/devices/0000:02:00.2
/sys/kernel/iommu_groups/35/devices/0000:02:00.0
/sys/kernel/iommu_groups/35/devices/0000:02:00.3
/sys/kernel/iommu_groups/35/devices/0000:02:00.1
/sys/kernel/iommu_groups/7/devices/0000:00:03.0
/sys/kernel/iommu_groups/116/devices/0000:ff:13.3
/sys/kernel/iommu_groups/63/devices/0000:7f:16.1
/sys/kernel/iommu_groups/91/devices/0000:80:04.6
/sys/kernel/iommu_groups/25/devices/0000:00:14.0
/sys/kernel/iommu_groups/106/devices/0000:ff:09.6
/sys/kernel/iommu_groups/53/devices/0000:7f:13.3
/sys/kernel/iommu_groups/134/devices/0000:ff:17.7
/sys/kernel/iommu_groups/134/devices/0000:ff:17.5
/sys/kernel/iommu_groups/134/devices/0000:ff:17.6
/sys/kernel/iommu_groups/134/devices/0000:ff:17.4
/sys/kernel/iommu_groups/81/devices/0000:80:03.0
/sys/kernel/iommu_groups/15/devices/0000:00:04.4
/sys/kernel/iommu_groups/43/devices/0000:7f:09.6
/sys/kernel/iommu_groups/124/devices/0000:ff:15.1
/sys/kernel/iommu_groups/71/devices/0000:7f:17.7
/sys/kernel/iommu_groups/71/devices/0000:7f:17.5
/sys/kernel/iommu_groups/71/devices/0000:7f:17.6
/sys/kernel/iommu_groups/71/devices/0000:7f:17.4
/sys/kernel/iommu_groups/33/devices/0000:00:1f.0
/sys/kernel/iommu_groups/33/devices/0000:00:1f.3
/sys/kernel/iommu_groups/5/devices/0000:00:02.2
/sys/kernel/iommu_groups/114/devices/0000:ff:13.1
/sys/kernel/iommu_groups/61/devices/0000:7f:15.1
/sys/kernel/iommu_groups/23/devices/0000:00:05.6
/sys/kernel/iommu_groups/104/devices/0000:ff:09.2
/sys/kernel/iommu_groups/104/devices/0000:ff:09.0
/sys/kernel/iommu_groups/51/devices/0000:7f:13.1
/sys/kernel/iommu_groups/132/devices/0000:ff:17.2
/sys/kernel/iommu_groups/13/devices/0000:00:04.2
/sys/kernel/iommu_groups/41/devices/0000:7f:09.2
/sys/kernel/iommu_groups/41/devices/0000:7f:09.0
/sys/kernel/iommu_groups/122/devices/0000:ff:14.7
/sys/kernel/iommu_groups/122/devices/0000:ff:14.5
/sys/kernel/iommu_groups/122/devices/0000:ff:14.6
/sys/kernel/iommu_groups/122/devices/0000:ff:14.4
/sys/kernel/iommu_groups/98/devices/0000:81:00.0
/sys/kernel/iommu_groups/31/devices/0000:00:1c.7
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/112/devices/0000:ff:12.6
/sys/kernel/iommu_groups/112/devices/0000:ff:12.4
/sys/kernel/iommu_groups/112/devices/0000:ff:12.2
/sys/kernel/iommu_groups/112/devices/0000:ff:12.0
/sys/kernel/iommu_groups/112/devices/0000:ff:12.5
/sys/kernel/iommu_groups/112/devices/0000:ff:12.1
/sys/kernel/iommu_groups/88/devices/0000:80:04.3
/sys/kernel/iommu_groups/21/devices/0000:00:05.2
/sys/kernel/iommu_groups/102/devices/0000:ff:08.3
/sys/kernel/iommu_groups/130/devices/0000:ff:17.0
/sys/kernel/iommu_groups/78/devices/0000:80:02.1
/sys/kernel/iommu_groups/11/devices/0000:00:04.0
/sys/kernel/iommu_groups/120/devices/0000:ff:14.2
/sys/kernel/iommu_groups/68/devices/0000:7f:17.1
/sys/kernel/iommu_groups/96/devices/0000:80:05.4
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.6
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.4
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.2
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.0
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.5
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.3
/sys/kernel/iommu_groups/110/devices/0000:ff:0f.1
/sys/kernel/iommu_groups/58/devices/0000:7f:14.3
/sys/kernel/iommu_groups/86/devices/0000:80:04.1
/sys/kernel/iommu_groups/100/devices/0000:88:00.0
/sys/kernel/iommu_groups/48/devices/0000:7f:10.0
/sys/kernel/iommu_groups/48/devices/0000:7f:10.7
/sys/kernel/iommu_groups/48/devices/0000:7f:10.5
/sys/kernel/iommu_groups/48/devices/0000:7f:10.1
/sys/kernel/iommu_groups/48/devices/0000:7f:10.6
/sys/kernel/iommu_groups/129/devices/0000:ff:16.7
/sys/kernel/iommu_groups/129/devices/0000:ff:16.6
/sys/kernel/iommu_groups/76/devices/0000:80:01.1
/sys/kernel/iommu_groups/38/devices/0000:7f:08.0
/sys/kernel/iommu_groups/38/devices/0000:7f:08.2
/sys/kernel/iommu_groups/119/devices/0000:ff:14.1
/sys/kernel/iommu_groups/66/devices/0000:7f:16.6
/sys/kernel/iommu_groups/66/devices/0000:7f:16.7
/sys/kernel/iommu_groups/94/devices/0000:80:05.1
/sys/kernel/iommu_groups/28/devices/0000:00:1c.2
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.5
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.3
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.1
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.4
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.2
/sys/kernel/iommu_groups/109/devices/0000:ff:0d.0
/sys/kernel/iommu_groups/56/devices/0000:7f:14.1
/sys/kernel/iommu_groups/84/devices/0000:80:03.3
/sys/kernel/iommu_groups/18/devices/0000:00:04.7
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.0
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.5
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.3
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.1
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.4
/sys/kernel/iommu_groups/46/devices/0000:7f:0d.2
/sys/kernel/iommu_groups/127/devices/0000:ff:16.2
/sys/kernel/iommu_groups/74/devices/0000:80:00.0
/sys/kernel/iommu_groups/36/devices/0000:03:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:03.1
/sys/kernel/iommu_groups/117/devices/0000:ff:13.7
/sys/kernel/iommu_groups/117/devices/0000:ff:13.6
/sys/kernel/iommu_groups/64/devices/0000:7f:16.2
/sys/kernel/iommu_groups/92/devices/0000:80:04.7
/sys/kernel/iommu_groups/26/devices/0000:00:1a.0
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.1
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.2
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.0
/sys/kernel/iommu_groups/107/devices/0000:ff:0b.3
/sys/kernel/iommu_groups/54/devices/0000:7f:13.7
/sys/kernel/iommu_groups/54/devices/0000:7f:13.6
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.4
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.2
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.0
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.3
/sys/kernel/iommu_groups/135/devices/0000:ff:1e.1
/sys/kernel/iommu_groups/82/devices/0000:80:03.1
/sys/kernel/iommu_groups/16/devices/0000:00:04.5
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.3
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.1
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.2
/sys/kernel/iommu_groups/44/devices/0000:7f:0b.0
/sys/kernel/iommu_groups/125/devices/0000:ff:16.0
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.4
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.2
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.0
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.3
/sys/kernel/iommu_groups/72/devices/0000:7f:1e.1
/sys/kernel/iommu_groups/34/devices/0000:01:00.4
/sys/kernel/iommu_groups/34/devices/0000:01:00.2
/sys/kernel/iommu_groups/34/devices/0000:01:00.0
/sys/kernel/iommu_groups/34/devices/0000:01:00.1
/sys/kernel/iommu_groups/6/devices/0000:00:02.3
/sys/kernel/iommu_groups/115/devices/0000:ff:13.2
/sys/kernel/iommu_groups/62/devices/0000:7f:16.0
/sys/kernel/iommu_groups/90/devices/0000:80:04.5
/sys/kernel/iommu_groups/24/devices/0000:00:11.0
/sys/kernel/iommu_groups/105/devices/0000:ff:09.3
/sys/kernel/iommu_groups/52/devices/0000:7f:13.2
/sys/kernel/iommu_groups/133/devices/0000:ff:17.3
/sys/kernel/iommu_groups/80/devices/0000:80:02.3
/sys/kernel/iommu_groups/14/devices/0000:00:04.3
/sys/kernel/iommu_groups/42/devices/0000:7f:09.3
/sys/kernel/iommu_groups/123/devices/0000:ff:15.0
/sys/kernel/iommu_groups/70/devices/0000:7f:17.3
/sys/kernel/iommu_groups/99/devices/0000:81:00.1
/sys/kernel/iommu_groups/32/devices/0000:00:1d.0
/sys/kernel/iommu_groups/4/devices/0000:00:02.1
/sys/kernel/iommu_groups/113/devices/0000:ff:13.0
/sys/kernel/iommu_groups/60/devices/0000:7f:15.0
/sys/kernel/iommu_groups/89/devices/0000:80:04.4
/sys/kernel/iommu_groups/22/devices/0000:00:05.4
/sys/kernel/iommu_groups/103/devices/0000:ff:08.6
/sys/kernel/iommu_groups/50/devices/0000:7f:13.0
/sys/kernel/iommu_groups/131/devices/0000:ff:17.1
/sys/kernel/iommu_groups/79/devices/0000:80:02.2
/sys/kernel/iommu_groups/12/devices/0000:00:04.1
/sys/kernel/iommu_groups/40/devices/0000:7f:08.6
/sys/kernel/iommu_groups/121/devices/0000:ff:14.3
/sys/kernel/iommu_groups/69/devices/0000:7f:17.2
/sys/kernel/iommu_groups/97/devices/0000:80:05.6
/sys/kernel/iommu_groups/30/devices/0000:00:1c.6
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/111/devices/0000:ff:10.0
/sys/kernel/iommu_groups/111/devices/0000:ff:10.7
/sys/kernel/iommu_groups/111/devices/0000:ff:10.5
/sys/kernel/iommu_groups/111/devices/0000:ff:10.1
/sys/kernel/iommu_groups/111/devices/0000:ff:10.6
/sys/kernel/iommu_groups/59/devices/0000:7f:14.7
/sys/kernel/iommu_groups/59/devices/0000:7f:14.5
/sys/kernel/iommu_groups/59/devices/0000:7f:14.6
/sys/kernel/iommu_groups/59/devices/0000:7f:14.4
/sys/kernel/iommu_groups/87/devices/0000:80:04.2
/sys/kernel/iommu_groups/20/devices/0000:00:05.1
/sys/kernel/iommu_groups/101/devices/0000:ff:08.2
/sys/kernel/iommu_groups/101/devices/0000:ff:08.0
/sys/kernel/iommu_groups/49/devices/0000:7f:12.1
/sys/kernel/iommu_groups/49/devices/0000:7f:12.6
/sys/kernel/iommu_groups/49/devices/0000:7f:12.4
/sys/kernel/iommu_groups/49/devices/0000:7f:12.2
/sys/kernel/iommu_groups/49/devices/0000:7f:12.0
/sys/kernel/iommu_groups/49/devices/0000:7f:12.5
/sys/kernel/iommu_groups/77/devices/0000:80:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:03.3
/sys/kernel/iommu_groups/39/devices/0000:7f:08.3
/sys/kernel/iommu_groups/67/devices/0000:7f:17.0
/sys/kernel/iommu_groups/95/devices/0000:80:05.2
/sys/kernel/iommu_groups/29/devices/0000:00:1c.4
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/57/devices/0000:7f:14.2
/sys/kernel/iommu_groups/85/devices/0000:80:04.0
/sys/kernel/iommu_groups/19/devices/0000:00:05.0
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.6
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.4
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.2
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.0
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.5
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.3
/sys/kernel/iommu_groups/47/devices/0000:7f:0f.1
/sys/kernel/iommu_groups/128/devices/0000:ff:16.3
/sys/kernel/iommu_groups/75/devices/0000:80:01.0
/sys/kernel/iommu_groups/37/devices/0000:08:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:03.2
/sys/kernel/iommu_groups/118/devices/0000:ff:14.0
/sys/kernel/iommu_groups/65/devices/0000:7f:16.3
/sys/kernel/iommu_groups/93/devices/0000:80:05.0
/sys/kernel/iommu_groups/27/devices/0000:00:1c.0
 

Attachments

yarii said:
Most important thing:
1. Kernel driver in use: vfio-pci
Kernel modules: ixgbe

2. Kernel driver in use: vfio-pci
Kernel modules: mlx4_core

So that means that is working?
Click to expand...
yes should be fine, as you can already passthrough them one by one.

your IOMMU groups also look fine (the cards are in different groups) so i'm not sure what it could be...
have you tried with different physical slots as well?

do you have any BIOS updates available for your motherboard?
 
The other thing I tried is add another virtual machine and passthrough second device at the same time.

It doesn't worked.

Do this matter if the devices got the same IRQ's:

88:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3] Physical Slot: 4 IOMMU group: 100 Interrupt: pin A routed to IRQ 16


08:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3] Physical Slot: 3 IOMMU group: 37 Interrupt: pin A routed to IRQ 16

?

Is it possible to change IRQ's for them?
 
When I am trying to start VM with passtrough two network devices there are errors in DMESG:

Code: 
root@rtx-proxmox:~# dmesg | grep 'Flags mismatch'
[   99.938242] genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:81:00.0)) vs. 00000000 (vfio-intx(0000:08:00.0))
[  302.258175] genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:81:00.0)) vs. 00000000 (vfio-intx(0000:08:00.0))
[  383.502167] genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:81:00.0)) vs. 00000000 (vfio-intx(0000:08:00.0))
 
I tried to juggle cards with different PCI-E - it didn't help.

When I turn on second network device (no matter whitch) it gives me an error:
genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:88:00.0)) vs. 00000000 (vfio-intx(0000:05:00.0))

It looks like some race condition bug when I trying to use two network devices in VM with passthrough.
 
This issue is only valid when there is a 2x Mellanox ConnectX-3 cards.
I changed the cards to ConnectX-4 - all works like a charm.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom