PCIE devices blocking each other

[nicolaeser]

Hello Guys,

I have a Gigabyte G292 Z20 with two GPUs plugged in a Tray. Now I try the PCIE Passthrough to two vms.
And no: Not the same gpu!
This is selected on the one VM:


And this is selected on the second vm:


I selected: "raw device", all functions, raw bars and pcie passthrough.

When I try to have both VMs on at the same time i get the error message:
"
generating cloud-init ISO
kvm: -device vfio-pci,host=0000:04:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: vfio 0000:04:00.0: Could not open '/dev/vfio/45': Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1
"

what can i do about that?

Thanks for your help!

 

[Impact]

They share the same IOMMU group. Read about IOMMU/ACS here and here. Also see here and try different PCI(e) slots.

 

[nicolaeser]

They share the same IOMMU group. Read about IOMMU/ACS here and here. Also see here and try different PCI(e) slots.

But why, all PCIE slots are there to use. Could removing the drivers / blacking nouvea on the Host of PVE fix / solve this problem?

 

[Impact]

Please don't quote the whole message that's directly above yours already. I cannot tell you why your mainboard decides to do that.
AFAIK this is not related to drivers. See if your mainbiooard's UEFI has IOMMU/ACS options you can enable or try a different slot.

 

[leesteken]

It's not specific to Proxmox (or even Linux): https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html

 

[nicolaeser]

Status Update: The BIOS doesn't seem to have ACS! The team of gigabyte is looking into this and maybe preparing some custom BIOS for that, since the 2019 Docs of the Server cover that feature but my 2021 bios doesnt.

 

[nicolaeser]

Status Update: Gigabyte responded and told me, that the ACS is a little bit hidden. Now with AER and then ACS it works perfectly.