2 GPUs, 2 VMs not working as intended

[RockisLife]

Hello Proxmox Forums,
I am trying to pass through 2 GPUs to 2 VMs. I have gone through the process of enabling and setting up PCIe Pass through. It works for 1 VM. But when I go to start VM number 2 with the other graphics card set, I get this error

kvm: -device vfio-pci,host=0000:02:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,multifunction=on: vfio 0000:02:00.0: failed to open /dev/vfio/1: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

Could it be a PCIe Lane issue?
I have two devices at 0000:01:00 and 0000:02:00
If any further information is required I am happy to provide. Any guidance is greatly appreciated
Thank you

 

[leesteken]

Check your IOMMU groups; you cannot share devices in the same group between VMs and/or the Proxmox host (securely).
EDIT:

How Do I separate them in IOMMU Groups. They both are in the same group

Well, that's your problem right there...

 

[RockisLife]

Check your IOMMU groups; you cannot share devices in the same group between VMs and/or the Proxmox host (securely).

How Do I separate them in IOMMU Groups. They both are in the same group

 

[Dunuin]

4 things you could try:
1.) IOMMU groups are defined by your BIOS/UEFI and how the mainboard is electrically engineered. So buying another mainboard with better IOMMU groups might help. Especially when buying enterprise hardware you got better chances of better IOMMU groups and features like more PCIe lanes, bifurication , ... This site for example might help to check iOMMU groups before buying hardware: https://iommu.info/
2.) upgrade your BIOS/UEFI. Maybe the manufacturer patched better IOMMU groups with later firmware versions
3.) put your GPU in another PCIe slot that maybe got it's own IOMMU group
4.) make use of "acs_override" to split those IOMMU groups. But this isn't always working and also a big security problem as PCIe devices (and RAM because of DMA?) won't be isolated anymore.

 

[RockisLife]

So I finally got it to where im not having the issue of the GPUs, Now the bigger issue is that I cant have both of the VMs Running at the same time