Dual Nvidia 3080 GPUs work on same VM, but not if I have 2 VMs with 1 3080 GPU on each.

[Nathan Stratton]

I have 2 Nvidia 3080s, on PCI 0000:01:00.0 and 0000:02:00.0, if I put them both on a VM, with x-vga=on and multifunction=on, it works, I nvidia-smi shows two GPUs. However, if I start vm1 with one of the GPUs say 0000:01:00.0, it will start fine, if I then try to start vm2 with GPU 0000:02:00.0 I get an error:

/dev/rbd4
kvm: -device vfio-pci,host=0000:02:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,x-vga=on,multifunction=on: vfio 0000:02:00.0: failed to open /dev/vfio/49: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

If I stop vm1 and start vm2 with GPU 0000:02:00.0 it starts fine, but then when I try to start vm1 again I get:

/dev/rbd4
kvm: -device vfio-pci,host=0000:01:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,x-vga=on,multifunction=on: vfio 0000:01:00.0: failed to open /dev/vfio/49: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

Does anyone know why I can start a vm with 2 GPUs, but can't start 2 separate VMs with the two separate GPUs? This is getting very frustrating. The only thing I can remotely think is the issue is that my GPUs are on x8 PCI 4 bus vs x16 bus. Would that for some reason let me start two on one GPU, but not split them?

 

[Stefan_R]

Which IOMMU groups are those cards on? You should be able to tell by opening the "Add PCI device" dialog in the hardware tab of your VM or by running find /sys/kernel/iommu_groups/ -type l.

 

[Nathan Stratton]

Yep, they are both on 49, going to try a few BIOS settings if that does not work I found:

https://gitlab.com/Queuecumber/linux-acs-override

But rather see if I can do this without a custom kernel.

 

[Stefan_R]

We include the ACS override patch in pve-kernel by default. You just need to enable it by putting the appropriate 'pcie_acs_override=' kernel parameter in your bootloader config.

That being said, do note that this compromises security and potentially stabiliity on a fundamental level, so while it may make it work, I do not recommend using it in any important production environment.