Issue: Unable to Separate GPU Devices into Different IOMMU Groups

[benjibananna]

Hello all,

I am facing an issue with my Proxmox setup where my GPU devices remain in the same IOMMU group due to hardware bifurcation on the motherboard. This prevents me from isolating them properly for use in different VMs.

System Details:​

• Motherboard: Gigabyte G292-Z20
• GPUs: Nvidia RTX A4000

IOMMU Grouping:​

Despite my efforts, the GPUs are grouped together in IOMMU Group 0. Here is the output:

IOMMU Group 0 c0:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 0 c0:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 0 c1:00.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4052]
IOMMU Group 0 c1:00.1 Memory controller [0580]: PMC-Sierra Inc. Device [11f8:4052]
IOMMU Group 0 c2:00.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4052]
IOMMU Group 0 c2:01.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4052]
IOMMU Group 0 c3:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
IOMMU Group 0 c3:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
IOMMU Group 0 c4:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
IOMMU Group 0 c4:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)

GRUB Configuration:​

Here is my current GRUB configuration:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX="quiet amd_iommu=on pci_acs_override=downstream,multifunction"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pci_acs_override=downstream,multifunction"

Motherboard Configuration:​

• IOMMU: Enabled
• ARI Support: Disabled
• ACS: Not visible in BIOS, but mentioned in the manual (considering BIOS downgrade)
• Build Date: 08/03/2021
• Firmware Version: R22

Error:​

When trying to run both VMs on the two GPUs that are in the same group, I encounter the following error:

kvm: -device vfio-pci,host=0000:c4:00.0,id=hostpci0,bus=pci.0,addr=0x10,rombar=0: vfio 0000:c4:00.0: failed to open /dev/vfio/0: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

I would appreciate any assistance or suggestions on how to resolve this issue and achieve proper separation of the GPU devices into different IOMMU groups.

Thanks everyone!

 

[leesteken]

The groups are determined by the physical PCIe layout and multiplexer chips. If you can't move the GPUs to different slots (that might be in different groups), try a different motherboard (but see if you can find the group information beforehand).
Please be aware that pcie_acs_override breaks VM isolation and the VM with the GPU can read and write all of the host memory, in principle.

 

[benjibananna]

Thanks for the quick reply. It's physically impossible to put them into a different slot, since there are 8 slots and 4 x PCIe 4.0 x16 lanes that are bifurcated. If i dig a bit deeper into the bios release notes I find this as well:

1. Modify PCIe device sequence.
1. Fix GPU resource issue.
4. Fixed PCIe device sequence.
1. Adjusting slot detection causes card loses issue.

It seems most likely, as this is enterprise datacenter hardware, that many enterprises run these with separate VMs.

 

[leesteken]

Can you show the output of cat /proc/cmdline to double check that pcie_acs_overide is active? I advocate agains using it, but that's all you can do in software.

It seems most likely, as this is enterprise datacenter hardware, that many enterprises run these with separate VMs.

If you think those groups are incorrect and do not resemble the PCIe infrastructure of the motherboard, please ask Gigabyte to fix this in the motherboard BIOS.

 

[benjibananna]

It seems that the output is not yet correct:

root@gigabyte-009:~# cat /proc/cmdline
initrd=\EFI\proxmox\6.8.8-3-pve\initrd.img-6.8.8-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on
root@gigabyte-009:~#

I am running AMD not INTEL.

 

[benjibananna]

 

[benjibananna]

Updated the /etc/kernel/cmdline to:

initrd=\EFI\proxmox\6.8.8-3-pve\initrd.img-6.8.8-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on pcie_acs_override=downstream,multifunction
proxmox-boot-tool refresh
update-grub
reboot

Rebooted the server and the IOMMU now shows the following:

IOMMU Group 16 c3:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 17 c3:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 18 c4:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 19 c4:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

Thank you!

 

[Ryohka233]

Thank you for this post, I'm encountering the same issue on a different MB and your solution fixed my issue as well. It's a ASRock B550 PG Riptied in case anyone need this info in the future.

 

[benjibananna]

Happy to hear that!

 

[LnxBil]

Please be aware that pcie_acs_override breaks VM isolation and the VM with the GPU can read and write all of the host memory, in principle.

Just to remind you of that.

 

[Ryohka233]

Just to remind you of that.

Yeah I'm aware of it, for a homelab environment it should be fine? I'm not running anything critical

 

[leesteken]

Yeah I'm aware of it, for a homelab environment it should be fine? I'm not running anything critical

If anyone breaks into one of your VMs, they can potentially read all of the host memory (and other VMs). You'll have to decide for yourself.