Multi-GPU Passthrough To One VM

[mhayhurst]

Hello Everyone,

I'm trying to passthrough two PCIe GPU's (Radeon HD 7970's) to a Windows 7 VM. I've been successful in getting one of the Radeon HD 7970's to passthrough but the second one shows this issue in the Windows 7 Device Manager:

This device cannot find enough free resources that it can use. (Code 12)

If you want to use this device, you will need to disable one of the other devices on this system.

The Microsoft support site indicates I might have an I/O port conflict or the BIOS did not allocate sufficient resources (neither of which is really helpful). Here are my config settings for that particular VM:

bootdisk: sata0
cores: 8
hostpci0: 01:00.0;01:00.1,pcie=1,x-vga=on
hostpci1: 02:00.0;02:00.1,pcie=1,x-vga=on
ide2: local:iso/Windows.iso,media=cdrom
machine: q35
memory: 8192
name: Windows-7
net0: e1000=7A:DA:A8:96:16:54,bridge=vmbr0
numa: 1
ostype: win7
sata0: local-lvm:vm-100-disk-1,size=80G
scsihw: virtio-scsi-pci
smbios1: uuid=d8e7e49e-d579-4719-9ed3-77adb93adbe7
sockets: 1
usb0: host=046d:c52b

And here is everything in the two video cards IOMMU grouping:

/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.2
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.1

And my PCIe devices as they pertain to the IOMMU grouping:

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

Not really sure what I'm doing wrong but any help is always appreciated!

 

[mhayhurst]

Is anyone able to shed some light on this issue?

 

[Magneto]

/following with interest.

 

[TheFunk]

I had this same issue with passing through a single RX 480 to a Windows 10 VM. Eventually I gave up on the AMD card entirely and moved to NVIDIA. Problem solved! But here's a list of things you could try, based on what I learned from my experience.

1- Pass through the audio controller and the vga controller separately for each card. In other words your config should change to:

hostpci0: 01:00.0,pcie=1,x-vga=on
hostpci1: 01:00.1,pcie=1
hostpci2: 02:00.0,pcie=1,x-vga=on
hostpci3: 02:00.1,pcie=1

2- Change the IRQ type for the cards and their corresponding audio controllers in the guest OS. For whatever reason, my Windows system wanted my card using message signal based interrupts as opposed to line based interrupts. Here's a good howto post. Don't forget the reboot after you make the changes!

3- Run dmesg | tail -n 20 on the pmox host immediately after starting the VM with the passed through cards. If you get any sort of error whatsoever, Google it or post it here and I'll get back to you.

4- Check your host system's BIOS. I'm not sure what kind of motherboard you're using but on my Supermicro I had a few issues with my AMD card not booting to an EFI oprom. Speaking of EFI compatibility, that's a slightly older model card. If your card doesn't have an EFI ROM it won't function appropriately if you're using OVMF as your BIOS type. Check out techpowerup for a list of your card's available ROMs.

Windows 7 doesn't really have true UEFI support either. I don't know if that could be playing a role, but if you're not averse to Win 10, I'd try that instead if you're using OVMF on the guest. If you're using SEABIOS you should be fine.

I think a couple options I had to check were to enable PCIERR support and I had to enable Above 4G Decoding in my pmox host BIOS. Be weary of this particular BIOS option if you use it! From what I've read it's only supposed to be necessary for the system to work with NVIDIA cards that have more than 4GB of RAM. It screwed with me for a bit, but ultimately I think it was necessary for my RX480 as well for whatever reason. Definitely use this option last, because it borked my system graphics once or twice and I had to reset my BIOS by pulling the CMOS battery a couple times.