Multi-GPU Passthrough To One VM

Discussion in 'Proxmox VE: Installation and configuration' started by mhayhurst, Jul 17, 2017.

  1. mhayhurst

    mhayhurst Member

    Joined:
    Jul 21, 2016
    Messages:
    51
    Likes Received:
    1
    Hello Everyone,

    I'm trying to passthrough two PCIe GPU's (Radeon HD 7970's) to a Windows 7 VM. I've been successful in getting one of the Radeon HD 7970's to passthrough but the second one shows this issue in the Windows 7 Device Manager:
    Code:
    This device cannot find enough free resources that it can use. (Code 12)
    
    If you want to use this device, you will need to disable one of the other devices on this system.
    The Microsoft support site indicates I might have an I/O port conflict or the BIOS did not allocate sufficient resources (neither of which is really helpful). Here are my config settings for that particular VM:
    Code:
    bootdisk: sata0
    cores: 8
    hostpci0: 01:00.0;01:00.1,pcie=1,x-vga=on
    hostpci1: 02:00.0;02:00.1,pcie=1,x-vga=on
    ide2: local:iso/Windows.iso,media=cdrom
    machine: q35
    memory: 8192
    name: Windows-7
    net0: e1000=7A:DA:A8:96:16:54,bridge=vmbr0
    numa: 1
    ostype: win7
    sata0: local-lvm:vm-100-disk-1,size=80G
    scsihw: virtio-scsi-pci
    smbios1: uuid=d8e7e49e-d579-4719-9ed3-77adb93adbe7
    sockets: 1
    usb0: host=046d:c52b
    
    And here is everything in the two video cards IOMMU grouping:
    Code:
    /sys/kernel/iommu_groups/1/devices/0000:00:01.0
    /sys/kernel/iommu_groups/1/devices/0000:00:01.2
    /sys/kernel/iommu_groups/1/devices/0000:01:00.0
    /sys/kernel/iommu_groups/1/devices/0000:01:00.1
    /sys/kernel/iommu_groups/1/devices/0000:02:00.0
    /sys/kernel/iommu_groups/1/devices/0000:02:00.1
    And my PCIe devices as they pertain to the IOMMU grouping:
    Code:
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
    02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
    02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
    
    Not really sure what I'm doing wrong but any help is always appreciated!
     
  2. mhayhurst

    mhayhurst Member

    Joined:
    Jul 21, 2016
    Messages:
    51
    Likes Received:
    1
    Is anyone able to shed some light on this issue?
     
    ONE FOTON likes this.
  3. SilverNodashi

    SilverNodashi Member

    Joined:
    Jul 30, 2017
    Messages:
    95
    Likes Received:
    2
    /following with interest.
     
  4. TheFunk

    TheFunk Member

    Joined:
    Oct 25, 2016
    Messages:
    35
    Likes Received:
    3
    I had this same issue with passing through a single RX 480 to a Windows 10 VM. Eventually I gave up on the AMD card entirely and moved to NVIDIA. Problem solved! But here's a list of things you could try, based on what I learned from my experience.

    1- Pass through the audio controller and the vga controller separately for each card. In other words your config should change to:

    Code:
    hostpci0: 01:00.0,pcie=1,x-vga=on
    hostpci1: 01:00.1,pcie=1
    hostpci2: 02:00.0,pcie=1,x-vga=on
    hostpci3: 02:00.1,pcie=1
    2- Change the IRQ type for the cards and their corresponding audio controllers in the guest OS. For whatever reason, my Windows system wanted my card using message signal based interrupts as opposed to line based interrupts. Here's a good howto post. Don't forget the reboot after you make the changes!

    3- Run dmesg | tail -n 20 on the pmox host immediately after starting the VM with the passed through cards. If you get any sort of error whatsoever, Google it or post it here and I'll get back to you.

    4- Check your host system's BIOS. I'm not sure what kind of motherboard you're using but on my Supermicro I had a few issues with my AMD card not booting to an EFI oprom. Speaking of EFI compatibility, that's a slightly older model card. If your card doesn't have an EFI ROM it won't function appropriately if you're using OVMF as your BIOS type. Check out techpowerup for a list of your card's available ROMs.

    Windows 7 doesn't really have true UEFI support either. I don't know if that could be playing a role, but if you're not averse to Win 10, I'd try that instead if you're using OVMF on the guest. If you're using SEABIOS you should be fine.

    I think a couple options I had to check were to enable PCIERR support and I had to enable Above 4G Decoding in my pmox host BIOS. Be weary of this particular BIOS option if you use it! From what I've read it's only supposed to be necessary for the system to work with NVIDIA cards that have more than 4GB of RAM. It screwed with me for a bit, but ultimately I think it was necessary for my RX480 as well for whatever reason. Definitely use this option last, because it borked my system graphics once or twice and I had to reset my BIOS by pulling the CMOS battery a couple times.
     
    ONE FOTON likes this.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice