1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

GPU passthrough tutorial/reference

Discussion in 'Proxmox VE: Installation and configuration' started by sshaikh, Apr 23, 2017.

  1. sshaikh

    sshaikh New Member

    Joined:
    Apr 23, 2017
    Messages:
    21
    Likes Received:
    2
    host a headless VM with full access to a modern GPU, in order to stream games from.

    Assumptions:
    • Recent CPU and motherboard that supports VT-d, interrupt mapping.
    • Recent GPU that has a UEFI bios.
    Instructions:

    1) Enable in BIOS: UEFI, VT-d, Multi-monitor mode

    This is done via the BIOS. Can be confirmed using dmesg (search for efi strings), or the existence of /sys/firmware/efi on the filesystem and "vmx" in /proc/cpuinfo. Multi Monitor mode had to be enabled in my bios otherwise the card wasn't detected at all (even by the host using lspci).

    2) Enable IOMMU via grub (Repeat post upgrade!)

    edit /etc/default/grub and change

    GRUB_CMDLINE_LINUX_DEFAULT="quiet"

    to

    GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifb:eek:ff"

    then run update-grub

    Confirm using dmesg | grep -e DMAR -e IOMMU - this should produce output.

    As of PVE 5, I had to also disable efifb.

    3) Blacklist nvidia/nouveu so that Proxmox doesn't load the card (Repeat post upgrade!)

    echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
    echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
    echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

    Run update-initramfs -u to apply the above. Confirm using lspci -v - this will tell you if a driver has been loaded or not by the VGA adaptor.

    4) Load kernel modules for virtual IO

    Add to /etc/modules the following:

    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd

    I'm not sure how to confirm the above.

    5) Get GPU IDs and addresses

    Run lspci -v to list all the devices in your PC. Find the relevant VGA card entry. For example:

    01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

    You may also have an audio device (probably for HDMI sound):

    01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

    Take note of the numbers at the front, in this case 01:00.0 and 01:00.1.

    Using this number run lspci -n -s 01:00. This will give you the vendor ids. For example:

    01:00.0 0000: 10de:1b81 (rev a1)
    01:00.1 0000: 10de:10f0 (rev a1)

    Take note of these vendor IDs, in this case 10de:1b81 and 10de:10f0.

    6) Assign GPU to vfio

    Use this to create the file that assigns the HW to vfio:

    echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

    After rebooting, running lspci -v will confirm that the GPU and Audio device are using the vfio driver:

    Kernel driver in use: vfio-pci

    7) Create VM (but do not start it!)

    Do this as normal, using SCSI VirtIO, VirtIO net and balloon virtual hardware. Also add the following to the vm's conf file (/etc/pve/qemu-server/<vmid>.conf):

    bios: ovmf
    machine: q35

    8) Install Windows 10 in the VM

    You can now install Win10, with it being aware of the UEFI bios. You may (will) need to provide VirtIO drivers during install.

    Once up and running, TURN ON REMOTE DESKTOP. Passing through the GPU will disable the virtual display, so you will not be able to access it via Proxmox/VNC. Remote desktop will be handy if you don't have a monitor connected or keyboard passed through.

    9) Pass through the GPU!

    This is the actual installing of the GPU into the VM. Add the following to the vm's conf file:

    hostpci0: <device address>,x-vga=on,pcie=1

    In the examples above, using 01:00 as the address will pass through both 01:00.0 and 01:00.1, which is probably what you want. x-vga will do some compatibility magic, as well as disabling the basic VGA adaptor.

    You can verify the passthrough by starting the VM and entering info pci into the respective VM monitor tab in the Proxmox webui. This should list the VGA and audio device, with an id of hostpci0.0 and hostpci0.1.

    Windows should automatically install a driver. You can allow this and confirm in device manager that the card is loaded correctly (ie without any "code 43" errors). Once that's done continue to set up the card (drivers etc).
     
    #1 sshaikh, Apr 23, 2017
    Last edited: Jul 17, 2017
    Pablo Alcaraz and fireon like this.
  2. capn_tack

    capn_tack New Member

    Joined:
    May 7, 2017
    Messages:
    1
    Likes Received:
    0
    Just want to let you know that you saved my bacon: your post contained the last little bit I needed to get my setup running! I'll do a full write up and post it later once I've done some testing to make sure it's 100%. I'll try to remember to link it here as well.
     
  3. sshaikh

    sshaikh New Member

    Joined:
    Apr 23, 2017
    Messages:
    21
    Likes Received:
    2
    That's great to hear, I'm glad it was of use. What bit helped exactly?
     
  4. websterxsd

    websterxsd New Member

    Joined:
    May 15, 2017
    Messages:
    1
    Likes Received:
    0
    Hi,

    Your guide has helped me immensly with getting the GPU to work on my VM, we use it for Terminal Services, so it should be a great one. Just a question. When I want to use a AMD GPU it totaly crashes for me, nVIDIA works just fine. Did you experience this yourself?

    Also, when trying to run PerformanceTest from PassMark Software it completly crashes the system, any ideas on that?

    Also, I'm not seeing any display shown when the cabbles are attached.

    Still, much thanks overall with helping with the issue!

    With regards,

    Webster
     
    #4 websterxsd, May 16, 2017
    Last edited: May 16, 2017
  5. sshaikh

    sshaikh New Member

    Joined:
    Apr 23, 2017
    Messages:
    21
    Likes Received:
    2
    I've not tried AMD myself, and although my instructions do refer to AMD (eg when blacklisting) they are by no means generic. That said, I don't see why they shouldn't work without some tweaking. Try messing with the bios of the host/guest, and check that your card is UEFI?

    I used Unigine Heaven to benchmark, and that was with a local monitor and mouse etc - it wouldn't run very well over RDP. So yes, I did have a display (of the guest) with a monitor connected.
     
  6. Dorin

    Dorin New Member

    Joined:
    Sep 11, 2017
    Messages:
    10
    Likes Received:
    0
    I followed your instructions, but i can't tell if is truly working because i don't have a monitor connected to the video card (GTX 1050 Ti, 4GB), but i can remotely connect on the guest (Win7 x64).
    -The info from "info pci" seems to be ok.
    -The status of the video adapter seems to not be ok.

    I have two questions:
    -Did someone from you tried to add "romfile" in vm's config file (if applicable)? "hostpci[n]: [host=]<HOSTPCIID[;HOSTPCIID2...]> [,pcie=<1|0>] [,rombar=<1|0>] [,romfile=<string>] [,x-vga=<1|0>]", https://pve.proxmox.com/wiki/Manual:_qm.conf
    -Does this procedure require to hide virtualization from the Windows guest?
    Something like:
    "
    <features>
    <hyperv>
    ...
    <vendor_id state='on' value='whatever'/>
    ...
    </hyperv>
    ...
    <kvm>
    <hidden state='on'/>
    </kvm>
    </features>
    ", taken from a QEMU XML config file. How could i do this in Proxmox?

    Win7 x64 VM - display adapter.PNG
    VM  monitor - info pci.PNG
     
  7. drdownload

    drdownload New Member

    Joined:
    Sep 7, 2017
    Messages:
    14
    Likes Received:
    0
    Hi, I've also followed the guide and Passthrough worked like a charm. 2 days later I try to start the VM and --- nothing just a black screen, no IP is issued by the Router.

    If I remove the hostpci line. The VM boots up without a hick.

    If I try to run info pci while starting with hostpci I just get an error that it could not connect.
     
  8. sshaikh

    sshaikh New Member

    Joined:
    Apr 23, 2017
    Messages:
    21
    Likes Received:
    2
    @Dorin For my set up, my instructions are pretty complete (I've rebuilt it a few times now!). During my trials, I established that a romfile wasn't needed (as its a newer card with ACPI or something, I'm not quite sure). As for hidden modes, ISTR that the x-vga option in proxmox does this for you. Also please bear in mind that remote desktop interferes with the graphics card operation (although it shouldn't stop it from starting like in your pictures), so stick to VNC.

    @drdownload perhaps 1 time out of 6 my VM hangs while booting. If I don't get a connection within five mins, I do a hard reset of the VM. I suspect there's a message or update that I can't see.
     
  9. drdownload

    drdownload New Member

    Joined:
    Sep 7, 2017
    Messages:
    14
    Likes Received:
    0
    @sshaikh: I don't now, It's quiet persistant ;) - I'll try to passthrough the device to my antegos install to see if its the VM or the setup.
     
  10. drdownload

    drdownload New Member

    Joined:
    Sep 7, 2017
    Messages:
    14
    Likes Received:
    0
    @sshaikh: I tired to pass it to a linux VM and this one also failed to start. If I look at the end of the VM protocol line: got timeout
     
  11. drdownload

    drdownload New Member

    Joined:
    Sep 7, 2017
    Messages:
    14
    Likes Received:
    0
    I remembered that I played with the Bios parameter to witch from ahci to intel rst for the drive access. (also I rebooted to switch it back) dont know what helped. but now it works again
     
  12. Dorin

    Dorin New Member

    Joined:
    Sep 11, 2017
    Messages:
    10
    Likes Received:
    0
    I still couldn't try to see if the video output is working by connecting a monitor to the graphic card, but i have installed the VNC Server in guest os.
    After connecting with VNC Viewer i saw an additional "VNC Mirror Driver" device in "Display adapters" section.

    1. The result after connecting with VNC Viewer when hostpci0 parameter is commented, "#hostpci0: 01:00,x-vga=on,pcie=1": "Device status" of "Standard VGA Graphics Adapter" is "This device is working properly".
    The same status is when i'm using Proxmox vm's console.

    2. The result after connecting with VNC Viewer when hostpci0 parameter is enabled, "hostpci0: 01:00,x-vga=on,pcie=1" (with or without "VNC Mirror Driver" enabled): just a black and blank screen.
     

    Attached Files:

  13. Dorin

    Dorin New Member

    Joined:
    Sep 11, 2017
    Messages:
    10
    Likes Received:
    0
    In my case i found that i have to use: "hostpci0: 01:00,x-vga=on,pcie=1,romfile=vbios.bin", otherwise the output of graphic card is not enabled.
    During VM's boot up process the VM seems to enter into an infinite loop.
    This happens every time, no matter what option i choose (Safe Mode / Normally) from Safe Mode boot menu.
    The time i have waited for boot up process is about 1 minute more compared with the setup when the guest is not set in passthrough mode for GPU.
    I also tried with different versions of vbios (GTX 1050 Ti, 4 GB; excepting the vbios of my card) and the machine still can't boot.
    What is interesting is that i'm allowed to remotelly connect on guest with VNC Viewer, but in any boot mode case selected the screen is black.
    Did you experienced this behavior?
     

    Attached Files:

    #13 Dorin, Sep 14, 2017
    Last edited: Sep 14, 2017
  14. drdownload

    drdownload New Member

    Joined:
    Sep 7, 2017
    Messages:
    14
    Likes Received:
    0
    Another question: If I do not use the Graphics Adapter (VM not started) can I also use it for the normal Proxmox terminal?

    I'm thinking about using the onboard intel for a mac os vm?
     
  15. Dorin

    Dorin New Member

    Joined:
    Sep 11, 2017
    Messages:
    10
    Likes Received:
    0
    My system has an integrated video device (in Xeon processor) and one PCI-E 3.0 slot (slot 1).
    After i plugged in the GPU in slot 1, the system (by default) has switched off the integrated video device (i had no output). When i unplugged the GPU from slot 1, the output on integrated video card has been re-enabled, but your system may have a different behavior.
    In my case, in BIOS, i don't have the possibility to select manually the primary video adapter.
     

Share This Page