GPU passthrough tutorial/reference

[sshaikh]

Aim:

To host a headless VM with full access to a modern GPU, in order to stream games from.

Assumptions:

• Recent CPU and motherboard that supports VT-d, interrupt mapping.
• Recent GPU that has a UEFI bios.

Instructions:

1) Enable in BIOS: UEFI, VT-d, Multi-monitor mode

This is done via the BIOS. Can be confirmed using dmesg (search for efi strings), or the existence of /sys/firmware/efi on the filesystem and "vmx" in /proc/cpuinfo. Multi Monitor mode had to be enabled in my bios otherwise the card wasn't detected at all (even by the host using lspci).

2) Enable IOMMU via grub (Repeat post upgrade!)

edit /etc/default/grub and change

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

to

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifbff"

then run update-grub

Confirm using dmesg | grep -e DMAR -e IOMMU - this should produce output.

As of PVE 5, I had to also disable efifb.

3) Blacklist nvidia/nouveu so that Proxmox doesn't load the card (Repeat post upgrade!)

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Run update-initramfs -u to apply the above. Confirm using lspci -v - this will tell you if a driver has been loaded or not by the VGA adaptor.

4) Load kernel modules for virtual IO

Add to /etc/modules the following:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I'm not sure how to confirm the above.

5) Get GPU IDs and addresses

Run lspci -v to list all the devices in your PC. Find the relevant VGA card entry. For example:

01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

You may also have an audio device (probably for HDMI sound):

01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

Take note of the numbers at the front, in this case 01:00.0 and 01:00.1.

Using this number run lspci -n -s 01:00. This will give you the vendor ids. For example:

01:00.0 0000: 10de:1b81 (rev a1)
01:00.1 0000: 10de:10f0 (rev a1)

Take note of these vendor IDs, in this case 10de:1b81 and 10de:10f0.

6) Assign GPU to vfio

Use this to create the file that assigns the HW to vfio:

echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

After rebooting, running lspci -v will confirm that the GPU and Audio device are using the vfio driver:

Kernel driver in use: vfio-pci

7) Create VM (but do not start it!)

Do this as normal, using SCSI VirtIO, VirtIO net and balloon virtual hardware. Also add the following to the vm's conf file (/etc/pve/qemu-server/<vmid>.conf):

bios: ovmf
machine: q35

8) Install Windows 10 in the VM

You can now install Win10, with it being aware of the UEFI bios. You may (will) need to provide VirtIO drivers during install.

Once up and running, TURN ON REMOTE DESKTOP. Passing through the GPU will disable the virtual display, so you will not be able to access it via Proxmox/VNC. Remote desktop will be handy if you don't have a monitor connected or keyboard passed through.

9) Pass through the GPU!

This is the actual installing of the GPU into the VM. Add the following to the vm's conf file:

hostpci0: <device address>,x-vga=on,pcie=1

In the examples above, using 01:00 as the address will pass through both 01:00.0 and 01:00.1, which is probably what you want. x-vga will do some compatibility magic, as well as disabling the basic VGA adaptor.

You can verify the passthrough by starting the VM and entering info pci into the respective VM monitor tab in the Proxmox webui. This should list the VGA and audio device, with an id of hostpci0.0 and hostpci0.1.

Windows should automatically install a driver. You can allow this and confirm in device manager that the card is loaded correctly (ie without any "code 43" errors). Once that's done continue to set up the card (drivers etc).

 

[capn_tack]

Just want to let you know that you saved my bacon: your post contained the last little bit I needed to get my setup running! I'll do a full write up and post it later once I've done some testing to make sure it's 100%. I'll try to remember to link it here as well.

 

[sshaikh]

That's great to hear, I'm glad it was of use. What bit helped exactly?

 

[websterxsd]

Hi,

Your guide has helped me immensly with getting the GPU to work on my VM, we use it for Terminal Services, so it should be a great one. Just a question. When I want to use a AMD GPU it totaly crashes for me, nVIDIA works just fine. Did you experience this yourself?

Also, when trying to run PerformanceTest from PassMark Software it completly crashes the system, any ideas on that?

Also, I'm not seeing any display shown when the cabbles are attached.

Still, much thanks overall with helping with the issue!

With regards,

Webster

 

[sshaikh]

I've not tried AMD myself, and although my instructions do refer to AMD (eg when blacklisting) they are by no means generic. That said, I don't see why they shouldn't work without some tweaking. Try messing with the bios of the host/guest, and check that your card is UEFI?

I used Unigine Heaven to benchmark, and that was with a local monitor and mouse etc - it wouldn't run very well over RDP. So yes, I did have a display (of the guest) with a monitor connected.

 

[Dorin]

host a headless VM with full access to a modern GPU, in order to stream games from.

Assumptions:

• Recent CPU and motherboard that supports VT-d, interrupt mapping.
• Recent GPU that has a UEFI bios.

Instructions:

1) Enable in BIOS: UEFI, VT-d, Multi-monitor mode

This is done via the BIOS. Can be confirmed using dmesg (search for efi strings), or the existence of /sys/firmware/efi on the filesystem and "vmx" in /proc/cpuinfo. Multi Monitor mode had to be enabled in my bios otherwise the card wasn't detected at all (even by the host using lspci).

2) Enable IOMMU via grub (Repeat post upgrade!)

edit /etc/default/grub and change

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

to

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifbff"

then run update-grub

Confirm using dmesg | grep -e DMAR -e IOMMU - this should produce output.

As of PVE 5, I had to also disable efifb.

3) Blacklist nvidia/nouveu so that Proxmox doesn't load the card (Repeat post upgrade!)

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Run update-initramfs -u to apply the above. Confirm using lspci -v - this will tell you if a driver has been loaded or not by the VGA adaptor.

4) Load kernel modules for virtual IO

Add to /etc/modules the following:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I'm not sure how to confirm the above.

5) Get GPU IDs and addresses

Run lspci -v to list all the devices in your PC. Find the relevant VGA card entry. For example:

01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

You may also have an audio device (probably for HDMI sound):

01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

Take note of the numbers at the front, in this case 01:00.0 and 01:00.1.

Using this number run lspci -n -s 01:00. This will give you the vendor ids. For example:

01:00.0 0000: 10de:1b81 (rev a1)
01:00.1 0000: 10de:10f0 (rev a1)

Take note of these vendor IDs, in this case 10de:1b81 and 10de:10f0.

6) Assign GPU to vfio

Use this to create the file that assigns the HW to vfio:

echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

After rebooting, running lspci -v will confirm that the GPU and Audio device are using the vfio driver:

Kernel driver in use: vfio-pci

7) Create VM (but do not start it!)

Do this as normal, using SCSI VirtIO, VirtIO net and balloon virtual hardware. Also add the following to the vm's conf file (/etc/pve/qemu-server/<vmid>.conf):

bios: ovmf
machine: q35

8) Install Windows 10 in the VM

You can now install Win10, with it being aware of the UEFI bios. You may (will) need to provide VirtIO drivers during install.

Once up and running, TURN ON REMOTE DESKTOP. Passing through the GPU will disable the virtual display, so you will not be able to access it via Proxmox/VNC. Remote desktop will be handy if you don't have a monitor connected or keyboard passed through.

9) Pass through the GPU!

This is the actual installing of the GPU into the VM. Add the following to the vm's conf file:

hostpci0: <device address>,x-vga=on,pcie=1

In the examples above, using 01:00 as the address will pass through both 01:00.0 and 01:00.1, which is probably what you want. x-vga will do some compatibility magic, as well as disabling the basic VGA adaptor.

You can verify the passthrough by starting the VM and entering info pci into the respective VM monitor tab in the Proxmox webui. This should list the VGA and audio device, with an id of hostpci0.0 and hostpci0.1.

Windows should automatically install a driver. You can allow this and confirm in device manager that the card is loaded correctly (ie without any "code 43" errors). Once that's done continue to set up the card (drivers etc).

I followed your instructions, but i can't tell if is truly working because i don't have a monitor connected to the video card (GTX 1050 Ti, 4GB), but i can remotely connect on the guest (Win7 x64).
-The info from "info pci" seems to be ok.
-The status of the video adapter seems to not be ok.

I have two questions:
-Did someone from you tried to add "romfile" in vm's config file (if applicable)? "hostpci[n]: [host=]<HOSTPCIID[;HOSTPCIID2...]> [,pcie=<1|0>] [,rombar=<1|0>] [,romfile=<string>] [,x-vga=<1|0>]", https://pve.proxmox.com/wiki/Manual:_qm.conf
-Does this procedure require to hide virtualization from the Windows guest?
Something like:
"
<features>
<hyperv>
...
<vendor_id state='on' value='whatever'/>
...
</hyperv>
...
<kvm>
<hidden state='on'/>
</kvm>
</features>
", taken from a QEMU XML config file. How could i do this in Proxmox?

 

[drdownload]

Hi, I've also followed the guide and Passthrough worked like a charm. 2 days later I try to start the VM and --- nothing just a black screen, no IP is issued by the Router.

If I remove the hostpci line. The VM boots up without a hick.

If I try to run info pci while starting with hostpci I just get an error that it could not connect.

 

[sshaikh]

@Dorin For my set up, my instructions are pretty complete (I've rebuilt it a few times now!). During my trials, I established that a romfile wasn't needed (as its a newer card with ACPI or something, I'm not quite sure). As for hidden modes, ISTR that the x-vga option in proxmox does this for you. Also please bear in mind that remote desktop interferes with the graphics card operation (although it shouldn't stop it from starting like in your pictures), so stick to VNC.

@drdownload perhaps 1 time out of 6 my VM hangs while booting. If I don't get a connection within five mins, I do a hard reset of the VM. I suspect there's a message or update that I can't see.

 

[drdownload]

@sshaikh: I don't now, It's quiet persistant - I'll try to passthrough the device to my antegos install to see if its the VM or the setup.

 

[drdownload]

@sshaikh: I tired to pass it to a linux VM and this one also failed to start. If I look at the end of the VM protocol line: got timeout

 

[drdownload]

I remembered that I played with the Bios parameter to witch from ahci to intel rst for the drive access. (also I rebooted to switch it back) dont know what helped. but now it works again

 

[Dorin]

@Dorin For my set up, my instructions are pretty complete (I've rebuilt it a few times now!). During my trials, I established that a romfile wasn't needed (as its a newer card with ACPI or something, I'm not quite sure). As for hidden modes, ISTR that the x-vga option in proxmox does this for you. Also please bear in mind that remote desktop interferes with the graphics card operation (although it shouldn't stop it from starting like in your pictures), so stick to VNC.

@drdownload perhaps 1 time out of 6 my VM hangs while booting. If I don't get a connection within five mins, I do a hard reset of the VM. I suspect there's a message or update that I can't see.

I still couldn't try to see if the video output is working by connecting a monitor to the graphic card, but i have installed the VNC Server in guest os.
After connecting with VNC Viewer i saw an additional "VNC Mirror Driver" device in "Display adapters" section.

1. The result after connecting with VNC Viewer when hostpci0 parameter is commented, "#hostpci0: 01:00,x-vga=on,pcie=1": "Device status" of "Standard VGA Graphics Adapter" is "This device is working properly".
The same status is when i'm using Proxmox vm's console.

2. The result after connecting with VNC Viewer when hostpci0 parameter is enabled, "hostpci0: 01:00,x-vga=on,pcie=1" (with or without "VNC Mirror Driver" enabled): just a black and blank screen.

 

[Dorin]

host a headless VM with full access to a modern GPU, in order to stream games from.

Assumptions:

• Recent CPU and motherboard that supports VT-d, interrupt mapping.
• Recent GPU that has a UEFI bios.

Instructions:

1) Enable in BIOS: UEFI, VT-d, Multi-monitor mode

This is done via the BIOS. Can be confirmed using dmesg (search for efi strings), or the existence of /sys/firmware/efi on the filesystem and "vmx" in /proc/cpuinfo. Multi Monitor mode had to be enabled in my bios otherwise the card wasn't detected at all (even by the host using lspci).

2) Enable IOMMU via grub (Repeat post upgrade!)

edit /etc/default/grub and change

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

to

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifbff"

then run update-grub

Confirm using dmesg | grep -e DMAR -e IOMMU - this should produce output.

As of PVE 5, I had to also disable efifb.

3) Blacklist nvidia/nouveu so that Proxmox doesn't load the card (Repeat post upgrade!)

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Run update-initramfs -u to apply the above. Confirm using lspci -v - this will tell you if a driver has been loaded or not by the VGA adaptor.

4) Load kernel modules for virtual IO

Add to /etc/modules the following:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I'm not sure how to confirm the above.

5) Get GPU IDs and addresses

Run lspci -v to list all the devices in your PC. Find the relevant VGA card entry. For example:

01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

You may also have an audio device (probably for HDMI sound):

01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

Take note of the numbers at the front, in this case 01:00.0 and 01:00.1.

Using this number run lspci -n -s 01:00. This will give you the vendor ids. For example:

01:00.0 0000: 10de:1b81 (rev a1)
01:00.1 0000: 10de:10f0 (rev a1)

Take note of these vendor IDs, in this case 10de:1b81 and 10de:10f0.

6) Assign GPU to vfio

Use this to create the file that assigns the HW to vfio:

echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

After rebooting, running lspci -v will confirm that the GPU and Audio device are using the vfio driver:

Kernel driver in use: vfio-pci

7) Create VM (but do not start it!)

Do this as normal, using SCSI VirtIO, VirtIO net and balloon virtual hardware. Also add the following to the vm's conf file (/etc/pve/qemu-server/<vmid>.conf):

bios: ovmf
machine: q35

8) Install Windows 10 in the VM

You can now install Win10, with it being aware of the UEFI bios. You may (will) need to provide VirtIO drivers during install.

Once up and running, TURN ON REMOTE DESKTOP. Passing through the GPU will disable the virtual display, so you will not be able to access it via Proxmox/VNC. Remote desktop will be handy if you don't have a monitor connected or keyboard passed through.

9) Pass through the GPU!

This is the actual installing of the GPU into the VM. Add the following to the vm's conf file:

hostpci0: <device address>,x-vga=on,pcie=1

In the examples above, using 01:00 as the address will pass through both 01:00.0 and 01:00.1, which is probably what you want. x-vga will do some compatibility magic, as well as disabling the basic VGA adaptor.

You can verify the passthrough by starting the VM and entering info pci into the respective VM monitor tab in the Proxmox webui. This should list the VGA and audio device, with an id of hostpci0.0 and hostpci0.1.

Windows should automatically install a driver. You can allow this and confirm in device manager that the card is loaded correctly (ie without any "code 43" errors). Once that's done continue to set up the card (drivers etc).

In my case i found that i have to use: "hostpci0: 01:00,x-vga=on,pcie=1,romfile=vbios.bin", otherwise the output of graphic card is not enabled.
During VM's boot up process the VM seems to enter into an infinite loop.
This happens every time, no matter what option i choose (Safe Mode / Normally) from Safe Mode boot menu.
The time i have waited for boot up process is about 1 minute more compared with the setup when the guest is not set in passthrough mode for GPU.
I also tried with different versions of vbios (GTX 1050 Ti, 4 GB; excepting the vbios of my card) and the machine still can't boot.
What is interesting is that i'm allowed to remotelly connect on guest with VNC Viewer, but in any boot mode case selected the screen is black.
Did you experienced this behavior?

 

[drdownload]

Another question: If I do not use the Graphics Adapter (VM not started) can I also use it for the normal Proxmox terminal?

I'm thinking about using the onboard intel for a mac os vm?

 

[Dorin]

Another question: If I do not use the Graphics Adapter (VM not started) can I also use it for the normal Proxmox terminal?

I'm thinking about using the onboard intel for a mac os vm?

My system has an integrated video device (in Xeon processor) and one PCI-E 3.0 slot (slot 1).
After i plugged in the GPU in slot 1, the system (by default) has switched off the integrated video device (i had no output). When i unplugged the GPU from slot 1, the output on integrated video card has been re-enabled, but your system may have a different behavior.
In my case, in BIOS, i don't have the possibility to select manually the primary video adapter.

 

[drdownload]

Yes. My Mainboard can force the intel onboard to stay on an be primary (post) display device.

But since I'm reinstalling Proxmox this weekend I will try what happends if I passthrough both the onboard and the pcie graphics card.

 

[VTOLfreak]

New on the forums here. I followed this guide but I ended up with a PCIe bus error on my console.
Motherboard: Asrock E3C236D4U
CPU: E3-1275v5
GPU: XFX Radeon 480 8GB

Note that this board also comes with IPMI. (aspeed2400)
After applying the instructions from this thread the console output jumps from the GPU output to the IPMI output.
The BIOS shows up on the Radeon and as soon as Proxmox boots the console output jumps to the IPMI adapter.

I get the error in the screenshot when I try to start the VM. Any clues or anything I can check?

 

[sshaikh]

@Dorin @drdownload I didn't realise there were replies, so I'm not sure which bits you still need help with. FWIW I noticed a Windows 7 boot screen at some point; I would suggest that later operating systems use UEFI and stuff which will def make a difference, and possibly why a romfile had to be used.

@VTOLfreak I've never seen that error before, but I would try disabling any remote access doohickies. Apart from the BIOS the GPU shouldn't display anything (in fact it should be off) until the VM starts, yes. Other things to check is your machine is q35 and as above that you're using UEFI everywhere.

 

[VTOLfreak]

@VTOLfreak I've never seen that error before, but I would try disabling any remote access doohickies. Apart from the BIOS the GPU shouldn't display anything (in fact it should be off) until the VM starts, yes. Other things to check is your machine is q35 and as above that you're using UEFI everywhere.

Just for testing I swapped my RX480 out with an old HD5450. VM started right up, both the passthrough GPU and virtual GPU is present in the VM. Proxmox console output shows up on IPMI.

Maybe I can sell my RX480 to a miner...

 

[Miguel Moreira]

in my case i do exactly the process:

1.- /etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifbff"
GRUB_CMDLINE_LINUX=""


2.- cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67df;1002:aaf0

3.- cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist amdgpu

4.- cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd


5.-CONFIG VM ***********************
bios: seabios
boot: cdn
bootdisk: ide1
cores: 4
hostpci0: 84:00.0,pcie=1,x-vga=on
ide1: local-lvm:vm-420-disk-1,size=50G
machine: q35
memory: 8200
name: test-ethos
net0: virtio=2A:53:68:3E:BD:20,bridge=vmbr10
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=16a6b477-2ccd-4a1b-a361-193e4b188719
sockets: 2


error ...

# qm start 420
kvm: -device ide-hd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1,bootindex=100: Can't create IDE unit 1, bus supports only 1 units
kvm: -device ide-hd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1,bootindex=100: Device initialization failed.
start failed: command '/usr/bin/kvm -id 420 -chardev 'socket,id=qmp,path=/var/run/qemu-server/420.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/420.pid -daemonize -smbios 'type=1,uuid=16a6b477-2ccd-4a1b-a361-193e4b188719' -name test-ethos -smp '8,sockets=2,cores=4,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,enforce,kvm=off' -m 8200 -object 'memory-backend-ram,id=ram-node0,size=4100M' -numa 'node,nodeid=0,cpus=0-3,memdev=ram-node0' -object 'memory-backend-ram,id=ram-node1,size=4100M' -numa 'node,nodeid=1,cpus=4-7,memdev=ram-node1' -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=84:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:aad8fe7d304f' -drive 'file=/dev/pve/vm-420-disk-1,if=none,id=drive-ide1,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap420i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=2A:53:68:3E:BD:20,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1


this is my messages or wrong.

I need HELP please !!