I have no idea why I can't get this to work (GPU Passthrough)

[TheKingOfRainyLands]

I have been following these tutorials.
forum.proxmox(dot)com/threads/gpu-passthrough-tutorial-reference.34303/
forum.level1techs(dot)com/t/single-gpu-passthrough-with-proxmox/113282
pve.proxmox(dot)com/wiki/Pci_passthrough#GPU_PASSTHROUGH

To attempt to set up GPU passthrough on a VM.

Here is the .conf for that machine.

bootdisk: sata0
cores: 4
cpu: IvyBridge
ide2: local:iso/Windows10.iso,media=cdrom
memory: 8192
name: windows
net0: e1000=FE:BD:DC:2E:D3:71,bridge=vmbr0
numa: 0
ostype: win10
sata0: local-lvm:vm-100-disk-1,size=120G
scsihw: virtio-scsi-pci
smbios1: uuid=b897e6d0-7c05-49ea-bb15-2b538ded6667
sockets: 1
bios: ovmf
machine: q35
hostpci0: 02:00,pcie=1,x-vga=on,romfile=Y.rom

lspci -v says that:

02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd GK104 [GeForce GTX 670]
        Flags: fast devsel, IRQ 17
        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=128M]
        Memory at e8000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000
        Expansion ROM at f7000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

02:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd GK104 HDMI Audio Controller
        Flags: fast devsel, IRQ 18
        Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

Package Versions:

Package versions
proxmox-ve: 5.0-15 (running kernel: 4.10.15-1-pve)
pve-manager: 5.0-23 (running version: 5.0-23/af4267bf)
pve-kernel-4.10.15-1-pve: 4.10.15-15
libpve-http-server-perl: 2.0-5
lvm2: 2.02.168-pve2
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-10
qemu-server: 5.0-12
pve-firmware: 2.0-2
libpve-common-perl: 5.0-16
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-5
libpve-storage-perl: 5.0-12
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-6
pve-qemu-kvm: 2.9.0-2
pve-container: 2.0-14
pve-firewall: 3.0-1
pve-ha-manager: 2.0-2
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-3
lxcfs: 2.0.7-pve2
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.9-pve16~bpo90

Yes, I have confirmed that UEFI is enabled as /sys/firmware/efi exists.

Machine Specs:
Intel Core i7 - 3770 (I have confirmed that this has VT-d on Intel's website)
GIGABYTE GeForce GTX 670 DirectX 11 GV-N670OC-2GD (I have confirmed that it has a BIOS at techpowerup(dot)com and wget and moved to /usr/share/kvm/Y.rom)
ASUS P8Z77-V LK

Things I have tried:
Moving the Graphics card from the 16x slot to the 8x slot (hence 02:00 instead of 01:00)
Changing the primary output to iGPU instead of dedicated in the bios
Not using a romfile
Installing then adding the hostpci0

I will start the virtual machine, and nothing will happen at all. I can no longer access console(which is to be expected) memory will jump up to 7Gb used then crash down and nothing happens until I stop it.
When I remove hostpci0 virtualization works fine.

Any assistance would be greatly appreciated I have been banging my head against a wall for three days.

 

[pabernethy]

You want to:

1. Make sure that VT-d is enabled in the UEFI and shows up in the OS. See here.
2. Make sure that the card or the downloaded ROM support UEFI. TS has more than one ROM for your model and not all support UEFI.
3. Make sure the card is bound to vfio-pci. You may need to blacklist the native driver.
4. Make sure your IOMMU groups are set up correctly or override, otherwise.

The wiki has some info on this topic, if you haven't been there already. The great ArchWiki has even more. Alex Williamson's blog (above) has lots of internals and many problems have already been solved in the VFIO mailing list.

 

[TheKingOfRainyLands]

You want to:

1. Make sure that VT-d is enabled in the UEFI and shows up in the OS. See here.
2. Make sure that the card or the downloaded ROM support UEFI. TS has more than one ROM for your model and not all support UEFI.
3. Make sure the card is bound to vfio-pci. You may need to blacklist the native driver.
4. Make sure your IOMMU groups are set up correctly or override, otherwise.

The wiki has some info on this topic, if you haven't been there already. The great ArchWiki has even more. Alex Williamson's blog (above) has lots of internals and many problems have already been solved in the VFIO mailing list.

1:
root@server:~# dmesg | grep Directed
[ 0.719267] DMAR: Intel(R) Virtualization Technology for Directed I/O

2:
This was a mistake I made, the rom I downloaded from TS did not support UEFI. I downloaded a new one.

3.
I believe this means that vfio-pci is the kernel driver in usage right? In which case my lspci -v shows that it is.

4.
I believe they are correct. 02:00 only has the Graphics part and the HD audio for HDMI which should be passed through with 02:00.

Still nothing seems to be working.

Thank you for the UEFI bios tip though, I didn't notice that. Figured a 2012 card is modern enough.

 

[TheKingOfRainyLands]

I ran apt-get install ovmf and for some reason it says ovmf wasn't installed.

That's probably not the problem then.

 

[TheKingOfRainyLands]

Using a script I found here are my IOMMU Groups.

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [8086:0150] (rev 09)
IOMMU Group 10 00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4)
IOMMU Group 11 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)
IOMMU Group 12 00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04)
IOMMU Group 12 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04)
IOMMU Group 12 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller [8086:1e22] (rev 04)
IOMMU Group 13 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 09)
IOMMU Group 14 07:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042]
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09)
IOMMU Group 1 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 670] [10de:1189] (rev a1)
IOMMU Group 1 02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0162] (rev 09)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 [8086:1e3a] (rev 04)
IOMMU Group 5 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)
IOMMU Group 6 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4)
IOMMU Group 8 00:1c.4 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 [8086:1e18] (rev c4)
IOMMU Group 9 00:1c.5 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev c4)
IOMMU Group 9 05:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 03)
IOMMU Group 9 06:00.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788]

The one in note is Group 1, which has the 670 that I want passed through.
Nothing that I tried editing my hostpci0 line to:

hostpci0: 00:01.0,00:01.1,02:00,pcie=1,x-vga=on,romfile=Y.rom

This allowed me to actually use the console and see into the VM finally, but no output on the GPU.... and when I tried installing Graphics drivers to see if the card was in.. it did not detect a graphics card.

 

[TheKingOfRainyLands]

Bump.

 

[pabernethy]

As Alex Williamson states in his blog, you need the card you want to pass through in a separate IOMMU group. So try putting the 670 back into the 16x slot (the card supports 16x, so I don't know why you tried that. The slowdown would be very noticeable.) If that puts it in its own group, great. If not you'll need to look into ACS overriding. The respective patches should be installed in the Proxmox kernel.

 

[dcsapak]

also one thing you could try is not to use uefi...

 

[Jeff H.]

Did you get this working? I have the exact same symptoms as you did: no console, memory jumps up for a bit and then back to normal with no other activity. I have to use forcefully "stop" the VM. I'm trying to pass through a GTX 1050.

 

[qonerybc]

For anyone who might still be stuck, here's how I enabled GPU passthrough using the GTX 670 on my system with UEFI.

1. Enable IOMMU by adding intel_iommu=on iommu=pt video=efifb:off to /etc/default/grub's GRUB_CMDLINE_LINUX_DEFAULT and running update-grub
2. Enable the VFIO kernel modules:

   cat > /etc/modules-load.d/vfio.conf << EOF
   vfio
   vfio_iommu_type1
   vfio_pci
   vfio_virqfd
   EOF

3. In my system (HP Z600), the IOMMU interrupt remapping has a bug and it's disabled, so I had to allow unsafe interrupts:

   echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

4. Configure the vfio-pci driver to bind to the graphics card before the nouveau driver does:

   cat > /etc/modprobe.d/vfio.conf << EOF
   softdep nouveau pre: vfio_pci
   options vfio-pci ids=10de:1189,10de:0e0a disable_vga=1 disable_idle_d3=1
   EOF

5. Ignore MSRS:

   echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

6. Update initramfs:

   update-initramfs -u

7. Reboot
8. Make sure your graphics card uses a BIOS compatible with UEFI. For my model (GV-N670OC-2GD), I had to flash version GV-N670OC-2GD/F5
   1. Note that there are BIOS versions with the same version number and ID, but with UEFI disabled. I thought my card's BIOS was up to date, but when I dumped it with nvflash I could verify UEFI was NOT enabled, despite being also version GV-N670OC-2GD/F5 80.04.31.00.58...
   2. WARNING: Be careful with this step, I bricked my card while flashing it and had to re-flash it using an SPI programmer
9. Create a patched BIOS using https://github.com/Marvo2011/NVIDIA-vBIOS-VFIO-Patcher from the ROM you downloaded in the previous step:

   python nvidia_vbios_vfio_patcher.py -i <ORIGINAL_ROM> -o <PATCHED_ROM>

   Copy this file to /usr/share/kvm/vbios_gtx670.bin, as we will use it later as the passthrough romfile
10. Add the PCI device to your VM (adjust the device number to your system):

    hostpci0: 0000:28:00,romfile=vbios_gtx670.bin,x-vga=1

    IMPORTANT: Enabling pcie=1 made my VM unable to boot, so it needs to be disabled (I was booting macOS, maybe this is different for another OS).
11. Disable the VM display:

    vga: none

12. Turn on your VM and cross your fingers!

References:

• https://pve.proxmox.com/wiki/Pci_passthrough
• https://bbs.archlinux.org/viewtopic.php?id=162768