GPU Passthrough VM crashes when using PCIe mode

theleviwasbri

Member
Dec 14, 2019
5
0
6
31
I've been trying to get this to work for the past week with no luck. I've tried multiple different guides, but still can't seem to get this working.

I'll start with symptoms:
-Attaching GPU to VM with PCIe and Primary GPU results in windows blue screen.
-Attaching GPU to VM with PCIe and no Primary GPU vm boots then freezes after about 60 seconds of run time.
-Attaching GPU to VM without PCIe (i.e. regular PCI mode) vm seems to run fine aside from the fact I can't use the GPU in any meaningful way because I get a code 43. The primary GPU flag doesn't seem to have any effect on this.
-I've also tried all of the above with PopOS as a vm in case this was a windows issue, but I got all of the same symptoms.

The configuration:
HP z440
PM 7.2-11 with a zfs system mirror which boots to uefi
I've tried two different GPUs to determine that wasn't the issue (GTX1050ti and GTX970)
Xeon E5-1660v3
I'm running a radeon card for on-board graphics which is why radeon isn't in the blacklist.

VM setup (usb device is just a keyboard for troubleshooting purposes):

proxmox_vm.jpg

Current /etc/kernel/cmdline (but I've tried most settings you'll find with a google search):
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

/etc/modules:
Code:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/blacklist.conf:
Code:
blacklist nouveau
blacklist nvidia
blacklist snd_hda_intel
blacklist nvidiafb

lspci -v output: attached in file, but the card is in its own IOMMU group and the vfio driver is the only one loaded.

/etc/modprobe.d/vfio.conf:
Code:
options vfio-pci ids=10de:1c82,10de:0fb9,10de:13c2,10de:0fbb disable_vga=1


Like I said in the beginning I've been at this for about a week and I feel like i've been beating my head against a wall any help at all is greatly appreciated.
 

Attachments

  • pm_lspci_output.txt
    30.8 KB · Views: 5
UPDATE

I have the VM booting with the GPU attached in PCIe mode and with the primary GPU flag set.

things I changed:
rolled the kernel back to 5.15.53-1-pve according to this post
added the GPU romfile to the VM config

The VM will boot now, but I get no output on the monitor I have attached to the GPU and when I RDP into the machine and go to device manager I'm getting an error code 43 again. Even with the latest Nvidia driver installed.

But the VM is up and running with no sign of freezing, so that's progress I suppose.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!