NVIDIA GPU Passthrough No Longer Working with Windows 10 Guest VM

[TorqueWrench]

Hi everyone,

I have successfully been passing my 1070Ti to a Windows 10 guest for over a year. However, with the latest Proxmox updates, my Windows 10 VM will no longer boot with the GPU passed through. Is anyone else experiencing the same issue? And if not, any suggestions on where I should go from here?

Technical Notes:

• With GPU passed through, and HDMI cable from GPU to a monitor, the BIOS screen appears just fine, but the screen goes black when Windows was supposed to boot (I know Windows isn't booting when this happens because the VM keeps restarting itself and eventually the physical display from the GPU will load the Windows troubleshooting screen).
• With GPU passed through, and on clean Windows 10 install, Windows loads just fine and even display on the monitor. It's only when a "real" driver, either from NVIDIA or from Windows Update is installed that Windows crashes (no green screen or BSOD, however). It will even crash during the install of the driver. This suggests to me that my Proxmox GPU passthrough configuration is correct, as I am successfully passing through my GPU initially, and that this is a Windows/NVIDIA driver issue. However, see below.

Things I Have Tried:

• I saw some sporadic comments about issues with Windows 10 May 2020 and GPU drivers. Thankfully, I had a backup from two weeks ago (when the VM was working), but no dice. This makes me believe this isn't a Windows/driver issue.
• Additionally, I have installed Windows 10 from scratch using both the latest Insider iso and from the latest stable ISO and then reinstalling the driver. No dice again.

Package Versions:

proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

I have cross-posted this from r/proxmox since it's likely a lot of the experts here do not frequent reddit.

Thanks in advance! I miss my virtual desktop!

-TorqueWrench

 

[TorqueWrench]

Additional update:

To pass through the NVIDIA GPU, I have been adding a profile. Removing the romfile allows me to boot the VM and access it via RDP. Originally, there was a code 43 error and installing the drivers from Nvidia removed it and shows a functional GPU. However, the output from the GPU on a physical monitor gives a black/no signal screen.

Respecifying the patched romfile once again results in a boot failure on the guest. Using an unpatched romfile results in a code 43.
It's interesting because, without the romfile, from the Windows perspective and looking at the GPU from device manager, it looks like I should have a fully functional 1070Ti, but the physical display output says otherwise.

 

[TorqueWrench]

Update: Figured it out!

I got everything working again. The issue appears to be due to Nvidia/Windows defaulting the GPU to line-based interrupts. Manually switching over the GPU to use message-based interrupts fixed the issue.

I will post a guide to help anyone else facing this.

Thank you for your help everyone,

TorqueWrench

 

[TorqueWrench]

Solution:

I believe many Proxmox users are likely impacted by this. For anyone impacted by this, I have written this short guide on how to fix a Windows 10 VM No Longer Boots on Proxmox with GPU Passthrough.

Obviously, there are a lot of things that can cause GPU passthrough to not work on a VM, but I think not having MSI-enabled devices ranks at the top, since this isn't the first time I've faced this. Even outside this particular issue, I think the section on Enabling MSI (Message Signaled Interrupts) on an Nvidia GPU in Windows 10 should be helpful to many, especially those setting up GPU passthrough on Proxmox for the first time.

Anyway, I hope this helps someone else,

TorqueWrench

 

[ferociousmilkyway]

Thank you so much for posting this! I had the same issue.

Is anyone aware of an underlying setting (or arg) that can mitigate this? I like to have MSI on anyways, but it gets reset on every driver install.

I saw some discussion of kernel_irqchip=on, but that doesn't resolve this for me.

 

[ferociousmilkyway]

Also worth noting, the title of the link below accurately describes a lingering problem, even if the above can mitigate it

"Proxmox 6.2 - nvidia gpu passthrough crashes after nvidia driver update"
https://forum.proxmox.com/threads/p...ugh-crashes-after-nvidia-driver-update.70531/

There is now no way to update drivers (for those of us affected) without the desktop crashing and a second PC / VM to remote in an apply the mitigate. This is because during a driver install, the MSI flag gets reset and the desktop crashes before you can do anything to prevent it.

 

[matthewpipes]

These links no longer work, what was the solution?

I believe many Proxmox users are likely impacted by this. For anyone impacted by this, I have written this short guide on how to fix a Windows 10 VM No Longer Boots on Proxmox with GPU Passthrough.

Obviously, there are a lot of things that can cause GPU passthrough to not work on a VM, but I think not having MSI-enabled devices ranks at the top, since this isn't the first time I've faced this. Even outside this particular issue, I think the section on Enabling MSI (Message Signaled Interrupts) on an Nvidia GPU in Windows 10 should be helpful to many, especially those setting up GPU passthrough on Proxmox for the first time.

Anyway, I hope this helps someone else,

TorqueWrench

 

[xiaopo]

These links no longer work, what was the solution?

have a try on this?

https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

Host lockup after virtual machine shutdown
This issue seems to primarily affect users running a Windows 10 guest and usually after the virtual machine has been run for a prolonged period of time: the host will experience multiple CPU core lockups (see [8]). To fix this try enabling Message Signal Interrupts on the GPU passed through to the guest. A good guide for how to do this can be found in [9]. You can also download this application for windows here [10] that should make the process easier.

 

[matthewpipes]

have a try on this?

https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

Host lockup after virtual machine shutdown
This issue seems to primarily affect users running a Windows 10 guest and usually after the virtual machine has been run for a prolonged period of time: the host will experience multiple CPU core lockups (see [8]). To fix this try enabling Message Signal Interrupts on the GPU passed through to the guest. A good guide for how to do this can be found in [9]. You can also download this application for windows here [10] that should make the process easier.

Thank you for the reply. I ended up figuring it out and enabling MSI with a tool I found online (https://www.mediafire.com/file/ewpy1p0rr132thk/MSI_util_v3.zip/file) and got it working but could not get audio passthrough the GPU. Bought an AMD card and still have the audio issue, tho video output works straight away

 

[RAJ-rios]

https://forum.proxmox.com/threads/windows-vm-gpu-passthrough-bootloop-bluescreen-fix.149054/