Windows, GPU passthrough, and Code 43 Help needed!

Itay1778

Active Member
Apr 23, 2021
44
4
28
22
Hi everyone, I'm asking for help here after hours of trying and following guides, but nothing helps.
I'm trying to pass through to a GTX 1060 on an HP ML350 Gen9 - This installation of PVE is new, currently just for testing, so I know the hardware before I install this server fully
Proxmox 8.4.1 - Kernel 6.8.12-11-pve
After I pass through to a Windows VM, I install the NVIDIA driver. After the installation, everything looks fine, but I still don't have a display from the GPU, so I reboot the VM, and then the famous code 43 appears!
It always appears after a restart after installing the drivers, no matter if I choose x-vga=1 or not.

Things I tried *not exactly in this order*
Reinstalling Windows several times after changing settings in the host
Create a new VM
Change the processor model in the VM
Change the version of the Machine to 9.2, 7.1, 8.2
Try changing settings in vIOMMU
Try the PCI Device settings - with/without All Functions, With/without PCI-Express, With/without ROM-Bar
I also tried changing the OS type to Linux
I tried other drivers from NVIDIA's website - the newest and the oldest they have on the site
I tried adding a ROM file - I downloaded the match version from TechPowerUp
I tried changing the kernel host version to 6.5
I also tried configuring vGPU with the scripts and guides that are available online -
But there is always a problem installing the NVIDIA drivers on the host, so I didn't get too far with it
And nothing helped!! Still code 43 after restarting after installing the drivers

But! And this is very important
I tried to install in a new Ubuntu VM. After installation, I pass the GPU
Without checking the Primary GPU, the VM recognized the card
I installed the drivers
Still no screen
I shut down the VM, then checked the Primary GPU, started the VM, and suddenly I had a screen with Ubuntu and everything seemed to work properly!

So something is not working with Windows, and I can't figure out why!!

My current host settings

/etc/default/grub
Code:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""
Yes, I know that after kernel 6.5, "intel_iommu=on" isn't necessary, but I tried anyway.

/etc/modprobe.d/blacklist.conf
Code:
blacklist nouveau
blacklist nvidia


/etc/modprobe.d/iommu_unsafe_interrupts.conf
Code:
options vfio_iommu_type1 allow_unsafe_interrupts=1

/etc/modprobe.d/vfio.conf
Code:
options vfio-pci ids=10de:1c03,1458:371a disable_vga=1

/etc/modprobe.d/kvm.conf
Code:
options kvm ignore_msrs=1

VM config

Code:
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 8
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:0a:00,pcie=1,x-vga=1
machine: q35
memory: 8192
meta: creation-qemu=9.2.0,ctime=1748813413
name: Win-2
net0: virtio=BC:24:11:7E:71:20,bridge=vmbr0,link_down=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-102-disk-1,iothread=1,size=75G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=db4de64e-784c-4512-8e0f-a6d1ffef333e
sockets: 1
usb0: host=0c45:7603
vmgenid: 5706de68-56df-4b88-86c8-83dcea92c0ad
 
Same, I've spent 10+ hours tried with a NV GT 650 and a Amd FirePro W5000, same issue "reset glitch" on both

Did a fresh install too

Used these guides plus a lot of searching
https://gist.github.com/KasperSkytte/6a2d4e8c91b7117314bceec84c30016b

As far as I can figure gpu passthru is busted on 8.4 or proxmox doesn't like my Dell Precision T3610 :rolleyes:
 
I got it working, disabling my TPM (Secure Boot was off anyways) switched to a GTX 1060 (Maybe my W5000 and 650 will work now? don't wanna F it up tho)
(TPM issue might just be a me thing being its a 2013 Workstation, It's less glitchy /stuttery and doesn't freak out running headless now too)


~# nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
(needed? not needed?? enabled by default in Kernel 6.8??? who TF knows, I put it on.)

~# update-grub

~# nano /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

~# update-initramfs -u -k all


~# reboot

This from the proxmox wiki

Verify IOMMU is enabled​


Reboot, then run:


dmesg | grep -e DMAR -e IOMMU


There should be a line that looks like "DMAR: IOMMU enabled". If there is no output, something is wrong.


Verify IOMMU interrupt remapping is enabled​


It is not possible to use PCI passthrough without interrupt remapping. Device assignment will fail with 'Failed to assign device "[device name]": Operation not permitted' or 'Interrupt Remapping hardware not found, passing devices to unprivileged domains is insecure.'.

All systems using an Intel processor and chipset that have support for Intel Virtualization Technology for Directed I/O (VT-d), but do not have support for interrupt remapping will see such an error. Interrupt remapping support is provided in newer processors and chipsets (both AMD and Intel).

To identify if your system has support for interrupt remapping:


dmesg | grep 'remapping'

If you see one of the following lines:


  • AMD-Vi: Interrupt remapping enabled
  • DMAR-IR: Enabled IRQ remapping in x2apic mode ('x2apic' can be different on old CPUs, but should still work)

then remapping is supported.

If your system doesn't support interrupt remapping, you can allow unsafe interrupts with:


echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

Verify IOMMU isolation​


For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.

When executing


# pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""

replacing {nodename} with the name of your node.

You should get a list similar to:


┌──────────┬────────┬──────────────┬────────────┬────────┬───────────────────────────────────────────────────────────────────┬...
│ class │ device │ id │ iommugroup │ vendor │ device_name │
╞══════════╪════════╪══════════════╪════════════╪════════╪═══════════════════════════════════════════════════════════════════╪
│ 0x010601 │ 0xa282 │ 0000:00:17.0 │ 5 │ 0x8086 │ 200 Series PCH SATA controller [AHCI mode] │
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼
│ 0x010802 │ 0xa808 │ 0000:02:00.0 │ 12 │ 0x144d │ NVMe SSD Controller SM981/PM981/PM983 │
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼
│ 0x020000 │ 0x15b8 │ 0000:00:1f.6 │ 11 │ 0x8086 │ Ethernet Connection (2) I219-V │
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼
│ 0x030000 │ 0x5912 │ 0000:00:02.0 │ 2 │ 0x8086 │ HD Graphics 630 │
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼
│ 0x030000 │ 0x1d01 │ 0000:01:00.0 │ 1 │ 0x10de │ GP108 [GeForce GT 1030] │
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼


(Mine checked out, didn't need a work around)
  • NVIDIA GPUs
~# echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

~# echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf


~# reboot

Done!

Follow this video from (3:35) to set up gfx card in whatever VM you want.
https://www.youtube.com/watch?v=Il6HhOCfDjI