GPU Passthrough on 7.2 - Stopped working since upgrade

[Astraea]

I have commented in a few threads about the issues with GPU passthrough but wanted to start my own thread and the main one I was commenting on has been marked a solved. Since I upgraded my install to 7.2 I cannot get my GPU passthrough working. I can get the GPU to show in the VM but always get an ERROR 43. My CPU is an i7-960 and my GPU is a 1060 and was working flawlessly on 7.1

So far I have tried the following:

• The script located in this thread: https://forum.proxmox.com/threads/gpu-passthrough-issues-after-upgrade-to-7-2.109051/post-470236
• The latest testing kernel
• Rolling back to the 5.13 kernel
• The various suggestions/modifications regarding adding or changing parameters in GRUB
• I have also created a brand new VM and installed windows again and used the 215 (217 didn't install) version of the VirtIO drivers and the latest Nvidia drivers

My next move is to backup the original VM and reinstall Proxmox using the latest 7.2 ISO and see if that works. Failing that I'll try the 7.1 ISO and find a way to update all the packages to the latest versions before 7.2 (though I am not sure how to do this yet).

 

[Astraea]

I have now rebuilt my Proxmox Install using the latest 7.2 ISO and still have the same problem where the GPU is detected inside the VM but windows shows error 43 still. I did use the same VM that I backed up as in my previous install of Proxmox installing windows again from a new VM resulted in the same outcome so I do not think it is the VM that is the issue but something with the changes from 7.1 to 7.2.

I am now trying to install 7.1 and see if I can get it working under that version of Proxmox, Ideally, I'll be able to update to the latest 7.1 and not move up to 7.2 if it does work. I report back my findings once I have 7.1 installed again.

 

[Astraea]

I installed 7.1 and could not get the VM to start the error I got was: TASK ERROR: start failed: QEMU exited with code 1

Updating the Proxmox install to 7.2 allowed the VM to start but still the same outcome of the GPU showing Error 43. Even after uninstalling it in Windows and rescanning or disabling and reenabling the GPU gave the same Error 43 result.

I have tried the various other methods used in other threads but cannot seem to get the GPU working again.

 

[TorqueWrench]

I installed 7.1 and could not get the VM to start the error I got was: TASK ERROR: start failed: QEMU exited with code 1

Updating the Proxmox install to 7.2 allowed the VM to start but still the same outcome of the GPU showing Error 43. Even after uninstalling it in Windows and rescanning or disabling and reenabling the GPU gave the same Error 43 result.

I have tried the various other methods used in other threads but cannot seem to get the GPU working again.

I usually associate code 43 error with Nvidia not liking PCI passthrough on consumer GPUs. The fix for this being patched ROM. (Though I got the impression this is no longer needed- can anyone comment on this/prove me wrong? I'd love to know.)

The last time I had an issue with GPU passthrough on a Windows guest, it was because the Device Instance Path (found within the Windows VM) had changed and needed to be updated. You can check out what I had to go through to fix that here:

Fixing GPU Passthrough On Windows VM In Proxmox​

Maybe a similar issue going on here?

 

[nick.kopas]

Can you dump your relevant configs? I run UEFI and ZFS, so mine might be a little different.

#!/bin/bash
for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b33] (rev 05)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2489] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
IOMMU Group 2 00:02.0 Display controller [0380]: Intel Corporation CometLake-S GT2 [UHD Graphics 630] [8086:9bc5] (rev 05)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller [8086:06ed]
IOMMU Group 3 00:14.2 RAM memory [0500]: Intel Corporation Comet Lake PCH Shared SRAM [8086:06ef]
IOMMU Group 4 00:15.0 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH Serial IO I2C Controller #0 [8086:06e8]
IOMMU Group 4 00:15.1 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH Serial IO I2C Controller #1 [8086:06e9]
IOMMU Group 5 00:16.0 Communication controller [0780]: Intel Corporation Comet Lake HECI Controller [8086:06e0]
IOMMU Group 6 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:06d2]
IOMMU Group 7 00:1b.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #17 [8086:06c0] (rev f0)
IOMMU Group 8 00:1b.4 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #21 [8086:06ac] (rev f0)
IOMMU Group 9 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:06b8] (rev f0)
IOMMU Group 10 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:06bc] (rev f0)
IOMMU Group 11 00:1c.5 PCI bridge [0604]: Intel Corporation Device [8086:06bd] (rev f0)
IOMMU Group 12 00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:06be] (rev f0)
IOMMU Group 13 00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:06bf] (rev f0)
IOMMU Group 14 00:1d.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #9 [8086:06b0] (rev f0)
IOMMU Group 15 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:0685]
IOMMU Group 15 00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH SMBus Controller [8086:06a3]
IOMMU Group 15 00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH SPI Controller [8086:06a4]
IOMMU Group 16 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 17 05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 02)
IOMMU Group 18 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 19 07:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
IOMMU Group 20 08:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
IOMMU Group 21 09:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)

root@pve:~# cat /proc/cmdline
initrd=\EFI\proxmox\5.15.35-1-pve\initrd.img-5.15.35-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt video=efifb:off video=simplefb:off video=vesafb:off

root@pve:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

root@pve:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2489,10de:228b,1987:5012,1912:0014,1b21:0612 disable_vga=1

root@pve:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

root@pve:~# cat /etc/pve/qemu-server/100.conf
agent: 1
balloon: 0
bios: ovmf
boot: order=hostpci0
cores: 8
cpu: host
efidisk0: local-zfs:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
hookscript: local:snippets/gpu-hookscript.sh
hostpci0: 0000:09:00,pcie=1
hostpci1: 0000:01:00,pcie=1,x-vga=1,romfile=08G-P5-3663-KL.rom
hostpci2: 0000:08:00,pcie=1
hostpci3: 0000:07:00,pcie=1
hostpci4: 0000:06:00,pcie=1
machine: pc-q35-6.2
memory: 16384
meta: creation-qemu=6.1.1,ctime=1648601835
name: raven
net0: virtio=5E:42:11:7E:16:FA,bridge=vmbr0
numa: 0
onboot: 1
ostype: win11
scsihw: virtio-scsi-pci
smbios1: uuid=5d6ae88a-76b6-4ab0-8aac-9ae36dcb518c
sockets: 1
tpmstate0: local-zfs:vm-100-disk-0,size=4M,version=v2.0
vmgenid: f3f742fa-3520-452f-98d6-1c6c5fbb2fb5

Some things to note about my configuration:

• I passthrough an NVME drive, 3060ti, 2x USB controllers and a SATA controller.
• I'm running Windows 11.
• I'm running virtio-win-0.1.215 drivers. (There's a known bug in 0.1.217.)
• I'm running 512.15 NVIDIA drivers.
• My setup works with or without the romfile.

I know NVIDIA enabled virtualization support for everything Kelper and newer starting with the 465.89 drivers.

 

[punjprateek]

You need a dummy hdmi dongle to resolve code 43, as consumer card does not work headless

 

[Astraea]

I have a monitor connected to the system and a physical mouse and keyboard are connected. I will try and post some configuration information later tonight or this week when I get a chance to get back to working on this machine.

 

[Astraea]

@nick.kopas I went through my configurations and made it so my Proxmox configurations matched yours and I still was unable to get the GPU working. in fact, the VM stopped booting once I made the changes to cmdline, vifo and blacklist configurations. I also tried before making those edits to build another new VM to see if there was any difference but still only got the Error 43 on the GPU but the network card I pass through was fine. Not sure if it is just my old hardware and the new methods used in 7.2 or what the issue is.

 

[nick.kopas]

@nick.kopas I went through my configurations and made it so my Proxmox configurations matched yours and I still was unable to get the GPU working. in fact, the VM stopped booting once I made the changes to cmdline, vifo and blacklist configurations. I also tried before making those edits to build another new VM to see if there was any difference but still only got the Error 43 on the GPU but the network card I pass through was fine. Not sure if it is just my old hardware and the new methods used in 7.2 or what the issue is.

Shoot, I was hoping to see the configurations you were running.

So, even after rebuilding on 7.1 you still weren't able to get things working like they were previously? Do you have any of those configs backed up?

 

[Astraea]

Nope, nothing seemed to get passed the Error 43. I don't have the Proxmox host backed up but I do have the VM backed up so I can hopefully pull off some data from it. I was using it only for my gaming machine to help with windows update mess-ups. My configuration for the cmdline line was:

intel_iommu=on

and then I changed it to match the end of yours so it was:

intel_iommu=on iommu=pt video=efifb:off video=simplefb:off video=vesafb:off

at the end before I blew the install away. My blacklist.conf was originally not even created but I matched it to yours as the last try and the same with the vfio configuration file. My modules file was always identical.

My VM was getting 2 things passed though 1 was the GPU and the other was a network card. The network card was on group 3 and the GPU was on group 4. and they were both set as PCI-e and ROM in abled but no custom file was set and they were both set to all functions and the GPU was set to primary.

 

[nick.kopas]

Nope, nothing seemed to get passed the Error 43. I don't have the Proxmox host backed up but I do have the VM backed up so I can hopefully pull off some data from it. I was using it only for my gaming machine to help with windows update mess-ups. My configuration for the cmdline line was:

intel_iommu=on

and then I changed it to match the end of yours so it was:

intel_iommu=on iommu=pt video=efifb:off video=simplefb:off video=vesafb:off

at the end before I blew the install away. My blacklist.conf was originally not even created but I matched it to yours as the last try and the same with the vfio configuration file. My modules file was always identical.

My VM was getting 2 things passed though 1 was the GPU and the other was a network card. The network card was on group 3 and the GPU was on group 4. and they were both set as PCI-e and ROM in abled but no custom file was set and they were both set to all functions and the GPU was set to primary.

Can you dump a couple things:

• The output from that IOMMU script I posted above.
• The output from cat /proc/iomem after a fresh boot.

 

[Astraea]

Unfortunately, I cannot without re-installing Proxmox. After spending a large amount of time and frustration on this I wiped Proxmox for now and just installed windows directly onto the system so I can play games again. I might revisit this again another time but for now, I needed to get the system/GPU running. I have a newer desktop that is running with Proxmox and has 2 VMs for my work workstations with one VM being windows and the other Linux which is still running 7.1 and I might upgrade that one once I backup the OS disk and the VMs in case it goes sideways again.

 

[nick.kopas]

Unfortunately, I cannot without re-installing Proxmox. After spending a large amount of time and frustration on this I wiped Proxmox for now and just installed windows directly onto the system so I can play games again. I might revisit this again another time but for now, I needed to get the system/GPU running. I have a newer desktop that is running with Proxmox and has 2 VMs for my work workstations with one VM being windows and the other Linux which is still running 7.1 and I might upgrade that one once I backup the OS disk and the VMs in case it goes sideways again.

game on friend...