Proxmox 9.2.2 GPU passthrough Error 43 Windows

[dropbrick]

Hi there,
Running Proxmox 9.2.2 here with kernel 7.0.2-6-pve (straight version non-updated from the 9.2-1 ISO installer). I've been also using 9.1.1 on other hypervisors too for a while.


My issue is with GPU passthrough(not vGPU) in Windows VMs. I have a RTX 6000, 6000 ADA or 5000 ADA and all have the same issue - I get the infamous error 43 in Windows. My hosts are DELL 7525s with AMD CPUs

First a big premise, this just works fine for Linux VMs, meaning the pass through BIOS/GRUB/IOMMU settings are correct otherwise the Linux VMs wouldn't start/work with GPU pass through from my understanding.


Have tried in the VM to set the usuals to make the nvidia driver happy.

root@proxmox:/etc/pve/qemu-server# cat 3849.conf
args: -cpu 'host,hv_vendor_id=null,kvm=off'
bios: ovmf
boot: order=ide0;ide2;net0
cores: 32
cpu: host,hidden=1
efidisk0: localraid:vm-3849-disk-0,efitype=4m,ms-cert=2023k,pre-enrolled-keys=1,size=1M
hostpci0: 0000:81:00,pcie=1,x-vga=1
ide0: localraid:vm-3849-disk-1,discard=on,size=500G,ssd=1
machine: pc-q35-10.1
memory: 65535
meta: creation-qemu=10.1.2,ctime=1786350381
name: windows11
net0: e1000e=BC:24:11:FC:18:A7,bridge=vmbr1,tag=2
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=3578e213-32dc-4055-8fb6-76535096df1d
sockets: 1
tpmstate0: localraid:vm-3849-disk-2,size=4M,version=v2.0
vmgenid: 42eae470-add3-4fea-9590-4fe1d3577401




root@proxmox:~# dmesg | grep MMU
[ 3.611262] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.613645] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.615458] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.618140] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.622413] pci 0000:e0:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.624866] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.627183] pci 0000:a0:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.629009] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported
[ 3.641469] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 3.641591] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[ 3.641720] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[ 3.641860] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[ 3.642004] perf/amd_iommu: Detected AMD IOMMU #4 (2 banks, 4 counters/bank).
[ 3.642133] perf/amd_iommu: Detected AMD IOMMU #5 (2 banks, 4 counters/bank).
[ 3.642278] perf/amd_iommu: Detected AMD IOMMU #6 (2 banks, 4 counters/bank).
[ 3.642393] perf/amd_iommu: Detected AMD IOMMU #7 (2 banks, 4 counters/bank).


root@proxmox:~# cat /etc/default/grub
......
GRUB_CMDLINE_LINUX_DEFAULT="nomodeset amd_iommu=on iommu=pt"



Anything else to try - known issues with Windows ? This worked just fine in vmware... on the same hosts. and again Linux is just as happy but unfortunately I need also Windows...


Thanks in advance for any pointers.

 

[emunt6]

If it works under Linux VMs, it should work under Windows.
- Can you tell me how many GPU in your system?

( Make sure the passthrough GPU did not get initialized by BIOS/UEFI when you boot the Proxmox host - thats why you need at least 2xGPU:
-1ea GPU primary for proxmox host
-1ea GPU for passthrough

Moreover, you need to add all the "GPUs subhardware" from the PCI-E raw devices list:
- 00.0 - GPU
- 00.1 - GPU Audio port

Example:
hostpci0: 0000:27:00,pcie=1,x-vga=1
hostpci1: 0000:27:00,pcie=1

 

[dropbrick]

Yes there's VGA onboard for Proxmox - it's a DELL 7525 - GPUs are passed using All functions, so sidecar audio device gets in there too. vGPU works fine as well if used, but whole card passthrough has the issue. Presume is something with the Nvidia driver detecting some VM stuff but unclear. Is anyone doing anything specific running Windows VMs with Nvidia RTXs on AMD CPU hosts ?