Unable to passthrough RTX 6000 Ada

[symysak]

Hi, I am building a server with the following configuration

i7-14700KF
128GB RAM
NVIDIA RTX 6000 Ada
ASUS PRIME Z790

I want to do a gpu passthrough to vm (not vGPU), but I get error code 12 in Windows VM.
Does anyone know how to solve this?
Is gpu pass-through (NOT vGPU) not possible on enterprise GPUs?

What I have tried is as follows

1. Configure GPU passthrough as follows
   https://forum.proxmox.com/threads/p...x-ve-8-installation-and-configuration.130218/
2. Put the RTX 6000 Ada in compute mode with displaymodeselector
   https://forum.proxmox.com/threads/pci-passthrough-not-working-with-proxmox8-and-rtx6000-ada.138848/

3. qm set 100 --args '-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536 -global q35-pcihost.pci-hole64-size=2048G'

   https://www.tsunderechen.io/2021/11/OVMF-PCIE-passthrough-with-large-VRAM-GPU/
   https://forum.proxmox.com/threads/gpu-passthrough-code-12.110407/post-504040
4. IOMMU, Resize Bar, Above 4G Decoding, SR-IOV are enabled.

root@pve:~# pveversion
pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.8.4-2-pve)

root@pve:~# qm config 100
args: -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536
balloon: 0
bios: ovmf
boot: order=scsi0;ide2;net0;ide0
cores: 8
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1
ide0: local:iso/virtio-win.iso,media=cdrom,size=612812K
ide2: local:iso/windows11.iso,media=cdrom,size=6561616K
machine: pc-q35-8.1
memory: 8192
meta: creation-qemu=8.1.5,ctime=1717567965
name: windows11
net0: virtio=BC:24:11:2F:73:26,bridge=vmbr0
numa: 0
ostype: win11
scsi0: local-lvm:vm-100-disk-1,iothread=1,size=256G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=808887e3-4b20-491b-b748-06bf112ed02c
sockets: 1
tpmstate0: local-lvm:vm-100-disk-2,size=4M,version=v2.0
vmgenid: 36dd65f0-4414-40fe-aa06-4733ad057fdd

root@pve:~# dmesg | grep "DMAR"
[    0.004032] ACPI: DMAR 0x000000007633C000 000050 (v01 INTEL  EDK2     00000002      01000013)
[    0.004054] ACPI: Reserving DMAR table memory at [mem 0x7633c000-0x7633c04f]
[    0.128610] DMAR: IOMMU enabled
[    0.286961] DMAR: Host address width 39
[    0.286962] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.286968] DMAR: dmar0: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.286970] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 0
[    0.286971] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.286971] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.288400] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.482046] DMAR: No RMRR found
[    0.482046] DMAR: No ATSR found
[    0.482047] DMAR: No SATC found
[    0.482048] DMAR: dmar0: Using Queued invalidation
[    0.482339] DMAR: Intel(R) Virtualization Technology for Directed I/O
root@pve:~# dmesg | grep "IOMMU"
[    0.128610] DMAR: IOMMU enabled
[    0.286970] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 0
root@pve:~# dmesg | grep "vfio"
[  118.248404] vfio_pci: add [10de:26b1[ffffffff:ffffffff]] class 0x000000/00000000
[  118.295996] vfio-pci 0000:01:00.0: Enabling HDA controller
[  119.557339] vfio-pci 0000:01:00.0: Enabling HDA controller
[  119.557358] vfio-pci 0000:01:00.0: enabling device (0000 -> 0002)
[  119.695648] vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs
[  120.319668] vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs
[  120.335904] vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs
[  120.348073] vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs

 

[dcsapak]

1. IOMMU, Resize Bar, Above 4G Decoding, SR-IOV are enabled.

did you try to disable resizable bar? this is not yet supported in qemu and can lead to weird guest driver errors when it sees no resizable bar, but the hardware sees it

 

[symysak]

did you try to disable resizable bar? this is not yet supported in qemu and can lead to weird guest driver errors when it sees no resizable bar, but the hardware sees it

I tried disabling it, but that didn't change the situation...

 

[dcsapak]

did you also try to uncheck the 'pcie' checkbox? also with an extracted rom maybe?

gpu passthrough (especially on consumer hardware) is always a bit hit & miss (as you can see from the linked forum thread, where the solution for one person does not fix it for others)

what does it say on the 'details' tab of the driver window?

 

[symysak]

I had the same situation without the pcie checkbox.

I will try the following for romfile.
https://pve.proxmox.com/wiki/PCI_Passthrough#The_'romfile'_option

The driver window is as follows