Issues passing RX 7900 XT to Windows 11 VM on Proxmox 7.3

gagarin04

New Member
Jan 25, 2023
1
0
1
I've been trying to pass my RX 7900 XT to a Windows 11 VM on Proxmox 7.3. I only manage to get video output via HDMI for the first few seconds while it's booting (showing the Proxmox logo, OVMF?). Then it stops, and it doesn't show anything on subsequent boots. I had the same hardware before, but with an NVIDIA 3060 Ti GPU, and everything just worked. I've reinstalled Proxmox since replacing the GPU.

If I manage to RDP to the Windows 11 VM, the device manager shows the GPU as "not working properly", displaying error code 31.

My settings are as follows:

Bash:
# cat /etc/modules

vendor-reset
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Bash:
# cat /etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist amdgpu

Bash:
# cat /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt initcall_blacklist=sysfb_init vfio-pci.ids=1002:744c,1002:ab30,1002:7446,1002:7444 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1

Bash:
# cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:744c,1002:ab30,1002:7446,1002:7444 disable_vga=1

The VM configuration is as follows (including dumped VBIOS, see further down):

YAML:
# cat /etc/pve/qemu-server/100.conf

agent: 1
bios: ovmf
boot: order=ide0;ide2;net0
cores: 12
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:03:00,pcie=1,x-vga=1,romfile=vbios.bin
ide0: local-zfs:vm-100-disk-1,size=500G
ide2: local:iso/Win11_22H2_Swedish_x64v1.iso,media=cdrom,size=5287896K
machine: pc-q35-7.1
memory: 8192
meta: creation-qemu=7.1.0,ctime=1674594855
name: game
net0: e1000=4E:FC:79:5A:82:10,bridge=vmbr0,firewall=1
numa: 1
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=a2e2b53a-79ce-442a-adf1-a484d1838f18
sockets: 1
tpmstate0: local-zfs:vm-100-disk-2,size=4M,version=v2.0
vga: none
vmgenid: 83fbe391-c75a-4731-a977-7a748b233457

dmesg on the Proxmox host shows the following when starting the VM:

Code:
[   37.248250] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[   37.248583] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[   37.248591] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[   37.248597] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x26@0x410
[   37.248600] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x27@0x450

I've dumped the VBIOS from the GPU and it seems to have been successful:

Bash:
# ls -latr /usr/share/kvm/vbios.bin

-rw-r--r-- 1 root root 111104 Jan 24 22:05 /usr/share/kvm/vbios.bin

Bash:
# strings /usr/share/kvm/vbios.bin

IBMb
 761295520
11/08/22,17:23:39
113-D70401-00
NAVI31
PCI_EXPRESS
GDDR6
AMD D704_XT_A0_20GB_MBA                                                     
(C) 1988-2022, Advanced Micro Devices, Inc.
ATOMBIOSBK-AMD VER022.001.002.008.000001
NAVI31.bin 
00000000
00040781

lspci output regarding the GPU:

Code:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:744c] (rev cc)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1002]
        Kernel driver in use: vfio-pci
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7444]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: vfio-pci

What am I missing? I'm grateful for any help.
 
I've done passthru with my Nvidia Quadro P900 to my Plex VM using this walkthru: https://youtu.be/-HCzLhnNf-A?t=679

Relevant info starts at about the 11:19 mark.

On an initial glance, there's some grub config that has to happen on the proxmox hypervisor, probably more missing. While the end goal may not be the same, I'd imagine a good bit of the process may be similar.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!