Issues with GPU passthrough Threadripper and AMD GPU

[iPenguin]

I'm new to Proxmox and still learning. I followed some guides to do GPU passthrough, but I'm still having issues. My 6700 XT is detected within my Windows 10 VM and shows no issues in Device Manager. But when I try to install the AMD Driver, the VM stops responding. When it responds again, I'm at the login screen. Thanks in advance!

My CPU is the Threadripper 3955WX which I'm running on the Gigabyte WRX80-SU8-IPMI and 6700 XT. I have a ZFS pool for my Proxmox drive and a separate pool for my VMs. I have Re-sizeable Bar turned off in BIOS.

Here is the VM setup, I did not touch the .conf file


pveversion -v

proxmox-ve: 7.4-1 (running kernel: 5.15.107-2-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-3
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.107-1-pve: 5.15.107-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-1
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.6
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.0
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-2
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

qm config 120

balloon: 0
bios: ovmf
boot: order=scsi0;net0;ide0
cores: 8
cpu: host
efidisk0: VMs:120/vm-120-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:87:00,pcie=1,x-vga=1
ide0: local:iso/virtio-win-0.1.225.iso,media=cdrom,size=519590K
machine: pc-q35-7.2
memory: 16384
meta: creation-qemu=7.2.0,ctime=1684875426
name: GamingVM
net0: virtio=E6:7C:03:E9:F9:3B,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: VMs:120/vm-120-disk-1.qcow2,size=120G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=1c371330-d730-4cd2-aa23-261370874297
sockets: 1
tpmstate0: VMs:120/vm-120-disk-2.raw,size=4M,version=v2.0
usb0: host=8-1.1
usb1: host=8-1.2
usb2: host=8-1.3
usb3: host=8-1.4
vga: none

I read that because I'm using a ZFS pool I will be on systemmd
/etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on

/etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/iommu_unsafe_interrupts.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

lspci

87:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 (rev c1)
87:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab28

lspci -n -s 87:00

87:00.0 0300: 1002:73df (rev c1)
87:00.1 0403: 1002:ab28

/etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:73df,1002:ab28 disable_vga=1

/etc/modprobe.d/blacklist.conf

blacklist amdgpu
blacklist radeon
blacklist nouveau
blacklist nvidia

 

[dcsapak]

I'm new to Proxmox and still learning. I followed some guides to do GPU passthrough, but I'm still having issues. My 6700 XT is detected within my Windows 10 VM and shows no issues in Device Manager. But when I try to install the AMD Driver, the VM stops responding. When it responds again, I'm at the login screen. Thanks in advance!

any logs on the host in the journal/syslog or in the guest (event viewer? ) ?

 

[iPenguin]

any logs on the host in the journal/syslog or in the guest (event viewer? ) ?

Sorry, I attached the Syslog from the time I started the VM to when it crashes when I attempt to install the AMD Driver. This time it didn't come back up. I had to stop/start it.

Here are the only errors I see in Event Viewer

 

[dcsapak]

can you also post your pveversion -v and the vm config ? (qm config ID)

 

[iPenguin]

can you also post your pveversion -v and the vm config ? (qm config ID)

Thank you for providing the commands. I updated the original post.

 

[dcsapak]

can you try without secure boot in the vm? you have to press esc during the uefi loading screen, and then go to 'device manager' -> 'secure boot configuration' -> disable it

 

[iPenguin]

can you try without secure boot in the vm? you have to press esc during the uefi loading screen, and then go to 'device manager' -> 'secure boot configuration' -> disable it

Same issue, freezes when I attempt to install the driver.

I verified Secure Boot is disabled by going back into the UEFI. No errors in device Manager

Here is the syslog if needed.

 

[dcsapak]

ok, nothing special in the syslog either.


could you try with a linux vm? just to see if the problem is with the driver or the passthrough itself?
if it works in a linux vm, my guess would be a problem with the windows driver...

edit: is your host booting in uefi or bios mode?

 

[iPenguin]

ok, nothing special in the syslog either.


could you try with a linux vm? just to see if the problem is with the driver or the passthrough itself?
if it works in a linux vm, my guess would be a problem with the windows driver...

edit: is your host booting in uefi or bios mode?

I can try Linux. Are there any guides / distros you recommend? I'm not super familiar with Linux, but i'm open to learning/trying.

UEFI with Secure Boot. I tried booting UEFI without Secure boot.

Not sure if this matters
Above 4G Decoding = Enabled
Re-size BAR = Disabled
SR-IOV Support = Enabled
BME DMA Mitigation = Disabled
Hot-Plug Support = Enabled

 

[dcsapak]

I can try Linux. Are there any guides / distros you recommend? I'm not super familiar with Linux, but i'm open to learning/trying.

i'd probably just use ubuntu desktop, should be newcomer friendly enough

 

[iPenguin]

i'd probably just use ubuntu desktop, should be newcomer friendly enough

Thank you for your help so far. I enabled the default display and noticed the error is a BSOD VIDEO DXGKRNL FATAL ERROR. I'm googling to see how others have resolved this error.

I was able to install the AMD graphics driver using the full version rather than the auto detect one, but that didn't resolve the error.

Using i440fx v7.2 works without crashing. I read that q35 is more desired. I'm trying different versions to see if they crash.

 

[ZoThyx]

Above 4G Decoding = Enabled

You can try to disable this. I new to GPU Passthrough too but I read that can resolve issues