Proxmox 6.2: Navi 5700xt GPU Passthrough to Win10 Guest

[cpukid00]

Hey guys! I'm pretty new into the proxmox game, but I'm hoping you can help me out with some GPU passthrough trouble I'm having.

First and foremost, I know Navi GPU's are kind of "problem children" when it comes to passing them through to guests, but it seems there are some success stories out there, so I'm really hoping I'm not SOL here.

So, my setup. I have an AMD Threadripper 2950X with an Asrock X399 Taichi motherboard. The setup runs bare-metal OS's fine with this GPU, so I'm confident my hardware itself is good/stable.

I've recently installed Proxmox 6.2, and am currently attempting something fairly simple; a Windows 10 guest with the 5700 passed through. In short, my problem is that once I install the AMD GPU drivers, the VM will boot and display output, but is so slow that it's un-usable. Before installing the GPU drivers, performance is totally fine (minus no graphics acceleration, of course).


I've gone ahead and created a VM with the following options:



Outside of the wizard, I've done the following:

I extracted the GPU rom myself using the following commands:

echo 1 > /sys/bus/pci/devices/0000:07:00.0/enable echo 1 > /sys/bus/pci/devices/0000:07:00.0/rom cat /sys/bus/pci/devices/0000:07:00.0/rom > 5700xt.rom echo 0 > /sys/bus/pci/devices/0000:07:00.0/rom echo 0 > /sys/bus/pci/devices/0000:07:00.0/enable


Based on searching around these forums before posting, I've also performed the following:



With this setup, I am able to get to the Windows installer on a monitor that's attached to the passed-through GPU. The install completes, I boot to the desktop, all is good (with low resolution, of course).

Now is where the trouble sets in. When I install the AMD GPU drivers, the system flickers as usual at around 60% into the driver install, and picture then returns, but performance comes to a screeching halt. The VM doesn't lock up; the mouse cursor will still move as you'd expect, but nothing else does. Every 60-90 seconds, some things on the screen will refresh, but not in a way that's usable.

Outside of what I've done here, I've also attempted to run the following, which didn't help, so I reverted:

qm set 101 -machine pc-q35-3.1


Any assistance that can be provided is super helpful. I've already sunk a good 4-5 hours (plus Googling time lol) into trying to get this going, and I'm getting to the point of throwing in the towel. It seems there's passthrough woes on either side of the fence, AMD or NVidia. I really love the 5700xt and would love to get it going in a guest OS.

Thank you for your time, and for the great product!

 

[Joris L.]

things to validate

* is the rom of the expected size and is it valid ?
* you may find this link useful https://www.techpowerup.com/vgabios...l=5700+XT&interface=&memType=&memSize=&since=

Installing the GPU drivers on proxmox will result in problems, do not do it.
The GPU drivers only need be installed in the guest.

 

[cpukid00]

Thanks for the response! I did try a rom from that site and I'm getting the same issues. Either the graphics driver not loading at all, or a black screen and/or awful performance. FWIW, I have the following card, and found the rom on that site that most closely matches the card:

https://www.amazon.com/dp/B082MQ4QTZ

For clarity, I didn't install any AMD drivers to the host proxmox OS. The only drivers I've attempted to install are within the Windows guest VM. Based on lspci -nnk output on the Proxmox host, the "kernel modules" returned is 'amdgpu', but the loaded driver is vfio, which I believe is necessary for the passthrough to work at all, regardless of whether or not the guest video drivers load. I didn't specifically install any AMD drivers to the Proxmox host OS. Here's my GPU output from lspci -nnk:




Thanks again for the help!

 

[jarekmor]

Shouldn't you set 'Display' in your VM config from 'Default' to 'None' if a GPU is used as Primary one?

 

[Joris L.]

Thanks for the response! I did try a rom from that site and I'm getting the same issues. Either the graphics driver not loading at all, or a black screen and/or awful performance. FWIW, I have the following card, and found the rom on that site that most closely matches the card:

https://www.amazon.com/dp/B082MQ4QTZ

For clarity, I didn't install any AMD drivers to the host proxmox OS. The only drivers I've attempted to install are within the Windows guest VM. Based on lspci -nnk output on the Proxmox host, the "kernel modules" returned is 'amdgpu', but the loaded driver is vfio, which I believe is necessary for the passthrough to work at all, regardless of whether or not the guest video drivers load. I didn't specifically install any AMD drivers to the Proxmox host OS. Here's my GPU output from lspci -nnk:


View attachment 20706

Thanks again for the help!

best way to understand what is going on is to enter the shell and stop start VM from there, ### must be replaced with VM-ID

qm stop ###
qm start ###

qm list

you must also configure like this

here is share a configuration that workd for me, the line is appended to what was in /etc/kernel/cmdline

iommu=pt amd_iommu=forced_isolation noats nobar ideo=efifb:off,vesafb:off hugepagesz=2M hugepages=1375 default_hugepagesz=1G hugepagesz=1G hugepages=8 text

do not forget to run pve-efiboot-tool refresh after changes to /etc/kernel/cmdline so it is active on reboot

/etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

aufs
overlay

/etc/modprobe.d/passtrough.conf

blacklist amdgpu

# GPU and AUDIO

options vfio-pci ids=1002:731f,1002:ab38 disable_vga=1 disable_idle_d3=1
options kvm ignore_msrs

 

[cpukid00]

Thanks for this! I did go through and verify all of these things were in place, and actually ended up finding the solution!

I had to add the following to my Windows 10 guest configuration:

qm set <VMID> -cpu "host,hidden=1,flags=+pcid"
qm set <VMID> -args '-cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off''


I'm honestly not sure if all of these were necessary to add, but I can certainly say that I am now passing through my XFX 5700xt flawlessly! Thank you so much for all of the help guys! Hopefully this will help someone else in the future who wants to pass through a Navi GPU.

Now if only AMD would fix this freakin' reset bug...

 

[Joris L.]

do you really need to add kvm=off ? that kind of kills all performance in the VM

 

[cpukid00]

do you really need to add kvm=off ? that kind of kills all performance in the VM

Honestly I'm not too sure at this point. I haven't done any in-depth testing (just haven't had the time, unfortunately). I only know that after adding those lines to the config, the VM was able to boot, both monitors came on, and performance (at a glance) seemed totally fine. Simple things like dragging windows across the screens was great, whereas before it wasn't even close to being usable. I haven't fired up a game or anything at this point.

I will certainly give removing that flag (and a few others) a try and report back! I'd love to get to a point where I only pass the flags that are absolutely needed.

 

[Joris L.]

Honestly I'm not too sure at this point. I haven't done any in-depth testing (just haven't had the time, unfortunately). I only know that after adding those lines to the config, the VM was able to boot, both monitors came on, and performance (at a glance) seemed totally fine. Simple things like dragging windows across the screens was great, whereas before it wasn't even close to being usable. I haven't fired up a game or anything at this point.

I will certainly give removing that flag (and a few others) a try and report back! I'd love to get to a point where I only pass the flags that are absolutely needed.

interesting, you actually have output sent to two physical screens ? I'm curious, can you share how you do that ?

 

[cpukid00]

I simply have 2 Displayport connections coming from the GPU out to 2 Dell displays. Since the whole GPU is claimed by vfio and passed through to the guest, I would assume this is expected behavior? (assuming there's a GPU driver installed in the guest, that is)

 

[Joris L.]

I simply have 2 Displayport connections coming from the GPU out to 2 Dell displays. Since the whole GPU is claimed by vfio and passed through to the guest, I would assume this is expected behavior? (assuming there's a GPU driver installed in the guest, that is)

no in my experience but i may have altered the outcome with making this more complicated, what does your /etc/kernel/cmdline look like ?

 

[cpukid00]

I actually don't have that file at all:

 

[Joris L.]

I actually don't have that file at all:

View attachment 20772

are you sure you're running proxmox ?
maybe you installed with grub ?

 

[cpukid00]

I am certainly running Proxmox lol. I do believe I am using Grub as the bootloader.



Here's a link to a video of my Windows guest booting up with both displays:

https://rochetechnology.com/prox/passthrough.mp4

I did try to remove the kvm=off flag from the VM, but that caused it to stop booting. So while I assume this isn't an ideal configuration, I fired up Minecraft for 2 minutes and was able to get 60fps at 1440p. I could possibly get more if I were to allocate more threads/memory to the guest.

 

[Joris L.]

I am certainly running Proxmox lol. I do believe I am using Grub as the bootloader.

View attachment 20773

Here's a link to a video of my Windows guest booting up with both displays:

https://rochetechnology.com/prox/passthrough.mp4

I did try to remove the kvm=off flag from the VM, but that caused it to stop booting. So while I assume this isn't an ideal configuration, I fired up Minecraft for 2 minutes and was able to get 60fps at 1440p. I could possibly get more if I were to allocate more threads/memory to the guest.

Impressive machine

i'm stumped as to why kvm=off works without performance penalty, things change, maybe it is not critical anymore in the way it used to be.

 

[cpukid00]

Thanks the ultimate goal is to have both MacOS and Windows virtual workstations, complete with snapshot ability for each VM. I’ve been super happy with the Threadripper platform.

I’m honestly not sure what the implications of kvm=off are. I’m admittedly pretty new to qemu/kvm/Proxmox, so I’m absolutely open to any advice or criticism for things that could/should be configured better/differently. I’ll be doing some googling later on for sure.

I really appreciate all the help!

 

[Joris L.]

Thanks the ultimate goal is to have both MacOS and Windows virtual workstations, complete with snapshot ability for each VM. I’ve been super happy with the Threadripper platform.

I’m honestly not sure what the implications of kvm=off are. I’m admittedly pretty new to qemu/kvm/Proxmox, so I’m absolutely open to any advice or criticism for things that could/should be configured better/differently. I’ll be doing some googling later on for sure.

I really appreciate all the help!

Well, proud to day i have all that and more (msw, osx, lnx) with a far more modest setup. I never get how all people have all the good stuff and I , as a professional, fail to find money for it

What do you mean with snapshotting ? This is built-in, no ?

 

[cpukid00]

What do you mean with snapshotting ? This is built-in, no ?

It's built into Proxmox for sure. I'm just saying that snapshotting is a key feature I love compared to physical/bare-metal installations. A VM just has so much more flexibility to it

 

[saberwolf]

It's built into Proxmox for sure. I'm just saying that snapshotting is a key feature I love compared to physical/bare-metal installations. A VM just has so much more flexibility to it

you could just do a parent and child disk for windows install and do a dynamic expand to the max hdd space avail this was an option in windows 7 to install windows in a vdisk it was not know or able to be setup out of the box you need to know a few diskpart commands but it is vary possible that is even on a bare metal machine you lose like 1-2% over all performance but backups and return to previous state took like 30s -2 mins to wipe child and create new one take a look into it you might like it

 

[nlegger]

Thanks the ultimate goal is to have both MacOS and Windows virtual workstations, complete with snapshot ability for each VM. I’ve been super happy with the Threadripper platform.

I’m honestly not sure what the implications of kvm=off are. I’m admittedly pretty new to qemu/kvm/Proxmox, so I’m absolutely open to any advice or criticism for things that could/should be configured better/differently. I’ll be doing some googling later on for sure.

I really appreciate all the help!

Same here, 3970x, Gigabyte Aorus Xtreme, my Proxmox 5700XT pass through is slow to a crawl vm at the moment, might need to revert a change.. 3min to open the Start menu then get device manager, when before I had a hard drive test at 15-25GB/s (4x1TB nvme zfs raid) Add in card.