[TUTORIAL] Simple Working GPU Passthrough on UpToDate PVE and AMD Hardware

[w.reimer]

- CSM (Boot) has to be enabled. Otherwise you will suffer the reset bug.

Should someone have boot up problems with his system, try to disable this setting. I don't now why, but since I'd done a bios upgrade to the newest version I need to have this setting disabled, instead of enabled for even getting the system to boot. But it works now just as fine as before.

Sadly I can't edit my first post anymore to make this remark on top

 

[MoxProxNoob]

First off, many thanks for the write up. I was completely stuck before seeing this.

Has anyone tried this with a Win 10 guest?

I have a working win10 VM with virtio drivers and qemu agent installed.

All works fine with default vga display.

TechPowerUp ROM.

When I switch to PCI device and display none, the vm fails to start and I get a dbus no reply error.

If I switch the display back to none it works fine again.

Also, I am running single box with xfce4 and lightdm as it’s my workstation not a server.

Am I missing something. I’m assuming this should work on 10 as it does on 11, right?

Do I need an extra gpu as the Ryzen doesn’t have integrated graphics?

Hardware
X570-P Prime
Ryzen 9 3900x
Palit 4070Ti Gaming Pro

 

[w.reimer]

First off, many thanks for the write up. I was completely stuck before seeing this.

Has anyone tried this with a Win 10 guest?

I have a working win10 VM with virtio drivers and qemu agent installed.

All works fine with default vga display.

TechPowerUp ROM.

When I switch to PCI device and display none, the vm fails to start and I get a dbus no reply error.

If I switch the display back to none it works fine again.

Also, I am running single box with xfce4 and lightdm as it’s my workstation not a server.

Am I missing something. I’m assuming this should work on 10 as it does on 11, right?

Hardware
X570-P Prime
Ryzen 9 3900x
Palit 4070Ti Gaming Pro

First of all: You have an nVidia GPU and a different Generation of CPU/Chipset. My Experience with Proxmox and GPU Passthrough is that it's at this time basically a hit or miss with working combinations of hardware. So while it runs with little work with my hardware, it would be possible that it won't work with your config. But hey, we can try

Windows 10 or 11 shouldn't be the problem, it's working for me with a dozen different systems.

1. Could you post your full vm config file?
2. Did you try both CSM enabled/disabled? Were there any differences?
3. The BIOS file from TechPowerUp is specific for you're GPU Modell (Palit 4070Ti Gaming Pro)?
4. Did you make the other settings I recommended in the first post?

 

[MoxProxNoob]

First of all: You have an nVidia GPU and a different Generation of CPU/Chipset. My Experience with Proxmox and GPU Passthrough is that it's at this time basically a hit or miss with working combinations of hardware. So while it runs with little work with my hardware, it would be possible that it won't work with your config. But hey, we can try

Windows 10 or 11 shouldn't be the problem, it's working for me with a dozen different systems.

1. Could you post your full vm config file?
2. Did you try both CSM enabled/disabled? Were there any differences?
3. The BIOS file from TechPowerUp is specific for you're GPU Modell (Palit 4070Ti Gaming Pro)?
4. Did you make the other settings I recommended in the first post?

Thanks for the speedy reply

1. I’ll post the config tomorrow once I return home.

2. I didn’t try disabled yet but the host boots fine on enabled.

3. The ROM file is the exact one.

4. I started again with a fresh Proxmox install and only did as you described.

After some research I believe the issue is my CPU. 3900X doesn’t have an iGPU.

So everything works up to attempting to boot the VM. My guess is the VM can’t access the GPU because the host can’t/won’t/quite obviously wouldn’t relinquish it.

Theoretically it would work with a headless proxmox server. But I’m running it as a single box workstation with ONLY ONE GPU.

[SLAPS SELF]

I checked the price of a 7800X3D, new X chipset mobo and RAM. This will have to wait.

My baby is still going strong. Lol. I’d say she’s got another 5 years left in her yet. Maybe 7 if I buy a used 5950X.

In reality, I’m gonna have to scrap this idea for now.

My options now are dual boot Windows 10 and Linux, or single boot Windows 11 and run Linux in GPU-enabled docker containers. Which is annoying. Lol.

I don’t even want windows full stop, much less windows 11. But… UE5 and the odd game here or there demand it.

I’m gonna go wash off the stench of failure and disappointment in an alcohol bath and decide tomorrow which of these two undesirable paths I’m gonna take.

Thanks for your help tho.

See you all once I replace the guts in my PC. Lol. (I’ll spend the money prematurely eventually. Just not now)

EDIT: To add insult to injury I have a 1060 sitting gathering dust but because I skimped on the mobo I only have one x16 slot. Lol. Moral of the story, motherboards cost A LOT of money and there’s no way around it. Spend a fortune on one or you will cry big fat baby loser tears somewhere down the road. For now, I cry.

 

[w.reimer]

There are some config settings (GRUB, Modules etc) you could still try, if you search the web. But it would cost you some time to try them out, you would need a few hours at least without a guarantee that it will work.

Should you decide to buy new hardware in the feature for a working passthrough system, I would recommend to buy the exact same configuration I have. (At least the exact same motherboard and CPU and GPU from the same generation).

For me it's awesome. My work/production system is fedora linux, the gaming part is windows 11. I can make snapshots/backups at critical times or if I want to try something out, don't have to reinstall/reactivate windows. I love trying different linux systems/DE out and can run everything native. And performance loss while gaming is around 1% compared to native. (At least in my tests).

You do lose some features compared to native, like standby/hibernate, direct access to monitoring your chipset/cpu etc, and the boot up time is like 15 seconds longer. But that's not a problem for me.

But do post you're vm config, maybe there's something you've missed, who knows

 

[jeenam]

Thanks for the speedy reply

1. I’ll post the config tomorrow once I return home.

2. I didn’t try disabled yet but the host boots fine on enabled.

3. The ROM file is the exact one.

4. I started again with a fresh Proxmox install and only did as you described.

After some research I believe the issue is my CPU. 3900X doesn’t have an iGPU.

So everything works up to attempting to boot the VM. My guess is the VM can’t access the GPU because the host can’t/won’t/quite obviously wouldn’t relinquish it.

Theoretically it would work with a headless proxmox server. But I’m running it as a single box workstation with ONLY ONE GPU.

[SLAPS SELF]

I checked the price of a 7800X3D, new X chipset mobo and RAM. This will have to wait.

My baby is still going strong. Lol. I’d say she’s got another 5 years left in her yet. Maybe 7 if I buy a used 5950X.

In reality, I’m gonna have to scrap this idea for now.

My options now are dual boot Windows 10 and Linux, or single boot Windows 11 and run Linux in GPU-enabled docker containers. Which is annoying. Lol.

I don’t even want windows full stop, much less windows 11. But… UE5 and the odd game here or there demand it.

I’m gonna go wash off the stench of failure and disappointment in an alcohol bath and decide tomorrow which of these two undesirable paths I’m gonna take.

Thanks for your help tho.

See you all once I replace the guts in my PC. Lol. (I’ll spend the money prematurely eventually. Just not now)

EDIT: To add insult to injury I have a 1060 sitting gathering dust but because I skimped on the mobo I only have one x16 slot. Lol. Moral of the story, motherboards cost A LOT of money and there’s no way around it. Spend a fortune on one or you will cry big fat baby loser tears somewhere down the road. For now, I cry.

iGPU has nothing to do with trying to get a dGPU working for GPU passthrough. I've set up passthrough on 5+ systems and been successful every single time. I would be very surprised if the problem you're encountering is NOT because of misconfiguration on your part.

The instructions in the OP's post leave out one very vital detail that cannot be overlooked or GPU passthrough will not work. Specifically, you must tell the kernel which PCI device to forcefully load the VFIO module for. In order to do that, follow these steps:

1. Discover the GPU PCI ID: lspci -nn | grep -i nvidia

Note the output in brackets (e.g. [1002:73ff]) as you'll need it for the next step. It will look something like this:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3060 Ti] [10de:2486] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)

2. Create /etc/modprobe.d/vfio.conf with the following contents:

options vfio-pci ids=10de:2486,10de:228b
blacklist nouveau
blacklist nvidiafb
softdep nvidiafb pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

Replace the ids=NNNN:NNNN with the output from step 1 to match the PCI ID of your GPU and audio device. The audio device is required in order to pass audio via HDMI or Displayport

3. Reboot the Proxmox host.

Next time I also advise reading a write-up for nVidia cards instead of trying to follow a write-up for AMD GPU's when trying to configure a nVidia card for GPU passthrough.

 

[MoxProxNoob]

There are some config settings (GRUB, Modules etc) you could still try, if you search the web. But it would cost you some time to try them out, you would need a few hours at least without a guarantee that it will work.

Should you decide to buy new hardware in the feature for a working passthrough system, I would recommend to buy the exact same configuration I have. (At least the exact same motherboard and CPU and GPU from the same generation).

For me it's awesome. My work/production system is fedora linux, the gaming part is windows 11. I can make snapshots/backups at critical times or if I want to try something out, don't have to reinstall/reactivate windows. I love trying different linux systems/DE out and can run everything native. And performance loss while gaming is around 1% compared to native. (At least in my tests).

You do lose some features compared to native, like standby/hibernate, direct access to monitoring your chipset/cpu etc, and the boot up time is like 15 seconds longer. But that's not a problem for me.

But do post you're vm config, maybe there's something you've missed, who knows

Fortunately, I walked away and didn’t wipe it and install windows, so I can give it a few more hours before giving up.

I’ll definitely post the vm config tomorrow.

Thanks.

 

[MoxProxNoob]

iGPU has nothing to do with trying to get a dGPU working for GPU passthrough. I've set up passthrough on 5+ systems and been successful every single time. I would be very surprised if the problem you're encountering is NOT because of misconfiguration on your part.

The instructions in the OP's post leave out one very vital detail that cannot be overlooked or GPU passthrough will not work. Specifically, you must tell the kernel which PCI device to forcefully load the VFIO module for. In order to do that, follow these steps:

1. Discover the GPU PCI ID: lspci -nn | grep -i nvidia

Note the output in brackets (e.g. [1002:73ff]) as you'll need it for the next step. It will look something like this:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3060 Ti] [10de:2486] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)

2. Create /etc/modprobe.d/vfio.conf with the following contents:

options vfio-pci ids=10de:2486,10de:228b
blacklist nouveau
blacklist nvidiafb
softdep nvidiafb pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

Replace the ids=NNNN:NNNN with the output from step 1 to match the PCI ID of your GPU and audio device. The audio device is required in order to pass audio via HDMI or Displayport

3. Reboot the Proxmox host.

Next time I also advise reading a write-up for nVidia cards instead of trying to follow a write-up for AMD GPU's when trying to configure a nVidia card for GPU passthrough.

Thanks for your help.

We’re those headless servers or single box workstations with a gui on the host?

And if they were single box workstations did they happen to have iGPUs in the CPU?

My understanding is that for the workstation setup u need at least two GPUs. One for the host and one for the VMs as the passed through device is unavailable for the host to use. Not sure if this is outdated info or not.

At this point I know the vendor:device ID off by heart (10de:2782). The ID for the audio device escapes me now.

I am missing the options line and the softdep lines.

This is all new to me. What do the softdep lines do? And I believe my mobo audio device might be Realtek, not intel.

As for the write up, I found this by looking for a relatively new 8.x.x write up for AMD CPU. Glad I did cos I found u guys.

 

[jeenam]

Thanks for your help.

We’re those headless servers or single box workstations with a gui on the host?

And if they were single box workstations did they happen to have iGPUs in the CPU?

My understanding is that for the workstation setup u need at least two GPUs. One for the host and one for the VMs as the passed through device is unavailable for the host to use. Not sure if this is outdated info or not.

If the host system only has a single display adapter, then Proxmox will not be able to use it for console output as it can only be assigned to either the host OS or a VM. For users whose systems only have a single GPU they rely on SSH for command line access. This isn't problematic as the majority of Linux systems don't have a desktop window manager GUI for user interface - they are managed via SSH. I can't even remember the last time I bothered installing Gnome/KDE/XFCE/Enlightenment as a desktop window manager on a Linux system.

At this point I know the vendor:device ID off by heart (10de:2782). The ID for the audio device escapes me now.

I am missing the options line and the softdep lines.

This is all new to me. What do the softdep lines do? And I believe my mobo audio device might be Realtek, not intel.

As for the write up, I found this by looking for a relatively new 8.x.x write up for AMD CPU. Glad I did cos I found u guys.

The softdep lines for the VFIO module are to load it before the normal module the device would typically use.

The fact you are using an AMD CPU doesn't matter. GPU passthrough configuration does not differ based on the CPU used by the host. The configuration is specific to the GPU type that is used (nvidia or AMD). So like I mentioned, look for instructions for nvidia cards.

This is all new to me. What do the softdep lines do? And I believe my mobo audio device might be Realtek, not intel.

ALC97 codec devices use the 'snd_hda_intel' kernel module regardless of who the chip vendor is.

 

[frankmanzhu]

Enabled Above 4G Decoding AND Resizable Bar Support

Woot? I'm shocked.

I never got it working if I enable 4G or not adding
initcall_blacklist=sysfb_init
to GRUB_CMDLINE_LINUX_DEFAULT to blacklist console output.
I'll get "BAR 3: can't reserve [mem]" error

Maybe your way to set rom will do the same result (like I avoid the host to temper anything with GPU)
If there is a more reliable way to dump rom myself instead of 3rd party, it will be great solution.