AMD iGPU passthrough won't work

[jhmc93]

Hello,
I have a trigkey mini pc with the following specs:
AMD Ryzen 7 5800H with Radeon graphics
24GB ram
512GB ssd
my problem is i can't get the integrated GPU to passthrough.
I have all the drivers and guest tools installed but nothing shows on my monitor but when i check device manager i get the error 43 warning,
can someone please help
thank you

 

[VMler]

Sounds familiar: please take a look at my latest post: AMD iGPU pass trough resolution

 

[jhmc93]

Sounds familiar: please take a look at my latest post: AMD iGPU pass trough resolution

See I have my monitor connected and downloaded the drivers manually but still can’t get any display and the error is there when I remote into it

 

[VMler]

Ok, and did you follow all the steps of that Github-Guide? Enabling IOMMU, blackisting the drivers, configuring vfio including the kernel command-line update in grub? In my case it was neccessary to dump the vbios and present it as a "romfile" to the guest-config, others have reported they didn't need to do this.

Can you please post the output of the following commands?

1. dmesg | egrep IOMMU

2. lspci -nn

 

[jhmc93]

Ok, and did you follow all the steps of that Github-Guide? Enabling IOMMU, blackisting the drivers, configuring vfio including the kernel command-line update in grub? In my case it was neccessary to dump the vbios and present it as a "romfile" to the guest-config, others have reported they didn't need to do this.

Can you please post the output of the following commands?

1. dmesg | egrep IOMMU

2. lspci -nn

output of dmesg | egrep IOMMU:


output of lspci -nn:


hope this is right and if u can help

 

[VMler]

Thank you. "dmesg | egrep IOMMU" shows no IOMMU-Groups and the warning that you use ACS override. You should use ACS override as "last resort" and not for default. Please check your Kernel command line in /etc/default/grub and change the CMDLLINE to the following:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init video=efifb:off"

then execute "update-grub"

After rebooting the host "dmesg | egrep IOMMU" should show something like "iommu: Default domain type: Passthrough" followed by the pcie to IOMMU grouping.

Then please post "lspci -vd 1002:1638"

 

[jhmc93]

the out

Thank you. "dmesg | egrep IOMMU" shows no IOMMU-Groups and the warning that you use ACS override. You should use ACS override as "last resort" and not for default. Please check your Kernel command line in /etc/default/grub and change the CMDLLINE to the following:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init video=efifb:off"

then execute "update-grub"

After rebooting the host "dmesg | egrep IOMMU" should show something like "iommu: Default domain type: Passthrough" followed by the pcie to IOMMU grouping.

Then please post "lspci -vd 1002:1638"

the output of lspci -vd 1002:1638:

 

[VMler]

Looks good, your GPU is in IOMMU group 13 and is occupied by vfio-pci as it should. Whats the content of your guest-config?

 

[jhmc93]

Looks good, your GPU is in IOMMU group 13 and is occupied by vfio-pci as it should. Whats the content of your guest-config?

config:

 

[leesteken]

config:
View attachment 64487

Why the hidden=1 and x-vga=1? Those are work-arounds for NVidia GPUs. I have seen AMD GPU drivers having trouble with Machine Version 8.1 but work fine with 7.2 instead.

 

[VMler]

Ok, well lets see:

balooning is off -> correct
bios is uefi -> correct
both pcie-devices (GPU and audio) are attached -> correct
machine type is q35 -> perfect

leesteken is right: please delete "hidden=1" and "x-vga=1" from the config. The explanation for this is that that "hidden=1" is unnecessary anymore and for "x-vga=1" you are too early at this point. Setting "x-vga=1" and "vga: none" is right when your GPU is working fine in your windows guest, but not yet.

For the configuration please add the following args to line 2 after "agent: 1":

args: -cpu 'host,-hypervisor,kvm=off'

Now it is important that you have to make sure that the GPU is completely isolated from the proxmox host. The Problem is that the iGPU cannot be shared between host and guest at the same time. The foundation therefor is the Kernel-Command line option

initcall_blacklist=sysfb_init video=efifb:off

The following files should look like:

#/etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist amdgpu
blacklist {your sound device driver, see the explanation below}

==========================
#/etc/modprobe.d/vfio.conf

options vfio-pci ids=[all your vendor:devids that are in the same iommu-group as your iGPU] disable_vga=1
#For example: options vfio-pci ids=1002:15bf,1002:1640,1022:15c7,1022:15b9,1022:15ba,1022:15e2,1022:15e3 disable_vga=1
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci #curiously this is the driver that linux want to use for my AMD sound device 1002:1640 - please check yours, it has to be handed over to vfio-pci either!

After this is done, please regenerate the initramfs with:

update-initramfs -u -k all

And reboot.

The last step is now that you present your vbios as a romfile to the hostpci0 option. For example:

hostpci0: 0000:07:00.0,pcie=1,romfile=vbios_7840hs.bin

This is now necessary because the host does not use/initialize the GPU anymore. It has now to be done by the hypervisor via the romfile. So please follow the instructions at the "Configuring the GPU in the Windows VM"-Section at isc30 / ryzen-7000-series-proxmox (the vbios.c-thing -- dont worry about "7000 series", this is generally valid for your 5800H too.

Theoretically all this measures should work now for your setup. But to be fair there is a little chance that your system has a vendor sided hardware issue that the software cannot work around. So fingers crossed and good luck!

 

[jhmc93]

Ok, well lets see:

balooning is off -> correct
bios is uefi -> correct
both pcie-devices (GPU and audio) are attached -> correct
machine type is q35 -> perfect

leesteken is right: please delete "hidden=1" and "x-vga=1" from the config. The explanation for this is that that "hidden=1" is unnecessary anymore and for "x-vga=1" you are too early at this point. Setting "x-vga=1" and "vga: none" is right when your GPU is working fine in your windows guest, but not yet.

For the configuration please add the following args to line 2 after "agent: 1":

args: -cpu 'host,-hypervisor,kvm=off'

Now it is important that you have to make sure that the GPU is completely isolated from the proxmox host. The Problem is that the iGPU cannot be shared between host and guest at the same time. The foundation therefor is the Kernel-Command line option

initcall_blacklist=sysfb_init video=efifb:off

The following files should look like:

#/etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist amdgpu
blacklist {your sound device driver, see the explanation below}

==========================
#/etc/modprobe.d/vfio.conf

options vfio-pci ids=[all your vendor:devids that are in the same iommu-group as your iGPU] disable_vga=1
#For example: options vfio-pci ids=1002:15bf,1002:1640,1022:15c7,1022:15b9,1022:15ba,1022:15e2,1022:15e3 disable_vga=1
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci #curiously this is the driver that linux want to use for my AMD sound device 1002:1640 - please check yours, it has to be handed over to vfio-pci either!

After this is done, please regenerate the initramfs with:

update-initramfs -u -k all

And reboot.

The last step is now that you present your vbios as a romfile to the hostpci0 option. For example:

hostpci0: 0000:07:00.0,pcie=1,romfile=vbios_7840hs.bin

This is now necessary because the host does not use/initialize the GPU anymore. It has now to be done by the hypervisor via the romfile. So please follow the instructions at the "Configuring the GPU in the Windows VM"-Section at isc30 / ryzen-7000-series-proxmox (the vbios.c-thing -- dont worry about "7000 series", this is generally valid for your 5800H too.

Theoretically all this measures should work now for your setup. But to be fair there is a little chance that your system has a vendor sided hardware issue that the software cannot work around. So fingers crossed and good luck!

Will try these a bit later when I am on the machine,
The section here:

initcall_blacklist=sysfb_init video=efifb:off

,
Do I add it to my grub file?,
I will let you know how I get on later hopefully it’ll work.
Thank you

 

[VMler]

Will try these a bit later when I am on the machine,
The section here:

initcall_blacklist=sysfb_init video=efifb:off

,
Do I add it to my grub file?,
I will let you know how I get on later hopefully it’ll work.
Thank you

Yes, your /etc/default/grub file should have the following options in it as i already mentioned earlier:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init video=efifb:off"

After changing the file do not forget to execute "update-grub"

Again: good luck!

 

[jhmc93]

Yes, your /etc/default/grub file should have the following options in it as i already mentioned earlier:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init video=efifb:off"

After changing the file do not forget to execute "update-grub"

Again: good luck!

hello, me again for the below code,

#/etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist amdgpu
blacklist {your sound device driver, see the explanation below}

The sound driver bit do I put blacklist for eg: 100:1640
also this:

options vfio-pci ids=[all your vendor:devids that are in the same iommu-group as your iGPU] disable_vga=1

do i put whats in the group listed here :

thanks again

 

[VMler]

Yes, you have to hand over all devices to vfio that are in the same pci bus. In your case 04:00:*

It is not necessary to put all of those devices in your guest-configuration but vfio has to be in control of the whole bus so it can handle all I/O accesses to the corresponding devices without conflicts. Unfortunately this is the point where all of your efforts can fail due to a bad hardware design.

 

[jhmc93]

Yes, you have to hand over all devices to vfio that are in the same pci bus. In your case 04:00:*

It is not necessary to put all of those devices in your guest-configuration but vfio has to be in control of the whole bus so it can handle all I/O accesses to the corresponding devices without conflicts. Unfortunately this is the point where all of your efforts can fail due to a bad hardware design.

So the sound driver bit I just put 1002:1637?
And just input the group into the vendor id’s great!, just wanted to know thank you

 

[jhmc93]

Yes, you have to hand over all devices to vfio that are in the same pci bus. In your case 04:00:*

It is not necessary to put all of those devices in your guest-configuration but vfio has to be in control of the whole bus so it can handle all I/O accesses to the corresponding devices without conflicts. Unfortunately this is the point where all of your efforts can fail due to a bad hardware design.

how does this look:




screenshot 2:






EDIT:
I tried running machine with the following settings, ^ (hope i got it right), but i get the following error when setting the gpu to primary.

screenshot of error:


here is the vm config file:


any help would be greatly appreciated! thanks again
SOLVED with changing the bios to uefi.
PROBLEM!:
Still getting error 43, nothing showing on monitor, can you have a look above to see if i set it up right, if radeon adrenaline is manually installed it keeps saying gpu driver crash error even after i've ran the radeon bug crash fix.
@VMler if u could check above to see if i've blacklisted the right hardware as i am still having problems.
thank you if u can help

 

[VMler]

Good evening, putting the pci device ids into the blacklist file is not correct. In this file you have to blacklist your sound driver that the kernel wants to use for this device. "radeon" and "amdgpu" are not device names/ids, they are driver (module) names. You have to find out what your kernel want to use for this specific device. "lspci" is an appropiate approach to do this. Second: why are you use "...romfile=7490hs.bin"? Didn't you say you have a 5800H? Personally i would dump the original vbios from your system because even when you think you have a matchig one that is already present on your host, there might be some vendor adjustments for your specific system that can undermine the overall stabillity or functionallity by using a "wrong" vbios. This can also cause error 43.

 

[jhmc93]

Good evening, putting the pci device ids into the blacklist file is not correct. In this file you have to blacklist your sound driver that the kernel wants to use for this device. "radeon" and "amdgpu" are not device names/ids, they are driver (module) names. You have to find out what your kernel want to use for this specific device. "lspci" is an appropiate approach to do this. Second: why are you use "...romfile=7490hs.bin"? Didn't you say you have a 5800H? Personally i would dump the original vbios from your system because even when you think you have a matchig one that is already present on your host, there might be some vendor adjustments for your specific system that can undermine the overall stabillity or functionallity by using a "wrong" vbios. This can also cause error 43.

hello again, thank you for your reply,
my sound device is here:

so what do I use in the blacklist???

and right here i thought you said use 7000 series in your message:
"Configuring the GPU in the Windows VM"-Section at isc30 / ryzen-7000-series-proxmox (the vbios.c-thing -- dont worry about "7000 series", this is generally valid for your 5800H too."

i thought you meant use the rom file of that series, that's probably mistake on my part for misreading i did follow the vbios.c thing and my bios came out as this "vbios_1002_1638.bin" do i use that?

how does the vfio.conf look?
let me know.
As trying to sort this out asap,
thanks

 

[jhmc93]

Good evening, putting the pci device ids into the blacklist file is not correct. In this file you have to blacklist your sound driver that the kernel wants to use for this device. "radeon" and "amdgpu" are not device names/ids, they are driver (module) names. You have to find out what your kernel want to use for this specific device. "lspci" is an appropiate approach to do this. Second: why are you use "...romfile=7490hs.bin"? Didn't you say you have a 5800H? Personally i would dump the original vbios from your system because even when you think you have a matchig one that is already present on your host, there might be some vendor adjustments for your specific system that can undermine the overall stabillity or functionallity by using a "wrong" vbios. This can also cause error 43.

Are you still there?