AMD APU (gpu) passthrough

[gingi]

Hello - Relative noob here. I am trying to passthrough my gpu (part of the APU) to a linux vm. I have followed the tutorial here and various permutations and I can't get it to work. Every time I start the VM it causes the host to freeze. Any ssh session is frozen and the web ui is basically frozen (I can click but many things a greyed out and I just see that loading circle icon). I have also tried many different settings in my bios.

Here is system specs:
CPU: AMD Ryzen 7 5700g
MB: MSI B550M VC Wifi
- IOMMU: Enabled
- SR-IOV: Enabled
RAM: 32 gb

$ lspci -nnk
[...]
30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1638] (rev c8)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1636]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
30:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller [1002:1637]
Subsystem: Micro-Star International Co., Ltd. [MSI] Renoir Radeon High Definition Audio Controller [1462:9c95]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
[...]

$ cat /etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist amdgpu

$ cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:1638,1002:1637

$ cat /etc/default/grub
[...]
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt nomodeset initcall_blacklist=sysfb_init"
[...]

$ dmesg | grep 'remapping'
[    0.776356] AMD-Vi: Interrupt remapping enabled

$ dmesg | grep -E "DMAR|IOMMU"
[    0.775501] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.776340] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.833806] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

$ dmesg | grep -i vfio
[    7.269840] VFIO - User Level meta-driver version: 0.3
[    7.325403] vfio-pci 0000:30:00.0: vgaarb: deactivate vga console
[    7.325408] vfio-pci 0000:30:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    7.325604] vfio_pci: add [1002:1638[ffffffff:ffffffff]] class 0x000000/00000000
[    7.348471] vfio_pci: add [1002:1637[ffffffff:ffffffff]] class 0x000000/00000000

$ cat /etc/modules

vfio
vfio_iommu_type1
vfio_pci

I then add the gpu device to my VM like so:

 

[fahadshery]

Hi,

were you able to solve it?

 

[gingi]

Yes. Though it wasn't pretty. You need to get the motherboard bios then I followed a mix of instructions from https://winraid.level1techs.com/t/tool-guide-news-uefi-bios-updater-ubu/30357 and https://www.reddit.com/r/homelab/comments/11l0s5j/boxmox_asrock_4x4_box_5800u_jbod_proxmox/?rdt=5185. You need to add that vbois as options in your .conf

Like
hostpci0: 0000:30:00.0,romfile=vbios_1638.dat,pcie=1,x-vga=1

 

[fahadshery]

Yes. Though it wasn't pretty. You need to get the motherboard bios then I followed a mix of instructions from https://winraid.level1techs.com/t/tool-guide-news-uefi-bios-updater-ubu/30357 and https://www.reddit.com/r/homelab/comments/11l0s5j/boxmox_asrock_4x4_box_5800u_jbod_proxmox/?rdt=5185. You need to add that vbois as options in your .conf

Like
hostpci0: 0000:30:00.0,romfile=vbios_1638.dat,pcie=1,x-vga=1

that's awesome...did you not make any notes step by step???

 

[leesteken]

that's awesome...did you not make any notes step by step???

x-vga=1 (Primary GPU) is a work-around for NVidia GPUs and probably not needed. You probably do need to "breaK" the iOMMU groups (with pcie_acs_override, use at your own risk) because the integrated graphics is not properly isolated. Maybe you need additional work-arounds if the integrated graphics does not reset properly.