Another GPU passthrough post (AMD Vega 7)

[encrypted_]

Hello there,

First of all, sorry for posting again about a recurring issue but I've read everything I could and still couldn't make it work.
My system:

Asrock A320M-HDR R4
Ryzen 4650Pro
32GB ECC UDIMM
I got some success following this video tutorial: https://www.bilibili.com/video/BV11d4y1G7Nk/

What I see:

• Each pci component in correctly in it's iommu group.
• I was able to boot the VM with both VIRTIO GPU display and PCIE device
• I was able to install the AMD drivers
• When I removed the VIRTIO GPU display and selected None, keeping the PCIE it won't boot.
• I can boot the VM when the PCIE line is not present
• The VM won't boot after I add the PCI line

/etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt amd_iommu=on pcie_acs_override=downstream,multifunction initcall_blacklist-sysfb_init"
GRUB_CMDLINE_LINUX=""

/etc/modprobe.d/blacklist.conf

blacklist amdgpu
blacklist snd_hda_intel

/etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1

/etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:1636,1002:1637
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci

/etc/modules

jc42
nct6775
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vendor-reset

/etc/pve/qemu-server/103.conf

bios: seabios
boot: order=virtio0;ide2;net0;ide0
cores: 4
cpu: host,hidden=1
hostpci0: 0000:06:00,pcie=1,romfile=vbios_1636.dat,x-vga=1
ide0: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
ide2: local:iso/Win10_22H2_BrazilianPortuguese_x64.iso,media=cdrom,size=5666236K
machine: pc-q35-7.2
memory: 8192
meta: creation-qemu=7.2.0,ctime=1698553568
name: win10-igpu
net0: virtio=E6:48:0A:D8:23:10,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=b725946b-3fd3-4f8e-aff2-ceb78533f4b0
sockets: 1
vga: qxl
virtio0: local-lvm:vm-103-disk-0,iothread=1,size=32G
vmgenid: a9ab640a-43de-409f-bd32-65d7c7d4c239

dmesg | grep -e DMAR -e IOMMU

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.369860] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.370438] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.689802] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

BOOT_IMAGE=/boot/vmlinuz-5.15.116-1-pve root=/dev/mapper/pve-root ro quiet iommu=pt amd_iommu=on pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 10 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b3] (rev 02)
IOMMU group 11 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 12 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 13 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)
IOMMU group 15 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 16 06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [1002:1636] (rev ff)
IOMMU group 17 06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637] (rev ff)
IOMMU group 18 06:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df] (rev ff)
IOMMU group 19 06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639] (rev ff)
IOMMU group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 20 06:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639] (rev ff)
IOMMU group 21 06:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3] (rev ff)
IOMMU group 22 07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU group 23 07:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU group 2 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU group 3 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 4 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 5 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
IOMMU group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 [1022:1448]
IOMMU group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 [1022:1449]
IOMMU group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 [1022:144a]
IOMMU group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 [1022:144b]
IOMMU group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 [1022:144c]
IOMMU group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 [1022:144d]
IOMMU group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 [1022:144e]
IOMMU group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 [1022:144f]
IOMMU group 8 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bc] (rev 02)
IOMMU group 9 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b8] (rev 02)

lspci -nnv

see attached

 

[encrypted_]

So I managed to make it work but only once each host reset.
If I reboot the VM it's stops working. Still looking for a solution.

 

[leesteken]

So I managed to make it work but only once each host reset.
If I reboot the VM it's stops working. Still looking for a solution.

This is very common when a GPU does not reset properly. Is you GPU supported by vendor-reset maybe?
Glad you got it working. Did you need to do anything special?

 

[encrypted_]

Unfortunatelly my iGPU (Vega 7 from Ryzen 4650G Pro) is not supported by vendor-reset.
My settings are exactly as above. It boots with those configuration when host is reseted but as I said, only once.

 

[Whisky]

Unfortunatelly my iGPU (Vega 7 from Ryzen 4650G Pro) is not supported by vendor-reset.
My settings are exactly as above. It boots with those configuration when host is reseted but as I said, only once.

Have you tried to reboot your Proxmox host?