iGPU Passthrough to Windows VM – PVE freeze when starting VM

[reb9l]

Hi everyone,

I'm trying to passthrough the iGPU to a Windows guest VM, following a guide I found online. However, when I attempt to start the Windows VM, the hypervisor completely hangs, forcing me to reboot the host machine. I've been stuck on this issue for a while and can't seem to find a solution.

I'm trying to passthrough the iGPU to a Windows 10 guest.
I've attached my VM and PVE configuration below for reference:

Device: Beelink SER7 with a Ryzen 7840H CPU and the integrated 780M graphics

#pveversion

pve-manager/8.2.4/faa83925c9641325 (running kernel: 6.8.12-1-pve)

#lspci -nn | grep -e 'AMD/ATI'

c5:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 [1002:15bf] (rev c2)
c5:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]

#cat /etc/modules

# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

#cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-6.8.12-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on initcall_blacklist=sysfb_init

#cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:15bf,1002:1640 disable_vga=1
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

#lspci -nnks c5:00

c5:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 [1002:15bf] (rev c2)
        Subsystem: Device [1f66:0031]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
c5:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
c5:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h (Model 74h) CCP/PSP 3.0 Device [1022:15c7]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Family 19h (Model 74h) CCP/PSP 3.0 Device [1022:15c7]
        Kernel driver in use: ccp
        Kernel modules: ccp
c5:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
c5:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ba]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
c5:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2] (rev 63)
        Subsystem: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2]
        Kernel driver in use: snd_pci_ps
        Kernel modules: snd_pci_acp3x, snd_rn_pci_acp3x, snd_pci_acp5x, snd_pci_acp6x, snd_acp_pci, snd_rpl_pci_acp6x, snd_pci_ps, snd_sof_amd_renoir, snd_sof_amd_rembrandt, snd_sof_amd_vangogh, snd_sof_amd_acp63
c5:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
        DeviceName: Realtek ALC256
        Subsystem: Device [1f66:0202]
        Kernel modules: snd_hda_intel
c5:00.7 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Device [1022:164a]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:164a]
        Kernel driver in use: pcie_mp2_amd
        Kernel modules: amd_sfh

#dmesg | egrep IOMMU

[    0.386157] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.390146] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

#cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init video=efifb:off"

#cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:15bf,1002:1640 disable_vga=1
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

#cat /etc/pve/qemu-server/305.conf

agent: 1
args: -cpu 'host,-hypervisor,kvm=off'
balloon: 0
bios: seabios
boot: order=virtio0;ide0;ide2;net0
cores: 4
cpu: host
hostpci0: 0000:c5:00.0,pcie=1,romfile=vbios_7840hs.bin
hostpci1: 0000:c5:00.1,pcie=1,romfile=AMDGopDriver.rom
ide0: local:iso/virtio-win-0.1.240.iso,media=cdrom,size=612812K
ide2: local:iso/ru-ru_windows_10_consumer_editions_version_22h2_updated_aug_2024_x64_dvd_2454bb57.iso,media=cdrom,size=6666126K
machine: pc-q35-9.0
memory: 16384
meta: creation-qemu=9.0.2,ctime=1726078344
name: win10gpu
net0: virtio=BC:24:11:A3:BF:A6,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=1684309a-a859-4038-81aa-b774ebc1b55e
sockets: 2
tpmstate0: local-lvm:vm-305-disk-0,size=4M,version=v2.0
virtio0: local-lvm:vm-305-disk-1,discard=on,iothread=1,size=80G
vmgenid: 668498c3-afd6-4ac6-8545-7c31568afeee

Are there any tips to solve this problem?

 

[leesteken]

Are there any tips to solve this problem?

Check your IOMMU groups: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation

 

[reb9l]

Check your IOMMU groups: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation

#for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU group 10 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
IOMMU group 11 00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]
IOMMU group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
IOMMU group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f0]
IOMMU group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f1]
IOMMU group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f2]
IOMMU group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f3]
IOMMU group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f4]
IOMMU group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f5]
IOMMU group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f6]
IOMMU group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14f7]
IOMMU group 14 01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU group 15 02:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU group 16 04:00.0 Non-Volatile memory controller [0108]: Micron Technology Inc 2550 NVMe SSD (DRAM-less) [1344:5416] (rev 01)
IOMMU group 17 c5:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 [1002:15bf] (rev c2)
IOMMU group 18 c5:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
IOMMU group 19 c5:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h (Model 74h) CCP/PSP 3.0 Device [1022:15c7]
IOMMU group 1 00:01.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
IOMMU group 20 c5:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b9]
IOMMU group 21 c5:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15ba]
IOMMU group 22 c5:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2] (rev 63)
IOMMU group 23 c5:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU group 24 c5:00.7 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Device [1022:164a]
IOMMU group 25 c6:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ec]
IOMMU group 26 c6:00.1 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] AMD IPU Device [1022:1502]
IOMMU group 27 c7:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ec]
IOMMU group 28 c7:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15c0]
IOMMU group 29 c7:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15c1]
IOMMU group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU group 30 c7:00.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller [1022:1668]
IOMMU group 31 c7:00.6 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Pink Sardine USB4/Thunderbolt NHI controller [1022:1669]
IOMMU group 3 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
IOMMU group 4 00:02.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
IOMMU group 5 00:02.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ee]
IOMMU group 6 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU group 6 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
IOMMU group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU group 7 00:04.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 19h USB4/Thunderbolt PCIe tunnel [1022:14ef]
IOMMU group 8 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ea]
IOMMU group 9 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14eb]

 

[leesteken]

It looks like you added the pcie_acs_override before showing the IOMMU groups, which makes the information less than useful. It also looks like integrated graphics, for which you might need a very specific work-around (which I don't know, sorry). Try to find out if someone succeeded with a 7840H with 780M and find out how they did it.

 

[sunnyboy]

Hi, did you succeed with the iGPU-passthru. I have the same behaviour (freeze) with a similar iGPU.

 

[fseesink]

Hey @reb9l . I also have a SER7 and am working on setting up GPU passthrough. And like you I ended up on the same GitHub repo post.

TL;DR: I can't give you answers, but maybe I can point you in a direction.

I'm not as far, as I've been chewing on so many different YouTube videos and blog posts on dealing with various bits around Proxmox, so bear with me here. The GitHub repo is great, but coalescing various bits together, I feel like there's a bit of inconsistency/confusion around what parameters are needed where.

My first question to you is whether you verified that you enabled IOMMU support in the BIOS. (e.g., what is the output of a command like dmesg | grep -i iommu ?)

Next question I have is which boot manager is your Proxmox set to use, GRUB or system-boot? You show both the output of both `cat /proc/cmdline` and `cat /etc/default/grub`. But I believe only one of those is going to be in use, and it depends on how your Proxmox file system was setup. For example, what is the output of

efibootmgr -v

Regarding all this, the key thing is that you have to have the bits in the right place for the particular boot manager you are using. There's no point in modifying the GRUB config if you're using system-boot or vice versa.

I ask this as you have amd_iommu=on in /etc/cmdline, while you have iommu=pt in /etc/default/grub. But you don't have both in either. I'm still not 100% sure if both or which one is truly needed. It seems if you use an invalid flag it gets ignored (based again on my various reading). But for now I have both in both of those files.

That said, have you seen this thread from that same repo, as it applies to the Minisforum UM780, which uses the same AMD Ryzen 7 7840HS chip that is in the SER7:

If not, you may find something useful in that thread, notably this post by 20TbCh20.

Anyway, if you get things working, would love to hear how you're setup. And if I have success, will do same when I get there.