iGPU passthorugh "pci_hp_register failed with error 16"

[monkee]

Hello,

I want to pass my intel iGPU (Intel Core i7-8700) through to a VM. It should replace my media PC, so I basically want to watch movies inside a VM and see it on a monitor (in this case my TV).
I followed this guide: https://pve.proxmox.com/wiki/PCI_Passthrough and this guide https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/#introduction

When I boot the VM I get the message "pci_hp_register failed with error 16" in the noVNC console and the physical monitor shows some weird pixel errors (see attached photo).

Any help is appreciated.
Thanks


Some more info:

root@pve:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init video=simplefb:off video=vesafb:off video=efifb:off video=vesa:off disable_vga=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,snd_hda_intel,snd_hda_codec_hdmi,i915"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
root@pve:~#

root@pve:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
root@pve:~#

root@pve:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
root@pve:~#

root@pve:~# dmesg | grep IOMMU
[    0.090222] DMAR: IOMMU enabled
[    0.238736] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.782673] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.782674] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.782675] DMAR: IOMMU feature nwfs inconsistent
[    0.782676] DMAR: IOMMU feature pasid inconsistent
[    0.782677] DMAR: IOMMU feature eafs inconsistent
[    0.782678] DMAR: IOMMU feature prs inconsistent
[    0.782679] DMAR: IOMMU feature nest inconsistent
[    0.782679] DMAR: IOMMU feature mts inconsistent
[    0.782680] DMAR: IOMMU feature sc_support inconsistent
[    0.782681] DMAR: IOMMU feature dev_iotlb_support inconsistent
root@pve:~#

root@pve:~# cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
root@pve:~#

root@pve:~# dmesg | grep 'remapping'
[    0.238738] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.241876] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@pve:~#

 

[keivan65]

Hello,

I want to pass my intel iGPU (Intel Core i7-8700) through to a VM. It should replace my media PC, so I basically want to watch movies inside a VM and see it on a monitor (in this case my TV).
I followed this guide: https://pve.proxmox.com/wiki/PCI_Passthrough and this guide https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/#introduction

When I boot the VM I get the message "pci_hp_register failed with error 16" in the noVNC console and the physical monitor shows some weird pixel errors (see attached photo).

Any help is appreciated.
Thanks


Some more info:

root@pve:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init video=simplefb:off video=vesafb:off video=efifb:off video=vesa:off disable_vga=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu,snd_hda_intel,snd_hda_codec_hdmi,i915"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
root@pve:~#

root@pve:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
root@pve:~#

root@pve:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
root@pve:~#

root@pve:~# dmesg | grep IOMMU
[    0.090222] DMAR: IOMMU enabled
[    0.238736] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.782673] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.782674] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.782675] DMAR: IOMMU feature nwfs inconsistent
[    0.782676] DMAR: IOMMU feature pasid inconsistent
[    0.782677] DMAR: IOMMU feature eafs inconsistent
[    0.782678] DMAR: IOMMU feature prs inconsistent
[    0.782679] DMAR: IOMMU feature nest inconsistent
[    0.782679] DMAR: IOMMU feature mts inconsistent
[    0.782680] DMAR: IOMMU feature sc_support inconsistent
[    0.782681] DMAR: IOMMU feature dev_iotlb_support inconsistent
root@pve:~#

root@pve:~# cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
root@pve:~#

root@pve:~# dmesg | grep 'remapping'
[    0.238738] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.241876] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@pve:~#

Did you find a solution? I have same problem.

 

[celemine1gig]

You want to pass-through the Intel iGPU, yet your list of blacklisted modules only lists modules for Nvidia and AMD. See the issue? hint: i915.
At some point you seem to have tried it on the kernel commandline, but then commented it out again. Right?

Next question: Did you pass the PCI-id of your iGPU to the vfio module?

The following might also be of interest:
https://github.com/qemu/qemu/blob/master/docs/igd-assign.txt
See "2) "Legacy" mode:" in that linked page.

From my own experience with a CoffeeLake system I can add, that for me working with the Legacy video BIOS was much better in this scenario, than working with the UEFI GOP. Buzz-words "CSM" and "Video" setting in that CSM-related BIOS sub-menu (settings are either "Legacy" or "UEFI"). Might be different for your setup, though.

This all will most likely also depend on your exact Mainboard and its BIOS setup.
Good luck!

 

[kjkent]

@celemine1gig the OP might be passing through mediated device functions, a la Intel GVT-g which doesn't necessitate blacklisting i915.

@monkee - disabling ACPI in the VM's options tab worked for me. It does mean, however, that you need to ensure qemu-guest-agent is installed and functioning in order for the guest to receive shutdown commands from the hypervisor.

 

[SInisterPisces]

@celemine1gig the OP might be passing through mediated device functions, a la Intel GVT-g which doesn't necessitate blacklisting i915.

@monkee - disabling ACPI in the VM's options tab worked for me. It does mean, however, that you need to ensure qemu-guest-agent is installed and functioning in order for the guest to receive shutdown commands from the hypervisor.

I was getting the same error on an Alder Lake 12th Gen iGPU passthrough attempt. Disabling ACPI in the VM options fixed it.

I've got the qemu-guest-tools installed in the VM, so when I send a shutdown command, the machine shuts down (I get a black screen), but the VM itself doesn't turn off until I issue a STOP command.

Meanwhile, whether or not I blacklist the Intel sound drivers in Proxmox, it's dragging the PCIE sound device and PCI bridges and a bunch of other stuff into the VM when passthrough is enabled. I'm kind of amazed that turning on this VM doesn't crash the machine.

Not trying to derail the thread, but I just wanted to make it clear that getting past error 16 doesn't mean the problem is fixed. At this point it boots in a q35 VM, but with way more PCIe devices than it should have access to from the host.

It also can't properly read device information from the iGPU.

 

[celemine1gig]

@kjkent:
The OP did mention "and see it on a monitor". And I guess a physical display was meant by that. As far as I know, this is only possible via full pass-though.

@SInisterPisces:
I am in no way sure, but maybe that Alderlake iGPU needs the same a similar reset-workaround, as some slighly older AMD GPUs or other devices? If I remember correctly a quirk, telling the system NOT to reset the device. Maybe worth a try with a self-compiled kernel?
See "quirk_no_bus_reset":
https://git.kernel.org/pub/scm/linu....git/tree/drivers/pci/quirks.c?h=v6.8.5#n3730

 

[SInisterPisces]

@kjkent:
The OP did mention "and see it on a monitor". And I guess a physical display was meant by that. As far as I know, this is only possible via full pass-though.

@SInisterPisces:
I am in no way sure, but maybe that Alderlake iGPU needs the same a similar reset-workaround, as some slighly older AMD GPUs or other devices? If I remember correctly a quirk, telling the system NOT to reset the device. Maybe worth a try with a self-compiled kernel?
See "quirk_no_bus_reset":
https://git.kernel.org/pub/scm/linu....git/tree/drivers/pci/quirks.c?h=v6.8.5#n3730

Thanks for taking the time to send me this recommendation, but I think I've reached the end of the road (at least for me) on trying to get this to work. I've never compiled a custom kernel and have no idea how to go about doing all that, and this is actually a production system, so I'm really wary of doing anything overly exciting with the kernel, especially given my lack of experience.

I'm even wary of using this known good technique to get SR-IOV working on kernel 6.5.x because I'd have to re-compile the kernel for every point release and re-inject the custom stuff. https://www.derekseaman.com/2023/11...u-vt-d-passthrough-with-intel-alder-lake.html

See also this alternate implementation of the same technique: https://www.michaelstinkerings.org/gpu-virtualization-with-intel-12th-gen-igpu-uhd-730/

But, new Intel GPU drivers are supposed to be coming with 6.8 that are supposed to either implement proper SR-IOV support or set the groundwork for it in kernel 6.9, so I've decided to wait and see what we get in the release repos in a couple of months. If it still doesn't work after that, I'll revisit the hack technique above to get SR-IOV working ... assuming it actually works in kernel 6.8.