Code 43 on Windows iGPU passthrough

[LameDave]

Hi! I'm on PVE 7.3 with a WIndows 11 install on an intel NUC, i5-1240p with integrated Iris XE graphics. After a lot of experimenting and today completely reinstalling PVE and going back to the start, I've got a seemingly stable Windows 11VM, that can see my GPU on passthrough, but stops it with a code 43 so I can't actually get to the point of Windows showing up on the local monitor, only over RDP. I successfuly installed the latest Intel driver for the VGA adapter, it seems happy when on "display=default" instead of "display=none".

VT-d is enabled on the bios. I'm using Grub (blue box when booting). IOMMU seems to be working, and the iGPU seems to be in its own group. Any ideas? I feel like I'm close!

Spoiler: pveversion -v:

proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-8
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

Spoiler: /etc/default/grub (you can see the other variants of this I've tried)

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init video=simplefbff video=vesafbff vi>
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifbff pcie_acs_override=downstream"

GRUB_CMDLINE_LINUX=""

Spoiler: /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Spoiler: /etc/modprobe.d/vfio.conf:

options vfio-pci ids=8086:46a6

Spoiler: /etc/modprobe.d/blacklist.conf:

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
blacklist i915

Spoiler: dmesg | grep IOMMU:

[ 0.055490] DMAR: IOMMU enabled
[ 0.126387] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.341062] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[ 0.968860] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.968860] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.968861] DMAR: IOMMU feature nwfs inconsistent
[ 0.968861] DMAR: IOMMU feature dit inconsistent
[ 0.968862] DMAR: IOMMU feature sc_support inconsistent
[ 0.968862] DMAR: IOMMU feature dev_iotlb_support inconsistent

Spoiler: vm conf file:

agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0;ide0
cores: 4
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:00:02,pcie=1
ide0: local:iso/virtio-win-0.1.225.iso,media=cdrom,size=519590K
ide2: local:iso/Win11_22H2_EnglishInternational_x64v1.iso,media=cdrom,size=5418024K
machine: pc-q35-7.1
memory: 8000
meta: creation-qemu=7.1.0,ctime=1670345874
name: windows2
net0: virtio=1A0:E8:98:CC:17,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: local-lvm:vm-102-disk-1,cache=writeback,discard=on,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=53f17861-5aa9-4242-8d4a-389fbb39f3f9
sockets: 1
tpmstate0: local-lvm:vm-102-disk-2,size=4M,version=v2.0
vga: none
vmgenid: bd6caec1-5610-473e-bc70-303ec69826b5

Spoiler: lspci -nnv | grep VGA

00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:46a6] (rev 0c) (prog-if 00 [VGA controller])

Spoiler: find /sys/kernel/iommu_groups/ -type l

/sys/kernel/iommu_groups/7/devices/0000:00:15.1
/sys/kernel/iommu_groups/7/devices/0000:00:15.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.5
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3
/sys/kernel/iommu_groups/11/devices/0000:00:1f.4
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:14.3
/sys/kernel/iommu_groups/4/devices/0000:00:0a.0
/sys/kernel/iommu_groups/12/devices/0000:01:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:06.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:17.0

 

[LameDave]

OK, so, having done a lot more reading, it seems like none of the 12th gen Intel iGPUs are supported for passthrough in the guest yet. I tried using DKMS inside an Ubuntu VM as detailed here, but the driver seemed to fail to start up. I also went for it and switched my whole proxmox install to kernel 5.19 and did the DKMS setup detailed here, which looked like it was working but didn't seem to actually populate the new virtual GPU PCI devices. So, back to headless for now I think, and wait to see if Intel release some new drivers...