Success!! TLDR: add a hook script to clear the reset_methods at VM pre-start
catting the reset_methods gave
flr bus, the 2 methods not working above. However clearing it completely allowed the VM to boot and see the attached dGPU and its Audio controller:
Code:
echo > /sys/bus/pci/devices/0000:03:00.0/reset_method
echo > /sys/bus/pci/devices/0000:04:00.0/reset_method
The commands above would only work for the 1st attempt after boot (as suggested above). Subsequent VM starts would lock up due to vfio bus resets (but not FLR??), even though the reset_method still appeared to be cleared in sysfs.
Subsequent VM starts appear to be successful if clearing reset_methods as per above is run prior to each VM restart. Hence the hook script listed below.
Observed the following working:
- Installed latest WHQL ARC drivers
- GPU-Z detects resizable bar enabled for the ARC, but Intel driver detects resizable bar not enabled in the VM BIOS (which I think is something still to be developed in QEMU).
- HDMI output via the Intel dGPU
- dGPU Audio device detected (but haven't tested)
- For fun,
- Passthrough of WIFI (PCI:
0000:00:14.3) and BlueTooth (USB: 8087:0033).
- Youtube/Edge video decode via dGPU
- Blender running and using dGPU
- Hot-plugged wireless USB mouse/keyboard.
- Ran another Win11 VM using a SR-IOV iGPU VF at the same time.
Note: Due to above success, I didn't end applying the softdeps.
hookscript (based on template from
https://gist.github.com/kiler129/215e2c8de853209ca429ad5ed40ce128):
Code:
#!/bin/bash
set -e -o errexit -o pipefail -o nounset
# located in /var/lib/vz/snippets/intel-dGPU-hookscript.sh or your 'snippets' location.
# Add to VM via: qm set VMID --hookscript local:snippets/intel-dGPU-hookscript.sh
# Do not modify these variables (set by Proxmox when calling the script)
vmId="$1"
runPhase="$2"
echo "Running $runPhase on VM=$vmId"
case "$runPhase" in
pre-start)
# Clear the reset methods before each start of the VM, to prevent the PVE host locking up
# flr and bus methods dont work and re-appear after reboots.
# bus method may still be attempted in subseqent VM starts, even if reset_method already cleared.
echo "VM=$vmId - $runPhase : Clearing Intel dGPU and audio device reset_methods."
echo > /sys/bus/pci/devices/0000:03:00.0/reset_method
echo > /sys/bus/pci/devices/0000:04:00.0/reset_method
# will appear as the following in journalctl:
# kernel: vfio-pci 0000:03:00.0: All device reset methods disabled by user
# kernel: vfio-pci 0000:04:00.0: All device reset methods disabled by user
;;
post-start)
# placeholder .
echo "VM=$vmId - $runPhase : No Action."
;;
pre-stop)
# placeholder .
echo "VM=$vmId - $runPhase : No Action."
;;
post-stop)
# placeholder .
echo "VM=$vmId - $runPhase : No Action."
;;
*)
echo "Unknown run phase \"$runPhase\"!"
;;
esac
echo "Finished $runPhase on VM=$vmId"
Working VM config:
Code:
root@nuc12:~# cat /etc/pve/qemu-server/103.conf
affinity: 0-11
agent: 1
balloon: 8192
bios: ovmf
boot: order=virtio0;net0
cores: 6
cpu: host
efidisk0: local-zfs:vm-103-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hookscript: local:snippets/intel-dGPU-hookscript.sh
hostpci0: 0000:03:00,pcie=1,x-vga=1
hostpci1: 0000:04:00,pcie=1
hostpci2: 0000:00:14.3,pcie=1
machine: pc-q35-8.0
memory: 16384
meta: creation-qemu=8.0.2,ctime=1688796354
name: win11vfio
net0: virtio=4A:3D:7D:C5:35:B8,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=d421488c-841e-43fa-90d9-33fe4349eb61
sockets: 1
tpmstate0: local-zfs:vm-103-disk-1,size=4M,version=v2.0
usb0: host=1997:2433,usb3=1
usb1: host=8087:0033,usb3=1
virtio0: local-zfs:vm-103-disk-2,iothread=1,size=256G
vmgenid: 1873c979-4993-4627-aa05-20bb0ac28468
. I'm still on fresh Vm with one driver back, still crashing in random intervals. Gotta try going back further. Quick Q: did you downgrade flash the card in a vm with passthough or baremetal? (Guess it shouldn't make a difference when vbios dumped the same checksum, but curious.) And, do you use a dummy plug? If yes hdmi or dp?