RTX4060Ti GPU passthrough VM, won't start after VM reset.

[masato]

Regarding RTX4060Ti pass-through operation.

When repeatedly starting and shutting down a Windows 11 VM, the GPU can be passed through successfully and the VM can be started and shut down continuously.
When resetting a running VM using the Proxmox GUI or a command, the UEFI screen is not displayed and the VM will not start.
To start the VM, a reboot of the host machine is required.

Proxmox VE:9.1.1
Motherboard: GIGABYTE X870E AORUS ELITE WiFi7
CPU: RYzen9 9950X
GPU: Nvidia RTX4060Ti
Memory: 128GB

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on ipcie_acs_override=downstream,multifunction acpi_enforce_resources=lax vfio-pci.ids=10de:2803,10de:22bd"

/etc/modules

vfio
vfio-pci
vfio_iommu_type1

/etc/modprobe.d/vfio.conf

options vfio-pci disable_vga=1 disable_idle_d3=1
options vfio-pci ids=10de:2803,10de:22bd
options vfio_iommu_type1 allow_unsafe_interrupts=1
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

/etc/modprobe.d/kvm.conf

options kvm nested=1
options kvm ignore_msrs=on
options kvm report_ignored_msrs=off

/etc/modprobe.d/blacklist-vfio.conf

blacklist snd_hda_intel
blacklist nouveau
blacklist nvidia
blacklist radeon

/etc/modprobe.d/pve-blacklist.conf

blacklist nvidiafb

/etc/pve/qemu-server/1001001.conf

agent: 1
bios: ovmf
boot: order=scsi0;ide2;ide0;net0
cores: 16
cpu: custom-host-fix
efidisk0: local-zfs:vm-1001001-disk-0,efitype=4m,ms-cert=2023,pre-enrolled-keys=1,size=1M
hookscript: local:snippets/pcie_hostdev.sh
hostpci0: 0000:01:00,pcie=1,x-vga=1,romfile=local/GeForce_RTX_4060Ti_Msi_Ventus_2x_Black_8G_OC.rom
hostpci1: 0000:7b:00.3,pcie=1,rombar=0
hostpci2: 0000:7b:00.4,pcie=1,rombar=0
hostpci3: 0000:13:00.0,pcie=1,rombar=0
ide0: local-nfs:iso/virtio-win-0.1.271.iso,media=cdrom,size=709474K
ide2: local-nfs:iso/Win11_24H2_Japanese_x64.iso,media=cdrom,size=5616576K
machine: pc-q35-10.1,viommu=virtio
memory: 32768
meta: creation-qemu=10.0.2,ctime=1762668878
name: PC-X870E-AORUS
net0: virtio=BC:24:11:FD:57:8C,bridge=vmbr0
numa: 0
ostype: win11
parent: Setup_Finish
scsi0: local-zfs:vm-1001001-disk-1,iothread=1,size=2000G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=03ff0210-04e0-05cc-5206-ba0700080009,manufacturer=R2lnYWJ5dGUgVGVjaG5vbG9neSBDby4sIEx0ZC4=,product=ODcwRSBBT1JVUyBFTElURSBXSUZJNw==,family=WDg3MEUgTUI=,base64=1
sockets: 1
tpmstate0: local-zfs:vm-1001001-disk-2,size=4M,version=v2.0
vga: none
vmgenid: f17c24c7-6d0c-454e-beb9-95ac4bcc4015

Processing in hook scripts

# PCI-E Remove
for device in /sys/bus/pci/devices/${DEVICE_ID}; do
    # Device Remove
    echo 1 > ${device}/remove
    # PCI-e Device Rescan
    echo "1" > /sys/bus/pci/rescan
done

dmesg

[ 5514.658140] vfio-pci 0000:01:00.0: resetting
[ 5514.658181] vfio-pci 0000:01:00.1: resetting
[ 5514.906213] vfio-pci 0000:01:00.0: reset done
[ 5514.906245] vfio-pci 0000:01:00.1: reset done
[ 5514.911545] vfio-pci 0000:13:00.0: resetting
[ 5515.018133] vfio-pci 0000:13:00.0: reset done
[ 5515.018431] vfio-pci 0000:7b:00.4: resetting
[ 5515.041914] vfio-pci 0000:7b:00.4: reset done
[ 5515.042183] vfio-pci 0000:7b:00.3: resetting
[ 5515.066706] vfio-pci 0000:7b:00.3: reset done
[ 5515.067373] vfio-pci 0000:01:00.0: resetting
[ 5515.170191] vfio-pci 0000:01:00.0: reset done