VM GPU Passthrough Proxmox Crash on Shutdown

[gtk1]

I've successfully passed though the GPU to Ubuntu through a few logs, but I'm having issues when shutting down the VM. I can reboot just fine, whether I reboot Proxmox or Ubuntu, and doesn't matter if I reboot Ubuntu through Proxmox or in the VM, it works. It's only shutdown. After about 5 seconds post shutdown, the Proxmox UI hangs and then connection is lost and isn't restored. I've looked and looked at all the configs, most of what I see is that initcall_blacklist=sysfb_init is most imporant, and the video off's don't matter, and it works for most, but I've tried everything I can find and think of. I'm assuming it's GPU but I'm new to Proxmox so, no really sure tbh.

Specs:
Intel i7-14700K
ASUS TUF Z790-Plus WiFi
RTX 5070ti

Spoiler: /etc/default/grub

# If you change this file, run 'update-grub' afterwards to update

# /boot/grub/grub.cfg.

# For full documentation of the options in this file, see:

#   info -f grub -n 'Simple configuration'



GRUB_DEFAULT=0

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`

#GRUB_CMDLINE_LINUX_DEFAULT="quiet"

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=realloc vfio-pci.ids=10de:2c05,10de:22e9 initcall_blacklist=sysfb_init video=vesafb:off video=efifb:off video=simplefb:off"

GRUB_CMDLINE_LINUX=""



# If your computer has multiple operating systems installed, then you

# probably want to run os-prober. However, if your computer is a host

# for guest OSes installed via LVM or raw disk devices, running

# os-prober can cause damage to those guest OSes as it mounts

# filesystems to look for things.

#GRUB_DISABLE_OS_PROBER=false



# Uncomment to enable BadRAM filtering, modify to suit your needs

# This works with Linux (no patch required) and with any kernel that obtains

# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)

#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"



# Uncomment to disable graphical terminal

#GRUB_TERMINAL=console



# The resolution used on graphical terminal

# note that you can use only modes which your graphic card supports via VBE

# you can see them in real GRUB with the command `vbeinfo'

#GRUB_GFXMODE=640x480



# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux

#GRUB_DISABLE_LINUX_UUID=true



# Uncomment to disable generation of recovery mode menu entries

#GRUB_DISABLE_RECOVERY="true"



# Uncomment to get a beep at grub start

#GRUB_INIT_TUNE="480 440 1"

Spoiler: /etc/modprobe.d/pve-blacklist.conf

blacklist nouveau

blacklist nvidia

blacklist nvidiafb

Spoiler: /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:2c05,10de:22e9 disable_vga=1 disable_idle_d3=1
#options vfio-pci ids=8086:a780

Spoiler: /etc/modules

# /etc/modules: kernel modules to load at boot time.

#

# This file contains the names of kernel modules that should be loaded

# at boot time, one per line. Lines beginning with "#" are ignored.

# Parameters can be specified after the module name.

vfio

vfio_iommu_type1

vfio_pci

vfio_virqfd

Spoiler: /etc/pve/qemu-server/100.conf

agent: 1

balloon: 0

bios: ovmf

boot: order=virtio0;ide2;net0

cores: 22

cpu: host

efidisk0: fast-vm-pool:100/vm-100-disk-0.raw,efitype=4m,size=528K

hostpci0: 01:00.0,pcie=1,rombar=0

hostpci1: 01:00.1,pcie=1

ide2: none,media=cdrom

machine: q35

memory: 81920

meta: creation-qemu=9.2.0,ctime=1745644353

name: clara-core

net0: virtio=BC:24:11:36:17:B5,bridge=vmbr0,firewall=1

numa: 0

ostype: l26

scsi0: docker-apps:vm-100-disk-0,backup=0,discard=on,iothread=1,size=1800G,ssd=1

scsihw: virtio-scsi-single

smbios1: uuid=3b3d02f8-34ae-46dc-b5e1-77a937aa3c03

sockets: 1

unused0: local-zfs:vm-100-disk-0

unused1: local-zfs:vm-100-disk-1

virtio0: fast-vm-pool:100/vm-100-disk-1.qcow2,cache=writeback,discard=on,iothread=1,size=1500G

vmgenid: 742d1006-8e92-4db0-8924-d521581ce720

#hookscript: local:snippets/100-prestop-hook.sh

hookscript: local:snippets/100-poststop-hook.sh

args: -rtc base=localtime,clock=host -no-shutdown

Spoiler: /var/lib/vz/snippets/100-poststop-hook.sh

#!/bin/bash



if [[ "$2" == "post-stop" ]]; then

  echo " Cleaning up GPU passthrough remnants"



  echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove || true

  echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove || true

  sleep 1

  echo 1 > /sys/bus/pci/rescan



  echo 0 > /sys/class/vtconsole/vtcon0/bind || true

  echo 0 > /sys/class/vtconsole/vtcon1/bind || true



  echo simple-framebuffer.0 > /sys/bus/platform/drivers/simple-framebuffer/unbind || true

fi

Spoiler: /proc/cmdline

initrd=\EFI\proxmox\6.8.12-10-pve\initrd.img-6.8.12-10-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=realloc vfio-pci.ids=10de:2c05,10de:22e9 initcall_blacklist=sysfb_init video=vesafb:off video=efifb:off video=simplefb:off kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 split_lock_detect=off

Spoiler: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770] [8086:a780] (rev 04)

IOMMU group 10 00:17.0 SATA controller [0106]: Intel Corporation Raptor Lake SATA AHCI Controller [8086:7a62] (rev 11)

IOMMU group 11 00:1a.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a48] (rev 11)

IOMMU group 12 00:1b.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a40] (rev 11)

IOMMU group 13 00:1b.4 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a44] (rev 11)

IOMMU group 14 00:1c.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a38] (rev 11)

IOMMU group 15 00:1c.2 PCI bridge [0604]: Intel Corporation Raptor Point-S PCH - PCI Express Root Port 3 [8086:7a3a] (rev 11)

IOMMU group 16 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:7a3c] (rev 11)

IOMMU group 17 00:1d.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a30] (rev 11)

IOMMU group 18 00:1f.0 ISA bridge [0601]: Intel Corporation Raptor Lake LPC/eSPI Controller [8086:7a04] (rev 11)

IOMMU group 18 00:1f.3 Audio device [0403]: Intel Corporation Raptor Lake High Definition Audio Controller [8086:7a50] (rev 11)

IOMMU group 18 00:1f.4 SMBus [0c05]: Intel Corporation Raptor Lake-S PCH SMBus Controller [8086:7a23] (rev 11)

IOMMU group 18 00:1f.5 Serial bus controller [0c80]: Intel Corporation Raptor Lake SPI (flash) Controller [8086:7a24] (rev 11)

IOMMU group 19 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2c05] (rev a1)

IOMMU group 19 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22e9] (rev a1)

IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation Raptor Lake-S 8+12 - Host Bridge/DRAM Controller [8086:a740] (rev 01)

IOMMU group 20 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d]

IOMMU group 21 05:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5027-E27T PCIe4 NVMe Controller (DRAM-less) [1987:5027] (rev 01)

IOMMU group 22 07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 06)

IOMMU group 23 08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)

IOMMU group 2 00:01.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express 5.0 Graphics Port (PEG010) [8086:a70d] (rev 01)

IOMMU group 3 00:06.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCIe 4.0 Graphics Port [8086:a74d] (rev 01)

IOMMU group 4 00:0a.0 Signal processing controller [1180]: Intel Corporation Raptor Lake Crashlog and Telemetry [8086:a77d] (rev 01)

IOMMU group 5 00:0e.0 RAID bus controller [0104]: Intel Corporation Volume Management Device NVMe RAID Controller Intel Corporation [8086:a77f]

IOMMU group 6 00:14.0 USB controller [0c03]: Intel Corporation Raptor Lake USB 3.2 Gen 2x2 (20 Gb/s) XHCI Host Controller [8086:7a60] (rev 11)

IOMMU group 6 00:14.2 RAM memory [0500]: Intel Corporation Raptor Lake-S PCH Shared SRAM [8086:7a27] (rev 11)

IOMMU group 7 00:14.3 Network controller [0280]: Intel Corporation Raptor Lake-S PCH CNVi WiFi [8086:7a70] (rev 11)

IOMMU group 8 00:15.0 Serial bus controller [0c80]: Intel Corporation Raptor Lake Serial IO I2C Host Controller [8086:7a4c] (rev 11)

IOMMU group 8 00:15.1 Serial bus controller [0c80]: Intel Corporation Raptor Lake Serial IO I2C Host Controller [8086:7a4d] (rev 11)

IOMMU group 8 00:15.2 Serial bus controller [0c80]: Intel Corporation Raptor Lake Serial IO I2C Host Controller [8086:7a4e] (rev 11)

IOMMU group 9 00:16.0 Communication controller [0780]: Intel Corporation Raptor Lake CSME HECI [8086:7a68] (rev 11)

Spoiler: lspci -nn

00:00.0 Host bridge [0600]: Intel Corporation Raptor Lake-S 8+12 - Host Bridge/DRAM Controller [8086:a740] (rev 01)

00:01.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express 5.0 Graphics Port (PEG010) [8086:a70d] (rev 01)

00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770] [8086:a780] (rev 04)

00:06.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCIe 4.0 Graphics Port [8086:a74d] (rev 01)

00:0a.0 Signal processing controller [1180]: Intel Corporation Raptor Lake Crashlog and Telemetry [8086:a77d] (rev 01)

00:0e.0 RAID bus controller [0104]: Intel Corporation Volume Management Device NVMe RAID Controller Intel Corporation [8086:a77f]

00:14.0 USB controller [0c03]: Intel Corporation Raptor Lake USB 3.2 Gen 2x2 (20 Gb/s) XHCI Host Controller [8086:7a60] (rev 11)

00:14.2 RAM memory [0500]: Intel Corporation Raptor Lake-S PCH Shared SRAM [8086:7a27] (rev 11)

00:14.3 Network controller [0280]: Intel Corporation Raptor Lake-S PCH CNVi WiFi [8086:7a70] (rev 11)

00:15.0 Serial bus controller [0c80]: Intel Corporation Raptor Lake Serial IO I2C Host Controller [8086:7a4c] (rev 11)

00:15.1 Serial bus controller [0c80]: Intel Corporation Raptor Lake Serial IO I2C Host Controller [8086:7a4d] (rev 11)

00:15.2 Serial bus controller [0c80]: Intel Corporation Raptor Lake Serial IO I2C Host Controller [8086:7a4e] (rev 11)

00:16.0 Communication controller [0780]: Intel Corporation Raptor Lake CSME HECI [8086:7a68] (rev 11)

00:17.0 SATA controller [0106]: Intel Corporation Raptor Lake SATA AHCI Controller [8086:7a62] (rev 11)

00:1a.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a48] (rev 11)

00:1b.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a40] (rev 11)

00:1b.4 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a44] (rev 11)

00:1c.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a38] (rev 11)

00:1c.2 PCI bridge [0604]: Intel Corporation Raptor Point-S PCH - PCI Express Root Port 3 [8086:7a3a] (rev 11)

00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:7a3c] (rev 11)

00:1d.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express Root Port [8086:7a30] (rev 11)

00:1f.0 ISA bridge [0601]: Intel Corporation Raptor Lake LPC/eSPI Controller [8086:7a04] (rev 11)

00:1f.3 Audio device [0403]: Intel Corporation Raptor Lake High Definition Audio Controller [8086:7a50] (rev 11)

00:1f.4 SMBus [0c05]: Intel Corporation Raptor Lake-S PCH SMBus Controller [8086:7a23] (rev 11)

00:1f.5 Serial bus controller [0c80]: Intel Corporation Raptor Lake SPI (flash) Controller [8086:7a24] (rev 11)

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2c05] (rev a1)

01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22e9] (rev a1)

02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d]

05:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5027-E27T PCIe4 NVMe Controller (DRAM-less) [1987:5027] (rev 01)

07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 06)

08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)

edit:
didn't know there was a difference in all these old configs, I'm using UEFI with systemd-boot, not Grub, apparently?

Spoiler: efibootmgr -v

BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,0001
Boot0000* Linux Boot Manager    HD(2,GPT,b25eb29f-d367-44af-a17e-a44b30f4aeed,0x800,0x200000)/File(\EFI\systemd\systemd-bootx64.efi)
Boot0001* UEFI OS       HD(2,GPT,b25eb29f-d367-44af-a17e-a44b30f4aeed,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

Spoiler: proxmox-boot-tool status

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
6E59-BE4B is configured with: uefi (versions: 6.8.12-10-pve, 6.8.12-9-pve)

Spoiler: /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci=realloc vfio-pci.ids=10de:2c05,10de:22e9 initcall_blacklist=sysfb_init video=vesafb:off video=efifb:off video=simplefb:off kvm.ignore_msrs=1 kvm.report_ignored_msrs=0 split_lock_detect=off

 

[gtk1]

So, while this is waiting admin approval, I'm probably just going to bail proxmox, but it would be interesting to know the issue. I ended up having these errors as well:

vfio_container_dma_map(0x5acadf6f7280, 0x380000000000, 0x400000000, 0x6ff280000000) = -22 (Invalid argument)

a lot of them, like crazy. I don't have a huge need to run more than ubuntu, so I'm just moving to straight ubuntu install.

Again if anyone doesn't mind pointing out an issue if it's obvious, neato and ty!