windows 11 VM crashing during benchmark

[bheem2021]

I have gpu passthrough working on a fresh install of proxmox ve 7.2-7. The vm boot up I can login and use like normal but when I run a benchmark or game the VM just immediately crashes.

VM: config:
agent: 1
bios: ovmf
boot:
cores: 16
cpu: host
efidisk0: local-zfs:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:16:00,pcie=1,x-vga=1
hostpci1: 0000:13:00,pcie=1
ide0: local-zfs:vm-102-disk-1,size=256G
machine: pc-q35-6.2
memory: 16384
meta: creation-qemu=6.2.0,ctime=1659007888
name: Loki
net0: virtio=2E:42:5C:1F:55:CC,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsihw: virtio-scsi-pci
smbios1: uuid=15b2fb98-ec0d-437c-8751-936e4029ba05
sockets: 1
tpmstate0: local-zfs:vm-102-disk-2,size=4M,version=v2.0
vga: virtio
vmgenid: dc9edfdf-2c13-4154-9e7d-30576b604b1e

lspci -v output:
16:00.0 VGA compatible controller: NVIDIA Corporation Device 2208 (rev ff) (prog-if ff)
!!! Unknown header type 7f
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau

16:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev ff) (prog-if ff)
!!! Unknown header type 7f
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

 

[fiona]

Hi,
please post the output of pveversion -v and an excerpt of /var/log/syslog from around the time the crash happens.
The output of the following might also be interesting:

find /sys/kernel/iommu_groups/ -type l | sort -V
lspci

 

[bheem2021]

logs:
Aug 8 07:12:08 pve systemd[32003]: Startup finished in 66ms.
Aug 8 07:12:08 pve systemd[1]: Started User Manager for UID 0.
Aug 8 07:12:08 pve systemd[1]: Started Session 3 of user root.
Aug 8 07:12:10 pve kernel: [ 275.106345] vfio-pci 0000:16:00.0: timed out waiting for pending transactio>
Aug 8 07:12:11 pve kernel: [ 276.354312] vfio-pci 0000:16:00.0: not ready 1023ms after FLR; waiting
Aug 8 07:12:12 pve kernel: [ 277.410310] vfio-pci 0000:16:00.0: not ready 2047ms after FLR; waiting
Aug 8 07:12:14 pve kernel: [ 279.650302] vfio-pci 0000:16:00.0: not ready 4095ms after FLR; waiting
Aug 8 07:12:19 pve kernel: [ 284.002330] vfio-pci 0000:16:00.0: not ready 8191ms after FLR; waiting
Aug 8 07:12:21 pve pvestatd[2572]: VM 102 qmp command failed - VM 102 qmp command 'query-proxmox-support'>
Aug 8 07:12:21 pve pvestatd[2572]: status update time (6.153 seconds)
Aug 8 07:12:27 pve kernel: [ 292.450269] vfio-pci 0000:16:00.0: not ready 16383ms after FLR; waiting
Aug 8 07:12:31 pve pvestatd[2572]: VM 102 qmp command failed - VM 102 qmp command 'query-proxmox-support'>
Aug 8 07:12:31 pve pvestatd[2572]: status update time (6.335 seconds)

cat /proc/cmdline:
initrd=\EFI\proxmox\5.15.39-3-pve\initrd.img-5.15.39-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt pcie_no_flr=1022:0300,1022:0403 vfio_iommu_type1.allow_unsafe_interrupts=1 video=astdrmfb video=vesafbff video=efifbff video=simplefbff

output of pvevesion -v:
proxmox-ve: 7.2-1 (running kernel: 5.15.39-3-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-8
pve-kernel-helper: 7.2-8
pve-kernel-5.15.39-3-pve: 5.15.39-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1
output of
find /sys/kernel/iommu_groups/ -type l | sort -V
lspci:
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/7/devices/0000:00:05.0
/sys/kernel/iommu_groups/8/devices/0000:00:07.0
/sys/kernel/iommu_groups/9/devices/0000:00:07.1
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/11/devices/0000:00:08.1
/sys/kernel/iommu_groups/12/devices/0000:00:14.0
/sys/kernel/iommu_groups/12/devices/0000:00:14.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.2
/sys/kernel/iommu_groups/13/devices/0000:00:18.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.4
/sys/kernel/iommu_groups/13/devices/0000:00:18.5
/sys/kernel/iommu_groups/13/devices/0000:00:18.6
/sys/kernel/iommu_groups/13/devices/0000:00:18.7
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/15/devices/0000:02:00.0
/sys/kernel/iommu_groups/16/devices/0000:03:02.0
/sys/kernel/iommu_groups/17/devices/0000:03:03.0
/sys/kernel/iommu_groups/18/devices/0000:03:04.0
/sys/kernel/iommu_groups/19/devices/0000:03:05.0
/sys/kernel/iommu_groups/20/devices/0000:03:06.0
/sys/kernel/iommu_groups/21/devices/0000:03:08.0
/sys/kernel/iommu_groups/21/devices/0000:13:00.0
/sys/kernel/iommu_groups/21/devices/0000:13:00.1
/sys/kernel/iommu_groups/21/devices/0000:13:00.3
/sys/kernel/iommu_groups/22/devices/0000:03:09.0
/sys/kernel/iommu_groups/22/devices/0000:14:00.0
/sys/kernel/iommu_groups/23/devices/0000:03:0a.0
/sys/kernel/iommu_groups/23/devices/0000:15:00.0
/sys/kernel/iommu_groups/24/devices/0000:04:00.0
/sys/kernel/iommu_groups/25/devices/0000:05:00.0
/sys/kernel/iommu_groups/26/devices/0000:06:00.0
/sys/kernel/iommu_groups/27/devices/0000:07:00.0
/sys/kernel/iommu_groups/28/devices/0000:08:01.0
/sys/kernel/iommu_groups/28/devices/0000:09:00.0
/sys/kernel/iommu_groups/28/devices/0000:09:00.1
/sys/kernel/iommu_groups/29/devices/0000:08:03.0
/sys/kernel/iommu_groups/30/devices/0000:0b:00.0
/sys/kernel/iommu_groups/30/devices/0000:0b:00.1
/sys/kernel/iommu_groups/30/devices/0000:08:05.0
/sys/kernel/iommu_groups/31/devices/0000:0c:00.0
/sys/kernel/iommu_groups/31/devices/0000:0c:00.1
/sys/kernel/iommu_groups/31/devices/0000:08:07.0
/sys/kernel/iommu_groups/32/devices/0000:0d:00.0
/sys/kernel/iommu_groups/33/devices/0000:0e:01.0
/sys/kernel/iommu_groups/33/devices/0000:0f:00.0
/sys/kernel/iommu_groups/33/devices/0000:0f:00.1
/sys/kernel/iommu_groups/34/devices/0000:0e:03.0
/sys/kernel/iommu_groups/35/devices/0000:0e:05.0
/sys/kernel/iommu_groups/35/devices/0000:11:00.0
/sys/kernel/iommu_groups/35/devices/0000:11:00.1
/sys/kernel/iommu_groups/36/devices/0000:0e:07.0
/sys/kernel/iommu_groups/36/devices/0000:12:00.0
/sys/kernel/iommu_groups/36/devices/0000:12:00.1
/sys/kernel/iommu_groups/37/devices/0000:16:00.0
/sys/kernel/iommu_groups/37/devices/0000:16:00.1
/sys/kernel/iommu_groups/38/devices/0000:17:00.0
/sys/kernel/iommu_groups/39/devices/0000:18:00.0
/sys/kernel/iommu_groups/40/devices/0000:18:00.1
/sys/kernel/iommu_groups/41/devices/0000:18:00.3
/sys/kernel/iommu_groups/42/devices/0000:18:00.4
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
01:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 5407
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
04:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
05:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10)
06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
07:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
08:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
08:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
08:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
08:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
09:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
0b:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] (rev a1)
0b:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
0c:00.0 VGA compatible controller: NVIDIA Corporation Device 2504 (rev a1)
0c:00.1 Audio device: NVIDIA Corporation Device 228e (rev a1)
0d:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
0e:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
0e:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
0e:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
0e:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port
0f:00.0 VGA compatible controller: NVIDIA Corporation Device 2504 (rev a1)
0f:00.1 Audio device: NVIDIA Corporation Device 228e (rev a1)
11:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] (rev a1)
11:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
12:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] (rev a1)
12:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
13:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
13:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
13:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
14:00.0 RAID bus controller: Advanced Micro Devices, Inc. [AMD] Device 7916 (rev 51)
15:00.0 RAID bus controller: Advanced Micro Devices, Inc. [AMD] Device 7916 (rev 51)
16:00.0 VGA compatible controller: NVIDIA Corporation Device 2208 (rev a1)
16:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
17:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
18:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
18:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
18:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
18:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

 

[fiona]

This thread is about a similar issue with helpful information. Do you have the latest BIOS version installed? What mother board do you have? Does the GPU support pass-through?

 

[bheem2021]

not sure if motherboard supports pcie passthrough. it does have iommu support and I have pcie passthrough working on 1x slot with risers to a different VM.
bios version: f36c
specs:
motherboard: gigabyte x570 aorus elite wi-fi;
processor: amd ryzen 9 5950x;
first pcie 16x slot: EVGA rtx-3080ti pluged
second pcie x16 slot: usb card not passedthrough
third pcie 1x slot: pcie 1x to 4 gpu
fourth pcie 1x: pcie 1x to 4 gpu

 

[fiona]

Did you already follow the guide on the Proxmox VE wiki? There's also some NVIDIA-specific tips in there.

 

[bheem2021]

yes
I didn't use the rom file