PCIe Passthrough NVMe Drive Causes Proxmox to Crash - Using Bifurcation Card

dizzydre21

Member
Apr 10, 2023
34
0
6
Hello,

I am attempting to pass through 2 NVMe drives to two different VMs. They are on an Asus bifurcation card with a total of 4 m.2 slots, all of which are populated. My BIOS has the PCIe slot set to x4x4x4x4 and all of the drives are seen correctly in Proxmox. Hardware is listed at the bottom.

The Issue:
Proxmox crashes every time I reboot the VM running EndeavourOS with one of the NVMe drives passed through.

What I've Tried:
- Cycled through all 4 of the NVMe drives on the bifurcation card - Crashes on EndeavourOS VM reboot with any of them
- Passed through one of the NVMe drives to a Windows VM - DOES NOT CRASH
- Forced use of Vfio and prevented NVMe driver from being used on drives with vfio.conf in modprobe.d - Proxmox crashes when STARTING either Windows or EndeavourOS VM - This happened after confirmed the Vfio driver was being used.

Additional Details:
- When Proxmox crashes there is a PCIe error in the IPMI logs associated with NVMe drive I'm trying to pass through
- I previously had all 4 drives in a Striped ZFS mirror and there were occasional correctable PCIe errors according to dmesg. This seemed to happen under a heavy load or sometimes when booting Proxmox.
- The correctable PCIe errors happened on two different bifurcation cards.


OS and Hardware:
Proxmox 8.1.4
Motherboard - Asrock Rack Rome8d-2t
CPU - Epyc 7443p
RAM - 256GB 3200MHZ ECC
GPU1 - Asus TUF RTX-4070ti Super, PCIe7
GPU2 - Sapphire Pulse 7900 XTX, PCIe1
OS Drive - Samsung 980 Pro 500GB
VM OS Drives - ZFS Mirror 2x960GB Samsung P9A3, Oculink 1/2
VM Storage - 2x1TB WD SN850 and 2x 1TB WD SN850x, Asus bifurcation card PCIe5
TrueNAS Drives - 4x12TB Seagate x16, SATA 4-7
PCIe NIC - 82599ES 10Gbe - passed through to TrueNAS, PCIe4
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!