Hello,
I am attempting to pass through 2 NVMe drives to two different VMs. They are on an Asus bifurcation card with a total of 4 m.2 slots, all of which are populated. My BIOS has the PCIe slot set to x4x4x4x4 and all of the drives are seen correctly in Proxmox. Hardware is listed at the bottom.
The Issue:
Proxmox crashes every time I reboot the VM running EndeavourOS with one of the NVMe drives passed through.
What I've Tried:
- Cycled through all 4 of the NVMe drives on the bifurcation card - Crashes on EndeavourOS VM reboot with any of them
- Passed through one of the NVMe drives to a Windows VM - DOES NOT CRASH
- Forced use of Vfio and prevented NVMe driver from being used on drives with vfio.conf in modprobe.d - Proxmox crashes when STARTING either Windows or EndeavourOS VM - This happened after confirmed the Vfio driver was being used.
Additional Details:
- When Proxmox crashes there is a PCIe error in the IPMI logs associated with NVMe drive I'm trying to pass through
- I previously had all 4 drives in a Striped ZFS mirror and there were occasional correctable PCIe errors according to
- The correctable PCIe errors happened on two different bifurcation cards.
OS and Hardware:
Proxmox 8.1.4
Motherboard - Asrock Rack Rome8d-2t
CPU - Epyc 7443p
RAM - 256GB 3200MHZ ECC
GPU1 - Asus TUF RTX-4070ti Super, PCIe7
GPU2 - Sapphire Pulse 7900 XTX, PCIe1
OS Drive - Samsung 980 Pro 500GB
VM OS Drives - ZFS Mirror 2x960GB Samsung P9A3, Oculink 1/2
VM Storage - 2x1TB WD SN850 and 2x 1TB WD SN850x, Asus bifurcation card PCIe5
TrueNAS Drives - 4x12TB Seagate x16, SATA 4-7
PCIe NIC - 82599ES 10Gbe - passed through to TrueNAS, PCIe4
I am attempting to pass through 2 NVMe drives to two different VMs. They are on an Asus bifurcation card with a total of 4 m.2 slots, all of which are populated. My BIOS has the PCIe slot set to x4x4x4x4 and all of the drives are seen correctly in Proxmox. Hardware is listed at the bottom.
The Issue:
Proxmox crashes every time I reboot the VM running EndeavourOS with one of the NVMe drives passed through.
What I've Tried:
- Cycled through all 4 of the NVMe drives on the bifurcation card - Crashes on EndeavourOS VM reboot with any of them
- Passed through one of the NVMe drives to a Windows VM - DOES NOT CRASH
- Forced use of Vfio and prevented NVMe driver from being used on drives with vfio.conf in modprobe.d - Proxmox crashes when STARTING either Windows or EndeavourOS VM - This happened after confirmed the Vfio driver was being used.
Additional Details:
- When Proxmox crashes there is a PCIe error in the IPMI logs associated with NVMe drive I'm trying to pass through
- I previously had all 4 drives in a Striped ZFS mirror and there were occasional correctable PCIe errors according to
dmesg. This seemed to happen under a heavy load or sometimes when booting Proxmox.- The correctable PCIe errors happened on two different bifurcation cards.
OS and Hardware:
Proxmox 8.1.4
Motherboard - Asrock Rack Rome8d-2t
CPU - Epyc 7443p
RAM - 256GB 3200MHZ ECC
GPU1 - Asus TUF RTX-4070ti Super, PCIe7
GPU2 - Sapphire Pulse 7900 XTX, PCIe1
OS Drive - Samsung 980 Pro 500GB
VM OS Drives - ZFS Mirror 2x960GB Samsung P9A3, Oculink 1/2
VM Storage - 2x1TB WD SN850 and 2x 1TB WD SN850x, Asus bifurcation card PCIe5
TrueNAS Drives - 4x12TB Seagate x16, SATA 4-7
PCIe NIC - 82599ES 10Gbe - passed through to TrueNAS, PCIe4