WD Black 250GB NVME Keeps crashing Proxmox

[KoldG33k]

Hello.
I have a Asrock X470D4U with Ryzen 2700X. with 64GB EEC memory
So I installed Proxmox and I pretty much can't do anything without TrueNAS Core which some stuff is on it. I tried to get it to work on Proxmox but I use NVME as a cache and proxmox is able to pass through 9211-8e with no problem but when i pass through NVME. It crashes Proxmox. I was able to boot up TrueNAS fine without NVME but with. It just crashes. I even manually add the NVME and it still crashes. I could not figure out why the NVMEs I have keep crash. I am lost and I looked every where for solution and only mentions mostly for GPU.

Thank you in advance.

(Edited)
Specs

Motherboard: Asrock X470D4U
CPU: 2700X
OS Drives: 2 x Transcend 32GB SSD
Cache Drives: 2 x WD Black NVME
Pool Drives: 4x4TB + 2 x 1TB
NIC : 2x 10GBPE

 

[leesteken]

I have a Asrock X470D4U with Ryzen 2700X. with 64GB EEC memory

The X470 chipset can only passthrough the first (SLOT6) and the second (SLOT4) x16 PCIe slot and only the one M.2 slot (all connected directly to the CPU and not the chipset). I fear that the ASRock X470D4U does not have a M.2 slot that is directly connected to the CPU (because of the "x2 PCIE 3.0 and x4 PCIe 2.0" remark).

I could not figure out why the NVMEs I have keep crash. I am lost and I looked every where for solution and only mentions mostly for GPU.

Check your IOMMU groups and make sure the NVMe drive is in a group without other PCIe devices that are needed for the Proxmox host ( as you cannot share devices in the same group between VMs and/or the host).

 

[KoldG33k]

The X470 chipset can only passthrough the first (SLOT6) and the second (SLOT4) x16 PCIe slot and only the one M.2 slot (all connected directly to the CPU and not the chipset). I fear that the ASRock X470D4U does not have a M.2 slot that is directly connected to the CPU (because of the "x2 PCIE 3.0 and x4 PCIe 2.0" remark).

Check your IOMMU groups and make sure the NVMe drive is in a group without other PCIe devices that are needed for the Proxmox host ( as you cannot share devices in the same group between VMs and/or the host).

Thank you for your response. That is what I seem to discovering about as I was told look at the manual and see what is connected to CPU and Chipset. I did check the IOMMU Groups and I'm still having difficulty deciphering on what is in the same group. Would that be under immougroup, correct?

 

[KoldG33k]

The X470 chipset can only passthrough the first (SLOT6) and the second (SLOT4) x16 PCIe slot and only the one M.2 slot (all connected directly to the CPU and not the chipset). I fear that the ASRock X470D4U does not have a M.2 slot that is directly connected to the CPU (because of the "x2 PCIE 3.0 and x4 PCIe 2.0" remark).

Check your IOMMU groups and make sure the NVMe drive is in a group without other PCIe devices that are needed for the Proxmox host ( as you cannot share devices in the same group between VMs and/or the host).

This Is the Screenshot of Iommu Group.

 

[leesteken]

This Is the Screenshot of Iommu Group.

As you can see, there are multiple additional devices in group 14 that contains your SN750, including the integrated graphics, a network controller, a SATA and a SAS controller. The Proxmox host probably needs most of those devices. As soon as you start a VM with passthrough of 2a:00.0 (of 26:00.0), Proxmox loses all devices in group 14 and crashes. You motherboard is not suitable for passthrough of devices the M.2 slots. The best chipset for passthrough with AM4 is X570(S), all other put almost everything in a big "chipset group" (as discussed many times on this forum already).

 

[KoldG33k]

As you can see, there are multiple additional devices in group 14 that contains your SN750, including the integrated graphics, a network controller, a SATA and a SAS controller. The Proxmox host probably needs most of those devices. As soon as you start a VM with passthrough of 2a:00.0 (of 26:00.0), Proxmox loses all devices in group 14 and crashes. You motherboard is not suitable for passthrough of devices the M.2 slots. The best chipset for passthrough with AM4 is X570(S), all other put almost everything in a big "chipset group" (as discussed many times on this forum already).

So I'm forced to get a new Board that have X570 chipeset basically. Only Asrock Rack still have server board capable of using AMD Ryzen AM4 or AM5.

 

[leesteken]

So I'm forced to get a new Board that have X570 chipeset basically. Only Asrock Rack still have server board capable of using AMD Ryzen AM4 or AM5.

This is a known limitation of every Ryzen chipset (except X570) since the introduction of Zen1 and I believe the groups for your motherboard have been shown on the internet,

Luckily, you can simply do disk passthrough instead or just use normal virtual disks instead of a whole NVMe drive. What is the purpose of breaking virtualization with passthrough of a drive anyway?

If you really don't care about VM isolation then you can use the pci_acs_override to make Proxmox basically ignore the IOMMU groups. But then the VM can in principle read and write all the memory of the Proxmox host (and other VMs).

 

[KoldG33k]

This is a known limitation of every Ryzen chipset (except X570) since the introduction of Zen1 and I believe the groups for your motherboard have been shown on the internet,

Luckily, you can simply do disk passthrough instead or just use normal virtual disks instead of a whole NVMe drive. What is the purpose of breaking virtualization with passthrough of a drive anyway?

If you really don't care about VM isolation then you can use the pci_acs_override to make Proxmox basically ignore the IOMMU groups. But then the VM can in principle read and write all the memory of the Proxmox host (and other VMs).

Yes. I use 2 NVMEs as my cache to help speed up read especially I set up iSCSI directly attached to my PC to use it for gaming and other purposes. I'm slowly learning to understand the architecture of motherboard with chipset now. So this makes sense.