PCIE Passthrough 2 of 4 NVME drives on a 4x4 Card That is Bifurcated.

[Seed]

Curious if it’s possible to dice up a 4x4 nvme card and pass only 2 of them. They come up as isolated ids so assume I could but proxmox crashes when I try and start the vm. It works if I pass all 4 then I can see all 4 on the vm but I want to assigned a pair if I can.

 

[leesteken]

Curious if it’s possible to dice up a 4x4 nvme card and pass only 2 of them. They come up as isolated ids so assume I could but proxmox crashes when I try and start the vm. It works if I pass all 4 then I can see all 4 on the vm but I want to assigned a pair if I can.

NVMe are just PCIe devices, so check the IOMMU groups. You cannot share devices from the same group between VMs and/or the Proxmox host (because that would break memory isolation).

 

[Seed]

So these are different groups but whenever I try to pass one into a vm Proxmox locks me out of the UI then I have to reboot the host.

 

[leesteken]

So these are different groups but whenever I try to pass one into a vm Proxmox locks me out of the UI then I have to reboot the host.

View attachment 57803

What is the output of cat /proc/cmdline? If you using pcie_acs_override, please check the groups without it (and breaking/ignoring them might not work on the same bifurcated PCIe slot).

 

[Seed]

What is the output of cat /proc/cmdline? If you using pcie_acs_override, please check the groups without it (and breaking/ignoring them might not work on the same bifurcated PCIe slot).

initrd=\EFI\proxmox\6.2.16-19-pve\initrd.img-6.2.16-19-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt

 

[leesteken]

initrd=\EFI\proxmox\6.2.16-19-pve\initrd.img-6.2.16-19-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt

amd_iommu=on iommu=pt don't really do anything for your system. Can you show the entire IOMMU groups 15, 16, 17 and 18? Maybe there are other devices in those groups?

 

[Seed]

amd_iommu=on iommu=pt don't really do anything for your system. Can you show the entire IOMMU groups 15, 16, 17 and 18? Maybe there are other devices in those groups?

Should I dump em? I followed some guide somewhere on here awhile ago when i was on version 7 and it worked so didn't really think about it.

Sorted by ID:

IOMMU Group 14 c1:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bc] (rev a1)
IOMMU Group 15 c2:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 16 c3:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 17 c4:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 18 c5:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 19 c6:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]

 

[leesteken]

Should I dump em? I followed some guide somewhere on here awhile ago when i was on version 7 and it worked so didn't really think about it.

It does not matter, your choice.

Sorted by ID:

IOMMU Group 14 c1:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bc] (rev a1)
IOMMU Group 15 c2:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 16 c3:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 17 c4:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 18 c5:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU Group 19 c6:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]

Then I don't know what to do. PCI(e) passthrough is always trial and error and sometimes it just doesn't work.

Note that Crucial P2 uses QLC flash memory and will get slower than an old HDD when writing more than a little to it. Those are not good drives for VMs. Please search the forum for QLC problems and enterprise SSD with PLP suggestions. Sorry for bringing more bad news.

 

[Seed]

It does not matter, your choice.

Then I don't know what to do. PCI(e) passthrough is always trial and error and sometimes it just doesn't work.

Note that Crucial P2 uses QLC flash memory and will get slower than an old HDD when writing more than a little to it. Those are not good drives for VMs. Please search the forum for QLC problems and enterprise SSD with PLP suggestions. Sorry for bringing more bad news.

Thanks,

The VMs run on U2 NVMe drives. These are just added for random tasks. Ill keep experimenting but this is my main too so its a little rough.

 

[Seed]

Just wanted to follow up on this. I upgraded to 8.1 and wanted to try this again. With an ASUS 4x16 bifurcated nvme card, Im now able to pass individual iOMMU devices to different virtual machines enabling rom-bar and all functions. The proxmox UI would freeze if I tried to break these devices off before so this is great.