16 PCI Device limitation

[bpinjc]

Hello,

I have installed ProxMox on a 24 NVME U.2 system and have configured TrueNAS as a VM with passthrough for the NVME drives. The challenge is that I have 21 drives and ProxMox supports only 16 PCI Devices attached to a VM in passthrough mode. I need to see all 21 drives. Right now I am attaching each drive individually to the VM as a PCI Device.

I played around with the "qm set" approach, however when I pulled a drive to emulate failure, the whole VM froze. I did read something about a parameter that may prevent this from happening, but I have to dig more into that. Ultimately, this is not my preferred method.

My question is, can I use an HBA that supports multiple NVME U.2 drives so that by adding a single HBA PCI Device, I can recognize an additional 6-8 drives using only one PCI Device attachment in the VM Hardware Configuration?

Better yet, is there any way to bypass the 16 PCI device limit?

Appreciate any responses.

 

[dcsapak]

My question is, can I use an HBA that supports multiple NVME U.2 drives so that by adding a single HBA PCI Device, I can recognize an additional 6-8 drives using only one PCI Device attachment in the VM Hardware Configuration?

i don't think that will make a difference, since the drives will still show up as pci devices and the hba is "just" a multiplexer... (though i could be wrong of course)

Better yet, is there any way to bypass the 16 PCI device limit?

no, currently there isn't without modifying the code

you can open a feature request here: https://bugzilla.proxmox.com to increase the number of pci slots though. if you do, maybe also describe the use case a little bit. increasing shouldn't be hard, but it needs modification at a few places in the back- and frontend

 

[bpinjc]

Dominik, Thank you for your response. I will file a report. Ultimately there is a fairly common use-case to run Proxmox with TrueNAS as a VM with Passthrough drives. TrueNAS offers advanced tuning of ZFS parameters and functions with a solid user interface and reporting.

In the past 8 or more SATA/SAS drives would be recognized by merely adding a single PCI HBA. With new NVME U.2 / E.1 SSD these technologies are directly recognized as individual PCI devices. Given the 128 channels that modern CPUs support, and if reasonably possible, my suggestion would be to support up to 30 PCI devices. The logic breakdown is: 24 PCI Storage + 4 PCI GPU + 2 PCI Network.

Thanks again for the follow up to my questions.

Babak