[SOLVED] What do i do when I have more than 16 PCIE devices.....?

[scyto]

What do i do when i have more that 16 devices to pass through to 1 VM?
i think QEMU supports 32, but it seems the UI only supports 16....

I have a mix of 4x SATA controllers, 2x coral TPUs, 7x NVMe, 4 x Optanes, and 1 x GPU.

I just realized while typin this, that the way the SATA works on this mobo 2 of the SATA controllers are illusory (i.e. in the chipset with no physical ports they can use). so i can actually get everything passed through

but i am i am at my limit of pass through PCIE devices.....

(well y'all told me to virtualize my truenas!)

 

[Neobin]

What do i do when i have more that 16 devices to pass through to 1 VM?

Just go bare-metal.

(well y'all told me to virtualize my truenas!)

I did not.

Post in thread '16 PCI Device limitation'

Sep 6, 2024

My question is, can I use an HBA that supports multiple NVME U.2 drives so that by adding a single HBA PCI Device, I can recognize an additional 6-8 drives using only one PCI Device attachment in the VM Hardware Configuration?

i don't think that will make a difference, since the drives will still show up as pci devices and the hba is "just" a multiplexer... (though i could be wrong of course)

Better yet, is there any way to bypass the 16 PCI device limit?

no, currently there isn't without modifying the code

you can open a feature request here...

• dcsapak

• 

(I could not find a Bugzilla ticket...)

 

[leesteken]

Why the passthrough of NVMe devices; what's wrong with virtual disks on top of those NVMe? It's slower but much more flexible (and easier to bacjkup) IMHO.

 

[scyto]

@Neobin weird i tried to use search to find that and failed, i ovioulsy, have no search-fu at the moment, thanks will mark you post as the answer (yes i know we can't do that here)

@leesteken because someone 'some say' thats a bad idea why doing ZFS - it tends to like full disk access, so i trused the people that seem to know.

 

[proxuser77]

Use the NAS as a NAS, and pass the rest to a Linux VM.

Or go baremetal with TrueNAS and buy a mini PC for the other two VMs you're running

Or go bare metal with TrueNAS and run the other two VMs on TrueNAS.

Or go baremetal with Linux and install Incus ;-)