Missing PCIe Items in add PCIe device

[juju01]

Hello

I have a Dell R740 server which has a PCIe backplane extender for nvme drives. I am looking to passthrough the extender to a truenas install . When I go to add PCIe device for the truenas vm, I don't see the extender listed though I can see it when I do an lspci on the proxmox node:

lspci -nnk | grep -e PLX
5e:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:06.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
af:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:06.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)

The same server also has a Dell HBA card for SAS drives. I have successfully setup the passthrough to truenas - I see the SAS drives listed in truenas. Just can't find the PCIe extender in the PCIe add device list to passthrough. Is there something to do to make them show up?

 

[leesteken]

There is no need to passthrough PCI bridges, nor will that work AFAIK. You can only passthrough devices (or actually whole IOMMU groups) and not the physical PCIe layout/structure, which will be show as a (different) virtual PCIe layout inside the VM. I think you need to passthrough all devices separately. Enable All Functions for each device and you only need to passthrouugh one function of each device (01:00.0 without 01:00.1, 01:00.2, etc.) to passthrough all functions of the device.

 

[juju01]

I think you need to passthrough all devices separately. Enable All Functions for each device and you only need to passthrouugh one function of each device (01:00.0 without 01:00.1, 01:00.2, etc.) to passthrough all functions of the device.

@leesteken - you mean pass each individual hard drive separately? What do you mean by "device"? I can't see any of the PLX devices in the dropdown list

 

[leesteken]

@leesteken - you mean pass each individual hard drive separately? What do you mean by "device"? I can't see any of the PLX devices in the dropdown list

I mean PCI(e) devices (but not bridges), like the "Dell HBA card for SAS drives". You wrote that you see the drives, so I assume everything is okay? I just tried to explain that you won't see the PCI(e) bridges and therefore no PLX devices inside a VM.

 

[juju01]

Yes, everything is ok ( for the sas drives). Not the nvme drives attached to the extender devices.

 

[leesteken]

Yes, everything is ok ( for the sas drives). Not the nvme drives attached to the extender devices.

Can you passthrough all the NVMe drives? Those are PCIe devices and I think you need to passthrough each of them separately. Sorry, you already said this in your first message and I missed it.

 

[juju01]

Done! I have just 8 nvme drives so didn't take too much time. I assumed I could add them via the extender. Using this method, I see the disks listed in both truenas and proxmox. Isn't that a problem? I wonder if Its better to add another controller in an available PCIe slot and connect all nvme drives to.