Missing PCIe Items in add PCIe device

juju01

Member
May 16, 2020
87
3
13
Hello

I have a Dell R740 server which has a PCIe backplane extender for nvme drives. I am looking to passthrough the extender to a truenas install . When I go to add PCIe device for the truenas vm, I don't see the extender listed though I can see it when I do an lspci on the proxmox node:

Code:
lspci -nnk | grep -e PLX
5e:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:06.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
5f:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
af:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:06.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)
b0:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:9733] (rev b0)

The same server also has a Dell HBA card for SAS drives. I have successfully setup the passthrough to truenas - I see the SAS drives listed in truenas. Just can't find the PCIe extender in the PCIe add device list to passthrough. Is there something to do to make them show up?
 
There is no need to passthrough PCI bridges, nor will that work AFAIK. You can only passthrough devices (or actually whole IOMMU groups) and not the physical PCIe layout/structure, which will be show as a (different) virtual PCIe layout inside the VM. I think you need to passthrough all devices separately. Enable All Functions for each device and you only need to passthrouugh one function of each device (01:00.0 without 01:00.1, 01:00.2, etc.) to passthrough all functions of the device.
 
I think you need to passthrough all devices separately. Enable All Functions for each device and you only need to passthrouugh one function of each device (01:00.0 without 01:00.1, 01:00.2, etc.) to passthrough all functions of the device.

@leesteken - you mean pass each individual hard drive separately? What do you mean by "device"? I can't see any of the PLX devices in the dropdown list
 
Last edited:
@leesteken - you mean pass each individual hard drive separately? What do you mean by "device"? I can't see any of the PLX devices in the dropdown list
I mean PCI(e) devices (but not bridges), like the "Dell HBA card for SAS drives". You wrote that you see the drives, so I assume everything is okay? I just tried to explain that you won't see the PCI(e) bridges and therefore no PLX devices inside a VM.
 
Yes, everything is ok ( for the sas drives). Not the nvme drives attached to the extender devices.
 
Yes, everything is ok ( for the sas drives). Not the nvme drives attached to the extender devices.
Can you passthrough all the NVMe drives? Those are PCIe devices and I think you need to passthrough each of them separately. Sorry, you already said this in your first message and I missed it.
 
Last edited:
Done! I have just 8 nvme drives so didn't take too much time. I assumed I could add them via the extender. Using this method, I see the disks listed in both truenas and proxmox. Isn't that a problem? I wonder if Its better to add another controller in an available PCIe slot and connect all nvme drives to.
 
Last edited:
  • Like
Reactions: leesteken

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!