Sequence or advanced setting with PCI-E device passthrough

[Sandbo]

Sorry if this is not related to Proxmox, but I am out of idea with a recent passthrough related issue.

I am trying to use some instruments through a PXI-E chassis. It is essentially a instrument chassis like eGPU box, which extends the PCI-E connections to outside of a workstation, by connecting installing a PCI-E interface card into the workstation, and an extension cable from the card to the chassis where modules of instruments are installed, all communicating through PCI-E interfaces.

On Proxmox 7.4, I was able to do the passthrough to a Windows VM (by selecting and passing each module), seeing all the modules and actually talking to the modules through the official drivers. However, there is an issue with the software recognizing the grouping of the modules. Ideally, one device which corresponds to the chassis has a chassis number, say Chassis 1. The modules inside this chassis will be assigned "Chassis 1", from which we can control all modules as a whole.

However, in my case, the modules were not recognized as they were inside Chassis 1, as if they were all separated PCI-E devices. I am not sure if this has anything to do with the VM PCI-E passthrough that might have messed up the sequence of the modules being recognized. I wonder if there was anything like sequence of booting up that I can config in the PCI-E passthrough?

Another question would be, it seems in Proxmox I am seeing the individual instruments (like individual PCI-E cards), instead of the single PCI-E interface card I have inside my workstation. Is it possible to pass the PCI-E interface card instead?

Any advice is appreciated.

 

[leesteken]

Another question would be, it seems in Proxmox I am seeing the individual instruments (like individual PCI-E cards), instead of the single PCI-E interface card I have inside my workstation. Is it possible to pass the PCI-E interface card instead?

Can you show the IOMMU group(s) of those instruments? This command gives a nice overview of all groups: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done,
Maybe they are separate functions of the same device and can be passed together using the All Functions option. Otherwise, they are really separate devices on the same PCIe bus (using a PCIe multiplexer/bridge) and you just need to pass them all explicitly.

 

[Sandbo]

Can you show the IOMMU group(s) of those instruments? This command gives a nice overview of all groups: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done,
Maybe they are separate functions of the same device and can be passed together using the All Functions option. Otherwise, they are really separate devices on the same PCIe bus (using a PCIe multiplexer/bridge) and you just need to pass them all explicitly.

From the list of passable items in Proxmox, there are 3 modules (bb, bc and bd( plus 2 more (bf and c9) probably for the PXI chassis itself which I think were the relevant entries, indented in the output below. The current issue is while I can see and drivers can connect to the module in the Window VM, the 3 modules (bb, bc and bd) were not recognized by the drivers to be belong to the chassis modules (bf and c9), as if they were in a different location/chassis.

I am not sure where to go from here to understand the issue.

Spoiler: IOMMU

IOMMU group 0 b2:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 07)
IOMMU group 100 64:0d.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 07)
IOMMU group 101 64:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 07)
IOMMU group 102 b2:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 07)
IOMMU group 103 b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 07)
IOMMU group 104 b2:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 07)
IOMMU group 10 b8:05.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 11 b8:08.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 12 b8:09.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 13 b8:0d.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 14 b9:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 15 ba:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 16 ba:08.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 17 ba:0c.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 18 ba:0d.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
    IOMMU group 19 bb:00.0 DPIO module [1100]: Xilinx Corporation Default PCIe endpoint ID [10ee:0007]
IOMMU group 1 b2:01.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port B [8086:2031] (rev 07)
    IOMMU group 20 bc:00.0 DPIO module [1100]: Xilinx Corporation Default PCIe endpoint ID [10ee:0007]
    IOMMU group 21 bd:00.0 DPIO module [1100]: Xilinx Corporation Default PCIe endpoint ID [10ee:0007]
    IOMMU group 22 bf:00.0 SMBus [0c05]: Adlink Technology Device [144a:9022] (rev 01)
IOMMU group 23 c1:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 24 c2:09.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 25 c3:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 26 c4:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 27 c4:04.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 28 c4:05.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 29 c4:06.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 2 b2:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 07)
IOMMU group 30 c4:08.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 31 c4:09.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 32 c4:0c.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 33 c4:0d.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 34 c7:00.0 PCI bridge [0604]: Pericom Semiconductor PI7C9X110 PCI Express to PCI bridge [12d8:e110] (rev 04)
    IOMMU group 35 c9:00.0 Unassigned class [ff00]: Agilent Technologies Device [15bc:123d] (rev 22)
IOMMU group 36 64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 07)
IOMMU group 37 65:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2704] (rev a1)
IOMMU group 37 65:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bb] (rev a1)
IOMMU group 38 00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 07)
IOMMU group 39 00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
IOMMU group 3 b3:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC Device [1bb1:5016] (rev 01)
IOMMU group 47 00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 07)
IOMMU group 48 00:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS [8086:2025] (rev 07)
IOMMU group 49 00:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOAPIC [8086:2026] (rev 07)
IOMMU group 4 b4:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
IOMMU group 50 00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 07)
IOMMU group 51 00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 07)
IOMMU group 52 00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 07)
IOMMU group 53 00:14.0 USB controller [0c03]: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [8086:a2af]
IOMMU group 53 00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
IOMMU group 54 00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
IOMMU group 54 00:16.3 Serial controller [0700]: Intel Corporation Device [8086:a2bd]
IOMMU group 55 00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
IOMMU group 56 00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
IOMMU group 57 00:1c.3 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #4 [8086:a293] (rev f0)
IOMMU group 58 00:1f.0 ISA bridge [0601]: Intel Corporation C422 Chipset LPC/eSPI Controller [8086:a2d3]
IOMMU group 58 00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
IOMMU group 58 00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
IOMMU group 58 00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
IOMMU group 59 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-LM [8086:15b7]
IOMMU group 5 b5:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab)
IOMMU group 60 01:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU group 61 02:00.0 Ethernet controller [0200]: Aquantia Corp. AQC108 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:d108] (rev 02)
IOMMU group 62 16:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 07)
IOMMU group 63 16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 07)
IOMMU group 64 16:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 07)
IOMMU group 6 b6:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab)
IOMMU group 73 64:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 07)
IOMMU group 74 64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 07)
IOMMU group 75 64:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 07)
IOMMU group 7 b7:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 8 b8:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)
IOMMU group 9 b8:04.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8764] (rev ab)

 

[leesteken]

Are you using pcie_acs_override? That would make the group information invalid but it still shows the PCIe layout of the system with the chassis connected.

PCIe passthrough does not allow you to pass through a whole sub-tree of the PCIe bus, like the chassis and its devices. Only separate devices, which will be connected to the virtual PCI(e) busses of the VM. People using Thunderbolt have the same issue (see this forum) where they can passthrough the connected devices but not the controller and have hot-plug work automatically.

Maybe you can manually (see the QEMU documentation) recreate a vrtual PCIe layout for the VM that matches the chassis and then put the real instrument devices and the chassis devices into the same spots and make it look like the real chassis? I have no idea what needs to be done to trick the Windows drivers into working.

 

[Sandbo]

Are you using pcie_acs_override? That would make the group information invalid but it still shows the PCIe layout of the system with the chassis connected.

PCIe passthrough does not allow you to pass through a whole sub-tree of the PCIe bus, like the chassis and its devices. Only separate devices, which will be connected to the virtual PCI(e) busses of the VM. People using Thunderbolt have the same issue (see this forum) where they can passthrough the connected devices but not the controller and have hot-plug work automatically.

Maybe you can manually (see the QEMU documentation) recreate a vrtual PCIe layout for the VM that matches the chassis and then put the real instrument devices and the chassis devices into the same spots and make it look like the real chassis? I have no idea what needs to be done to trick the Windows drivers into working.

I don't remember enabling ACS override, as the board should by default support VT-d and passthrough so I didn't try but I should check. (The board is Supermicro X11SRA).

It's a good idea to check those issues people might have experience regarding Thunderbolt passthrough, the situation maybe similar for some applications. And I think the direction you pointed out is right, there maybe something that I can do to trick the drivers to believe these devices are under one another.