PCI Passthrough - Device not listed

[fpdragon]

Hi,

I am currently building a Proxmox system for virtualizing old testequipments. For that purpose I am using multiple interface cards (PCI and PCIe) that shall be passed through to an WinXP VM.
I managed to setup the VM and I also passed through some PCIe cards successfully.

However, doing the same with an PCI card seems to be more problematic since the card is currently not listed in the Proxmox HW GUI for selecting PCI devices.

The host system is the following:
Dell Precision 5820 Tower Workstation
36 x Intel(R) Xeon(R) W-2295 CPU @ 3.00GHz (1 Socket)
The device has multiple full lane PCIe slots and one PCI slot.

I made sure that the PCI socket is activated in the BIOS. There is an option that can be checked.

I used the "lspci" to list all PCI devices from the shell with the card included and without.
Diffing the outputs it came out that this device appears if I plugin the card only:

0000:03:02.0 Memory controller: PLX Technology, Inc. PCI6540/6466 PCI-PCI bridge (non-transparent mode, primary side) (rev cb)

However, as said above, I can't select this device with the "Add PCI device" dialog.
It seems that the listed choices are just a subset compared to the output of the "lspci" command.

The PCI card I want to passthrough is an:
AIM PCI-1553
that was working fine in the old test equipment.

Any ideas? What could be the problem?
Is the "0000:03:02.0 Memory controller" the representation of that card or could it be a missleading effect that just this entry appears when I plugin the device?
Why is this device not shown in the GUI?
Can I force the passthrough of this device?

Thanks!

Edit 1:

I tried to manually edit the configuration of:
/etc/pve/qemu-server/1019.conf
(that is the VM)
Here is the line I added:

hostpci2: 0000:03:02

When I try to boot the VM, the whole proxmox server goes offline.
No solution.

 

[leesteken]

I used the "lspci" to list all PCI devices from the shell with the card included and without.
Diffing the outputs it came out that this device appears if I plugin the card only:

0000:03:02.0 Memory controller: PLX Technology, Inc. PCI6540/6466 PCI-PCI bridge (non-transparent mode, primary side) (rev cb)

However, as said above, I can't select this device with the "Add PCI device" dialog.
It seems that the listed choices are just a subset compared to the output of the "lspci" command.

Indeed "bridges" are filtered out because there is no need to pass them through. VFIO takes case of that when you passthrough devices connected to those bridges.

The PCI card I want to passthrough is an:
AIM PCI-1553
that was working fine in the old test equipment.

What is the difference with your current system and the old test equipment? Did it use an earlier Proxmox version and passthrough?

Any ideas? What could be the problem?
Is the "0000:03:02.0 Memory controller" the representation of that card or could it be a missleading effect that just this entry appears when I plugin the device?
Why is this device not shown in the GUI?

Because of the (PCI) "bridge" in the description.

Can I force the passthrough of this device?

Just add it manually to the VM configuration file. Select another device in the GUI (and don't start the VM) and edit the PCI-ID VM configuration file to 03:02.

Thanks!

Edit 1:

I tried to manually edit the configuration of:
/etc/pve/qemu-server/1019.conf
(that is the VM)
Here is the line I added:

hostpci2: 0000:03:02

When I try to boot the VM, the whole proxmox server goes offline.
No solution.

Check your IOMMU groups. Often the device is in a group with essential devices for Proxmox like disk, network, etc. (lots of threads about this here). It's also possible that resetting the device cause it to misbehave and take down the whole system. My experience is that PCI cards (from before PCIe) are impossible to get working correctly with passthrough.

 

[fpdragon]

Indeed "bridges" are filtered out because there is no need to pass them through. VFIO takes case of that when you passthrough devices connected to those bridges.

What is the difference with your current system and the old test equipment? Did it use an earlier Proxmox version and passthrough?

Because of the (PCI) "bridge" in the description.

Just add it manually to the VM configuration file. Select another device in the GUI (and don't start the VM) and edit the PCI-ID VM configuration file to 03:02.

Check your IOMMU groups. Often the device is in a group with essential devices for Proxmox like disk, network, etc. (lots of threads about this here). It's also possible that resetting the device cause it to misbehave and take down the whole system. My experience is that PCI cards (from before PCIe) are impossible to get working correctly with passthrough.

Thanks for you answer.

That maybe was unclear in my first post:
I am trying to replace old physical testequipment PCs with a modern HW and Proxmox hypervisor with WinXP VM.
So modern HW and still old and compatible SW.

Ok, as far as I understand, I have to read into IOMMU groups...

But in general, what would be the correct behavior?
Should the card get it's own lspci entry below the bridge or is the bridge enty already the corresponding entry?

Thx.

 

[leesteken]

I am trying to replace old physical testequipment PCs with a modern HW and Proxmox hypervisor with WinXP VM.
So modern HW and still old and compatible SW.

Unless others already succeeded with passthrough of the device and/or found work-arounds, I fear the chances are slim.I'm the resident pessimist, so try anyway.

Ok, as far as I understand, I have to read into IOMMU groups...

Devices in the same IOMMU group cannot be securely/safely shared between VMs and/or the Proxmox host. The IOMMU groups are determined by the physical motherboard (and BIOS). Moving the add-in card to another slot can make it change groups but I gather that you need a different motherboard for that.

You could try the pcie_acs_override for testing but there just because the groups are then ignored, nothing physically changes and it does not fix passthrough problems.
If we can get the system to not crash and see the device inside the VM, then it might be worth to make sure Proxmox does not touch it before the VM starts and we hope the driver inside the VM can get it to function. No guarantees that you can stop and restart the VM without a Proxmox host reboot, though.

But in general, what would be the correct behavior?
Should the card get it's own lspci entry below the bridge or is the bridge enty already the corresponding entry?

It's quite possible that the description is wrong because a (newer) device uses the same device ID as your add-in card. And that other one happens to be a PCI bridge. I would assume the description is wrong and ignore it.

I fear your setup is not going to work and I doubt other setups will work with passthrough of your PCI add-in card, but I would love to be wrong and learn something new. Alternatives I can think of are: Can you try a newer PCIe alternative for your PCI add-in card? Or are there USB variants that might work (with USB passthrough or PCIe passthrough of a USB controller)?

 

[fpdragon]

Unless others already succeeded with passthrough of the device and/or found work-arounds, I fear the chances are slim.I'm the resident pessimist, so try anyway.

Devices in the same IOMMU group cannot be securely/safely shared between VMs and/or the Proxmox host. The IOMMU groups are determined by the physical motherboard (and BIOS). Moving the add-in card to another slot can make it change groups but I gather that you need a different motherboard for that.

You could try the pcie_acs_override for testing but there just because the groups are then ignored, nothing physically changes and it does not fix passthrough problems.
If we can get the system to not crash and see the device inside the VM, then it might be worth to make sure Proxmox does not touch it before the VM starts and we hope the driver inside the VM can get it to function. No guarantees that you can stop and restart the VM without a Proxmox host reboot, though.

It's quite possible that the description is wrong because a (newer) device uses the same device ID as your add-in card. And that other one happens to be a PCI bridge. I would assume the description is wrong and ignore it.

I fear your setup is not going to work and I doubt other setups will work with passthrough of your PCI add-in card, but I would love to be wrong and learn something new. Alternatives I can think of are: Can you try a newer PCIe alternative for your PCI add-in card? Or are there USB variants that might work (with USB passthrough or PCIe passthrough of a USB controller)?

In parallel I will try to find a corresponding PCIe card.
Chances seem to be higher that PCIe will work since I already have two PCIe cards that work.
We will see...

However, I don't expected that passing through that one PCI port is such a problem allthough I have done that with other PCIe slots/cards.
I guess you never stop learning.

 

[leesteken]

In parallel I will try to find a corresponding PCIe card.
Chances seem to be higher that PCIe will work since I already have two PCIe cards that work.
We will see...

However, I don't expected that passing through that one PCI port is such a problem allthough I have done that with other PCIe slots/cards.
I guess you never stop learning.

Check the IOMMU groups and maybe make Proxmox ignore them. You can still try to get this to work, even if it's just for learning and continued use of the hardware.
What is the output of for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done? Add pcie_acs_override=downstream,multifunction to the kernel parameters to see what happens to the problematic group.

 

[fpdragon]

Sorry to come back late.

I think IOMMU groups are no problem.

The card is on it's own group with:
IOMMU group 23 0000:02:00.0 Memory controller [0580]: AIM GmBH Device [1447:1310]

I still have problems with that card.
The card is detected in WinXP Device Manager and I can communicate with it.
The problem is that no Interrupts are thrown.

Any idea what's with the interrupts?

Maybe the missing Q35 PCI-Express support?

Thanks

 

[leesteken]

Looks like you started a new thread (where I replied, not remembering this conversation) here: https://forum.proxmox.com/threads/pcie-passthrough-no-interrupts-aim-ape1553.150841/

 

[fpdragon]

Looks like you started a new thread (where I replied, not remembering this conversation) here: https://forum.proxmox.com/threads/pcie-passthrough-no-interrupts-aim-ape1553.150841/

yes, sorry for the double post.
I just noticed.