PCIe passthrough - no interrupts - AIM APE1553

[fpdragon]

I am trying to create a backwards compatible test system in which a modern HW system with Proxmox and a VM with an old WinXP is running.

In addition, several different PCIe and USB interface cards are used.

All interface cards work except:
Company: AIM
Card Name: APE1553

This PCIe card was also configured via PCI passthrough.

All Functions: checked
ROM-Bar: checked
PCI-Express: unchecked (Q35 only)

The card was correctly recognized in the WinXP VM in the device manager and can also be addressed in principle. The only problem is that the card unfortunately does not trigger any interrupts.

I am already in contact with the manufacturer but I am still trying to find out whether a misconfiguration on my part could be the cause.

Here are some details:
PVE: 7.4-3
CPU: Intel Xeon W-2295
IOMMU: enabled
IOMMU interrupts: enabled

Thoughts on this:
1.) Can the pc-i440fx configuration have something to do with the interrupt handling?
2.) The card is physically PCIe but can only be passed through as PCI. Could this be a cause?
3.) Do I perhaps have to blacklist the card driver like GPUs during hardware boot?
4.) The PVE UI has not listed the PCIe card. I needed to list all IDs with lspci command to find the representation.

Hope someone can help and inform.

 

[leesteken]

PVE: 7.4-3

You might want to upgrade as it's soon out of support.

Thoughts on this:
1.) Can the pc-i440fx configuration have something to do with the interrupt handling?

Unlikely. Maybe it's an IOMMU compatibility issue? Do you know of other people (on the internet) who got the same device working with passthrough?

2.) The card is physically PCIe but can only be passed through as PCI. Could this be a cause?

Maybe the Windows drivers expect, assume or require PCIe instead of PCI? You could try a Windows VM with q35 and PCIe.

3.) Do I perhaps have to blacklist the card driver like GPUs during hardware boot?

This (or early binding to vfio-pci) is a good idea. Some devices don't reset properly (and can only be used once per host reboot) and in that case you want to make sure Proxmox does not touch the device or loads a driver for it.

4.) The PVE UI has not listed the PCIe card. I needed to list all IDs with lspci command to find the representation.

There are some filters to not show PCI(e) bridges and such. Maybe it unintentionally filters your device. It does not make a difference for passthrough.

Looks like passthrough is setup correctly as Windows detects the device, so that's good. Make sure to reboot the Proxmox host in between experiments, since the device might need that.
Try with and without ROM, try PCIe and PCI (but that requires a Windows reinstall probably, or a separate VM).
Try to find out if other people had success (regardless of what hypervisor they used) and try to implement their settings and work-arounds.
Test with another device that is known to work well with passthrough to rule out wrong motherboard/BIOS settings.
Test with Windows installed on bare metal to see if it works to rule out wrong motherboard/BIOS settings.

 

[fpdragon]

Is it possible to install WinXP VM on Q35?

I will try the other stuff.

Thanks a lot!

 

[leesteken]

Is it possible to install WinXP VM on Q35?

I don't know. I just noticed the PCIe was designed after Windows XP was released (and q35 was even later), so probably not. But since you appear to have Windows XP drivers for a PCIe add-in card, maybe?

 

[fpdragon]

Ok, as far as I understood, the issue is that MSI (Message Signaled Interrupts) are lost.

Is there a way to map the MSI from the physical PCIe card and pass it through to the guest as a simulated PCI device with line driven interrupts?

Thanks

 

[leesteken]

Ok, as far as I understood, the issue is that MSI (Message Signaled Interrupts) are lost.

Is there a way to map the MSI from the physical PCIe card and pass it through to the guest as a simulated PCI device with line driven interrupts?

I don't know but maybe MSI can be disabled for the device (which I think is a common thing) and maybe then interrupts do get throuigh?

EDIT: You could try the kernel parameter pci=nomsi to disable MSI system wide to test.
EDIT2: Maybe have a look at Disabling MSIs below a bridge here: https://github.com/torvalds/linux/blob/master/Documentation/PCI/msi-howto.rst

 

[fpdragon]

I don't know but maybe MSI can be disabled for the device (which I think is a common thing) and maybe then interrupts do get throuigh?

EDIT: You could try the kernel parameter pci=nomsi to disable MSI system wide to test.
EDIT2: Maybe have a look at Disabling MSIs below a bridge here: https://github.com/torvalds/linux/blob/master/Documentation/PCI/msi-howto.rst

Thanks, but I don't think that this is the solution.

I can use the PCIe card and initialize, configure and start it but no interrupts coming through. I programmed a software the uses the card's api and all api calls are working except of interrupts.
By disabling the MSIs at top, I guess that won't change my situation and I still won't have interrupts. Or does PCIe devices still have a line driver interrupt fallback as an alternative to MSI?

 

[leesteken]

Or does PCIe devices still have a line driver interrupt fallback as an alternative to MSI?

I don't know but with some devices/drivers MSI is optional, therefore I though I suggest trying without MSI if they are the (only) problem.

 

[fpdragon]

I don't know but with some devices/drivers MSI is optional, therefore I though I suggest trying without MSI if they are the (only) problem.

Thanks a lot,

Yes they are the only problem but essential for the application.

I will try so.