[SOLVED] Issues passing through PCIe card (non-GPU)

[bqmt]

Hi,
I'm trying to passthrough a PCIe card, specifically a digitiser from Spectrum Instrumentation. As far as I can tell I've done all the required configuration for PCIe passthrough to work, I have IOMMU groups showing but when I pass through the device I can never see anything on the VM side. I've tried Linux and Windows VMs, installing the drivers on both even but no dice.

My mobo is an Asrock Z590 Taichi, VT-d enabled in BIOS, i7-10700.

My cmdline options in grub

root@VH-002:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

Kernel modules loaded

root@VH-002:~# lsmod | grep vfio
vfio_pci               16384  0
vfio_pci_core          86016  1 vfio_pci
irqbypass              12288  2 vfio_pci_core,kvm
vfio_iommu_type1       49152  0
vfio                   57344  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                77824  1 vfio

Dmesg output, the feature inconsistent errors go away if I use intel_iommu=igfx_off but it doesn't seem to effect whether I can pass this card through.

root@VH-002:~# dmesg | grep -e DMAR -e IOMMU
[    0.009214] ACPI: DMAR 0x000000005D0C2000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.009250] ACPI: Reserving DMAR table memory at [mem 0x5d0c2000-0x5d0c2087]
[    0.050123] DMAR: IOMMU enabled
[    0.135170] DMAR: Host address width 39
[    0.135171] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.135177] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.135180] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.135183] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.135185] DMAR: RMRR base: 0x0000005f000000 end: 0x0000006f7fffff
[    0.135187] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.135189] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.135190] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.136777] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.492141] DMAR: No ATSR found
[    0.492142] DMAR: No SATC found
[    0.492143] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.492144] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.492145] DMAR: IOMMU feature nwfs inconsistent
[    0.492145] DMAR: IOMMU feature pasid inconsistent
[    0.492146] DMAR: IOMMU feature eafs inconsistent
[    0.492147] DMAR: IOMMU feature prs inconsistent
[    0.492147] DMAR: IOMMU feature nest inconsistent
[    0.492148] DMAR: IOMMU feature mts inconsistent
[    0.492148] DMAR: IOMMU feature sc_support inconsistent
[    0.492149] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.492150] DMAR: dmar0: Using Queued invalidation
[    0.492153] DMAR: dmar1: Using Queued invalidation
[    0.493099] DMAR: Intel(R) Virtualization Technology for Directed I/O

Output from pvesh, the device I'm trying to passthrough is group 13.

root@VH-002:~# pvesh get /nodes/VH-002/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬────────────────────────────────────────────────────┬──────┬─────────────
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                        │ mdev │ subsystem_de
╞══════════╪════════╪══════════════╪════════════╪════════╪════════════════════════════════════════════════════╪══════╪═════════════
│ 0x010601 │ 0x43d2 │ 0000:00:17.0 │          4 │ 0x8086 │                                                    │      │ 0x43d2     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x010601 │ 0x0612 │ 0000:71:00.0 │         25 │ 0x1b21 │ ASM1062 Serial ATA Controller                      │      │ 0x0612     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x010802 │ 0x5405 │ 0000:02:00.0 │         14 │ 0x1344 │                                                    │      │ 0x0100     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x020000 │ 0x15fa │ 0000:00:1f.6 │         12 │ 0x8086 │ Ethernet Connection (14) I219-V                    │      │ 0x15fa     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x020000 │ 0x3000 │ 0000:03:00.0 │         15 │ 0x10ec │ Killer E3000 2.5GbE Controller                     │      │ 0x3000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x028000 │ 0x2725 │ 0000:04:00.0 │         16 │ 0x8086 │ Wi-Fi 6 AX210/AX211/AX411 160MHz                   │      │ 0x1674     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x030000 │ 0x9bc5 │ 0000:00:02.0 │          0 │ 0x8086 │ CometLake-S GT2 [UHD Graphics 630]                 │      │ 0x9bc5     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x040300 │ 0xf0c8 │ 0000:00:1f.3 │         12 │ 0x8086 │                                                    │      │ 0x222b     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x050000 │ 0x43ef │ 0000:00:14.2 │          2 │ 0x8086 │ Tiger Lake-H Shared SRAM                           │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060000 │ 0x9b43 │ 0000:00:00.0 │          1 │ 0x8086 │ 10th Gen Core Processor Host Bridge/DRAM Registers │      │ 0x9b43     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060100 │ 0x4385 │ 0000:00:1f.0 │         12 │ 0x8086 │                                                    │      │ 0x4385     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x43c0 │ 0000:00:1b.0 │          5 │ 0x8086 │                                                    │      │ 0x43c0     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x43c4 │ 0000:00:1b.4 │          6 │ 0x8086 │                                                    │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x43b8 │ 0000:00:1c.0 │          7 │ 0x8086 │                                                    │      │ 0x43b8     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x43b9 │ 0000:00:1c.1 │          8 │ 0x8086 │                                                    │      │ 0x43b9     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x43ba │ 0000:00:1c.2 │          9 │ 0x8086 │                                                    │      │ 0x43ba     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x43bc │ 0000:00:1c.4 │         10 │ 0x8086 │ Tiger Lake-H PCI Express Root Port #5              │      │ 0x43bc     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x43b5 │ 0000:00:1d.0 │         11 │ 0x8086 │                                                    │      │ 0x43b5     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x1136 │ 0000:06:00.0 │         18 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x1136 │ 0000:07:00.0 │         19 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x1136 │ 0000:07:01.0 │         20 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x1136 │ 0000:07:02.0 │         21 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x060400 │ 0x1136 │ 0000:07:03.0 │         22 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x078000 │ 0x43e0 │ 0000:00:16.0 │          3 │ 0x8086 │ Tiger Lake-H Management Engine Interface           │      │ 0x43e0     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x0c0330 │ 0x43ed │ 0000:00:14.0 │          2 │ 0x8086 │ Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller  │      │ 0x43ed     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x0c0330 │ 0x1142 │ 0000:05:00.0 │         17 │ 0x1b21 │ ASM1042A USB 3.0 Host Controller                   │      │ 0x1142     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x0c0330 │ 0x1138 │ 0000:0a:00.0 │         24 │ 0x8086 │ Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x0c0340 │ 0x1137 │ 0000:08:00.0 │         23 │ 0x8086 │ Thunderbolt 4 NHI [Maple Ridge 4C 2020]            │      │ 0x0000     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x0c0500 │ 0x43a3 │ 0000:00:1f.4 │         12 │ 0x8086 │ Tiger Lake-H SMBus Controller                      │      │ 0x43a3     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x0c8000 │ 0x43a4 │ 0000:00:1f.5 │         12 │ 0x8086 │ Tiger Lake-H SPI Controller                        │      │ 0x43a4     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼─────────────
│ 0x118000 │ 0x2100 │ 0000:01:00.0 │         13 │ 0x18f1 │                                                    │      │ 0x2100     
└──────────┴────────┴──────────────┴────────────┴────────┴────────────────────────────────────────────────────┴──────┴─────────────

Output of lspci (device in question is the "01:00.0 Signal processing controller: Spectrum GmbH Device 2100")

root@VH-002:~# lspci
00:00.0 Host bridge: Intel Corporation 10th Gen Core Processor Host Bridge/DRAM Registers (rev 05)
00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 05)
00:14.0 USB controller: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller (rev 11)
00:14.2 RAM memory: Intel Corporation Tiger Lake-H Shared SRAM (rev 11)
00:16.0 Communication controller: Intel Corporation Tiger Lake-H Management Engine Interface (rev 11)
00:17.0 SATA controller: Intel Corporation Device 43d2 (rev 11)
00:1b.0 PCI bridge: Intel Corporation Tiger Lake-H PCIe Root Port (rev 11)
00:1b.4 PCI bridge: Intel Corporation Device 43c4 (rev 11)
00:1c.0 PCI bridge: Intel Corporation Tiger Lake-H PCIe Root Port (rev 11)
00:1c.1 PCI bridge: Intel Corporation Device 43b9 (rev 11)
00:1c.2 PCI bridge: Intel Corporation Tiger Lake-H PCIe Root Port (rev 11)
00:1c.4 PCI bridge: Intel Corporation Tiger Lake-H PCI Express Root Port #5 (rev 11)
00:1d.0 PCI bridge: Intel Corporation Device 43b5 (rev 11)
00:1f.0 ISA bridge: Intel Corporation Z590 LPC/eSPI Controller (rev 11)
00:1f.3 Audio device: Intel Corporation Device f0c8 (rev 11)
00:1f.4 SMBus: Intel Corporation Tiger Lake-H SMBus Controller (rev 11)
00:1f.5 Serial bus controller: Intel Corporation Tiger Lake-H SPI Controller (rev 11)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (14) I219-V (rev 11)
01:00.0 Signal processing controller: Spectrum GmbH Device 2100
02:00.0 Non-Volatile memory controller: Micron Technology Inc 2300 NVMe SSD [Santana]
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Killer E3000 2.5GbE Controller (rev 06)
04:00.0 Network controller: Intel Corporation Wi-Fi 6 AX210/AX211/AX411 160MHz (rev 1a)
05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
06:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
07:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
07:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
07:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
07:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
08:00.0 USB controller: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020]
0a:00.0 USB controller: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020]
71:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

Running lspci in the VMs before and after passing through the card shows no difference. On the Windows side I can't see the device showing up either. My VMs are q35, OVMF. I'm passing through the raw device and I've tried the different options of pcie, ROM-bar, All functions but so far nothing has made a difference. Any pointers would be much appreciated.

Thank you

 

[bqmt]

I resolved this by setting the CPU type to host.