PCI Passthrough mystery ?!? "Feature X inconsistent" messages . . .

[rfox]

I have three machines running latest Proxmox 7.2-11 - I use PCI passthrough on one for OPNSense Firewall (passing two NICs)

Everything seems to work except I see something strange when I check the IOMMU with "dmesg | grep -e DMAR -e IOMMU" - many "feature inconsistent" messages . . .

On one machine I get:

root@ProxFox:~# dmesg | grep -e DMAR -e IOMMU
[ 0.007949] ACPI: DMAR 0x000000008C512010 000070 (v01 INTEL EDK2 00000002 01000013)
[ 0.007979] ACPI: Reserving DMAR table memory at [mem 0x8c512010-0x8c51207f]
[ 0.078713] DMAR: IOMMU enabled
[ 0.201983] DMAR: Host address width 39
[ 0.201985] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.201990] DMAR: dmar0: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.201992] DMAR: RMRR base: 0x0000008cea0000 end: 0x0000008d0e9fff
[ 0.201995] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0
[ 0.201996] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.201997] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.204973] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.481721] DMAR: No ATSR found
[ 0.481721] DMAR: No SATC found
[ 0.481722] DMAR: dmar0: Using Queued invalidation
[ 0.482056] DMAR: Intel(R) Virtualization Technology for Directed I/O

on the other I get:

root@OPNFox:~# dmesg | grep -e DMAR -e IOMMU
[ 0.020297] ACPI: DMAR 0x00000000799E1000 0000A8 (v01 INTEL GLK-SOC 00000003 BRXT 0100000D)
[ 0.020440] ACPI: Reserving DMAR table memory at [mem 0x799e1000-0x799e10a7]
[ 0.125409] DMAR: IOMMU enabled
[ 0.388668] DMAR: Host address width 39
[ 0.388673] DMAR: DRHD base: 0x000000fed64000 flags: 0x0
[ 0.388696] DMAR: dmar0: reg_base_addr fed64000 ver 1:0 cap 1c0000c40660462 ecap 9e2ff0505e
[ 0.388706] DMAR: DRHD base: 0x000000fed65000 flags: 0x1
[ 0.388724] DMAR: dmar1: reg_base_addr fed65000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.388733] DMAR: RMRR base: 0x00000079948000 end: 0x00000079967fff
[ 0.388740] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fffffff
[ 0.388748] DMAR-IR: IOAPIC id 1 under DRHD base 0xfed65000 IOMMU 1
[ 0.388754] DMAR-IR: HPET id 0 under DRHD base 0xfed65000
[ 0.388759] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.391549] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.880415] DMAR: No ATSR found
[ 0.880418] DMAR: No SATC found
[ 0.880423] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.880427] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.880431] DMAR: IOMMU feature nwfs inconsistent
[ 0.880434] DMAR: IOMMU feature eafs inconsistent
[ 0.880437] DMAR: IOMMU feature prs inconsistent
[ 0.880440] DMAR: IOMMU feature nest inconsistent
[ 0.880442] DMAR: IOMMU feature mts inconsistent
[ 0.880445] DMAR: IOMMU feature sc_support inconsistent
[ 0.880448] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.880452] DMAR: dmar0: Using Queued invalidation
[ 0.880461] DMAR: dmar1: Using Queued invalidation
[ 0.881903] DMAR: Intel(R) Virtualization Technology for Directed I/O

Question of the day: Why are there so many "feature XXX inconsistent" messages on the second? What do they mean and can I get ride of them??

Thanks in advance,
R.Fox

 

[jimlad]

I'm getting the same. I'm trying to get ACS to work. I'm actually trying to achive the same as you by passing a single NIC to a OPNSense machine for the WAN. The LAN NIC is virtualised. My issue is both NIC are group 5 along with the WLAN. So by passing a NIC in, all host networking stops.

[ 0.506256] DMAR: No ATSR found
[ 0.506258] DMAR: No SATC found
[ 0.506262] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.506264] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.506266] DMAR: IOMMU feature nwfs inconsistent
[ 0.506268] DMAR: IOMMU feature eafs inconsistent
[ 0.506270] DMAR: IOMMU feature prs inconsistent
[ 0.506271] DMAR: IOMMU feature nest inconsistent
[ 0.506273] DMAR: IOMMU feature mts inconsistent
[ 0.506274] DMAR: IOMMU feature sc_support inconsistent
[ 0.506276] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.506278] DMAR: dmar0: Using Queued invalidation
[ 0.506285] DMAR: dmar1: Using Queued invalidation
[ 0.506331] Trying to unpack rootfs image as initramfs...
[ 0.506433] pci 0000:00:00.0: Adding to iommu group 0
[ 0.506461] pci 0000:00:02.0: Adding to iommu group 1
[ 0.506493] pci 0000:00:0e.0: Adding to iommu group 2
[ 0.506521] pci 0000:00:0f.0: Adding to iommu group 3
[ 0.506542] pci 0000:00:12.0: Adding to iommu group 4
[ 0.506601] pci 0000:00:13.0: Adding to iommu group 5
[ 0.506631] pci 0000:00:13.1: Adding to iommu group 5
[ 0.506657] pci 0000:00:13.2: Adding to iommu group 5
[ 0.506684] pci 0000:00:15.0: Adding to iommu group 6
[ 0.506734] pci 0000:00:16.0: Adding to iommu group 7
[ 0.506754] pci 0000:00:16.1: Adding to iommu group 7
[ 0.506776] pci 0000:00:16.2: Adding to iommu group 7
[ 0.506797] pci 0000:00:16.3: Adding to iommu group 7
[ 0.506846] pci 0000:00:17.0: Adding to iommu group 8
[ 0.506866] pci 0000:00:17.1: Adding to iommu group 8
[ 0.506887] pci 0000:00:17.2: Adding to iommu group 8
[ 0.506907] pci 0000:00:17.3: Adding to iommu group 8
[ 0.506958] pci 0000:00:18.0: Adding to iommu group 9
[ 0.506980] pci 0000:00:18.1: Adding to iommu group 9
[ 0.507001] pci 0000:00:18.2: Adding to iommu group 9
[ 0.507023] pci 0000:00:18.3: Adding to iommu group 9
[ 0.507044] pci 0000:00:1b.0: Adding to iommu group 10
[ 0.507065] pci 0000:00:1c.0: Adding to iommu group 11
[ 0.507090] pci 0000:00:1e.0: Adding to iommu group 12
[ 0.507125] pci 0000:00:1f.0: Adding to iommu group 13
[ 0.507148] pci 0000:00:1f.1: Adding to iommu group 13
[ 0.507163] pci 0000:01:00.0: Adding to iommu group 5
[ 0.507177] pci 0000:02:00.0: Adding to iommu group 5
[ 0.507190] pci 0000:03:00.0: Adding to iommu group 5
[ 0.507261] DMAR: Intel(R) Virtualization Technology for Directed I/O

 

[juched]

Same here, curious if this is a problem, or just a message.

[ 0.371928] DMAR: No ATSR found
[ 0.371929] DMAR: No SATC found
[ 0.371930] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.371932] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.371932] DMAR: IOMMU feature nwfs inconsistent
[ 0.371933] DMAR: IOMMU feature pds inconsistent
[ 0.371934] DMAR: IOMMU feature eafs inconsistent
[ 0.371935] DMAR: IOMMU feature prs inconsistent
[ 0.371935] DMAR: IOMMU feature nest inconsistent
[ 0.371936] DMAR: IOMMU feature mts inconsistent
[ 0.371936] DMAR: IOMMU feature sc_support inconsistent
[ 0.371937] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.371938] DMAR: dmar0: Using Queued invalidation
[ 0.371941] DMAR: dmar1: Using Queued invalidation

 

[jimlad]

Are you getting individual IOMMI groups per device? or are you see what I do, devices are still grouped togehter?

 

[TiagoLardaCrianca]

I'm having exactly the same problem on a server board from aliexpress, already tried everything and i cant get to passtrough the NICs.. didnt try with nothing else - other pcie, or graphics card, since its a small factor board for a firewall (for installing pfsense).

Also, the list of available pci devices to passtrough, on the proxmox webadmin console, appears with the descriptions empty.



Although i can put a device, and i know wich i want to pass trough, the vm - in this case pfsense, doesnt detect any passed-trough device:





No one has a viable solution yet?

 

[leesteken]

According to this kernel mailinglist message hardware can support features inconsistently and those messages report those.

Modern platforms have more than one IOMMU. Each IOMMU has its own
feature set. Some of these features must be consistent among IOMMUs.
Otherwise, these differences can lead to improper behavior in the system.
On the other hand, for some features, each IOMMU can have different
capacity values. So, different actions are required to deal with the
inconsistencies depending on the IOMMU features.

Currently, some inconsistencies are ignored by the IOMMU driver.
This patchset checks IOMMU capabilities and extended capabilities
centralizedly during boot and take different actions according to
the impacts caused by the mismatches.

I don't expect that this can be fixed by software but maybe a BIOS, firmware or microcode update can help?

 

[TiagoLardaCrianca]

According to this kernel mailinglist message hardware can support features inconsistently and those messages report those.

I don't expect that this can be fixed by software but maybe a BIOS, firmware or microcode update can help?

thanks for the link and the info, but i'm not sure there will be a firmware/bios update available, already tried googling it... with no success, unfortunately.

 

[TiagoLardaCrianca]

Ok, it was not a Proxmox problem or even a BIOS problem. Installing OPNsense instead of pfSense detected the network interfaces and the passtrough was made. I didn't quite read all the documentation on FreeBSD 12 before, but it appears that pfSense FreeBSD version didn't yet support the i226-v controllers. On OPNsense they work just fine and they are successfully passed-trough.

Cheers

 

[vanity2761]

Based on the messages, it seems you were able to pass through devices even though they weren't correctly listed on the Proxmox UI, correct?

I'm facing the same problem on the Asrock B760M Steel Legend board.

Also, how did you identify which PCIe address belongs to which device when all devices incorrectly show the vendor as "Intel Corporation"?

 

[vanity2761]

I was able to ascertain the type of device by comparing the device id listed in lspc with the output from pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""