iommu - setup with messages in logfile

[jeffgreg]

Hi Everyone,

I believe that I have followed the directions located on the Proxmox website to "properly" setup iommu support - Proxmox: PCI(e) Passthrough

The issue I am having is one device is clogging up my logifle with error messages:

cat /proc/cmdline
initrd=\EFI\proxmox\6.8.8-1-pve\initrd.img-6.8.8-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

dmesg (partial)
[  521.687388] DMAR: [DMA Read NO_PASID] Request device [07:00.0] fault addr 0xffc34000 [fault reason 0x06] PTE Read access is not set
[  552.143199] DMAR: DRHD: handling fault status reg 3

The motherboard is a Supermicro X13SAE-F with all of the SRV- options enabled (VT-d, SRV-IO, etc.)

I can't find any good way to suppress this device from IOMMU to stop the messages. Perhaps I misunderstand the nature of the messages/problem hence my posting looking for advice.

Some debugging information:

> cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

> cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci

> dmesg | grep 'remapping'
[    0.417633] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.419350] DMAR-IR: Enabled IRQ remapping in x2apic mode

> pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""  (pruned due to size)
│ 0x0c0310 │ 0x0035 │ 0000:07:00.0 │         19 │ 0x1033 │ OHCI USB Controller                                   │      │ 0x0001           │ Root Hub              │ 0x179
│ 0x0c0310 │ 0x0035 │ 0000:07:00.1 │         19 │ 0x1033 │ OHCI USB Controller                                   │      │ 0x0001           │ Root Hub              │ 0x179
│ 0x0c0320 │ 0x00e0 │ 0000:07:00.2 │         19 │ 0x1033 │ uPD72010x USB 2.0 Controller                          │      │ 0x0002           │ Root Hub              │ 0x179

In an effort to remove device 0.7.0 from IOMMU I added the following blacklist:
> cat /etc/modprobe.d/iommu_disable.conf
# Disable certain devices from being supported via IOMMU
options vfio-pci ids=1033:0035

> lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4640] (rev 02)
00:01.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x16 Controller #1 [8086:460d] (rev 02)
00:06.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 [8086:464d] (rev 02)
00:0a.0 Signal processing controller [1180]: Intel Corporation Platform Monitoring Technology [8086:467d] (rev 01)
00:14.0 USB controller [0c03]: Intel Corporation Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller [8086:7ae0] (rev 11)
00:14.2 RAM memory [0500]: Intel Corporation Alder Lake-S PCH Shared SRAM [8086:7aa7] (rev 11)
00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #0 [8086:7acc] (rev 11)
00:15.1 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH Serial IO I2C Controller #1 [8086:7acd] (rev 11)
00:16.0 Communication controller [0780]: Intel Corporation Alder Lake-S PCH HECI Controller #1 [8086:7ae8] (rev 11)
00:16.3 Serial controller [0700]: Intel Corporation Device [8086:7aeb] (rev 11)
00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-S PCH SATA Controller [AHCI Mode] [8086:7ae2] (rev 11)
00:1a.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #25 [8086:7ac8] (rev 11)
00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:7ac0] (rev 11)
00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:7ac4] (rev 11)
00:1c.0 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 [8086:7ab8] (rev 11)
00:1c.1 PCI bridge [0604]: Intel Corporation Alder Lake-S PCH PCI Express Root Port #2 [8086:7ab9] (rev 11)
00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:7abb] (rev 11)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:7a88] (rev 11)
00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11)
00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake-S PCH SMBus Controller [8086:7aa3] (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-S PCH SPI Controller [8086:7aa4] (rev 11)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (17) I219-LM [8086:1a1c] (rev 11)
01:00.0 Ethernet controller [0200]: Intel Corporation 82599 10 Gigabit Network Connection [8086:1557] (rev 01)
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal] [144d:a80c]
03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal] [144d:a80c]
05:00.0 Non-Volatile memory controller [0108]: Intel Corporation Optane NVME SSD P1600X Series [8086:2525]
06:00.0 PCI bridge [0604]: Integrated Technology Express, Inc. IT8893E PCIe to PCI Bridge [1283:8893] (rev 41)
07:00.0 USB controller [0c03]: NEC Corporation OHCI USB Controller [1033:0035] (rev 43)
07:00.1 USB controller [0c03]: NEC Corporation OHCI USB Controller [1033:0035] (rev 43)
07:00.2 USB controller [0c03]: NEC Corporation uPD72010x USB 2.0 Controller [1033:00e0] (rev 04)
08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-LM [8086:15f2] (rev 03)
09:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 06)
0a:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 52)

Everything seems to be working just fine. This is a brand new system that I am just provisioning - haven't yet done PCIe passthrough, trying to cleanup. However the number of messages in log make the logfiles mostly unusable so I really want to stop them.

Any help, suggestions, etc. would be greatly appreciated.
-Jeff

 

[leesteken]

Since kernel 6.8, intel_iommu=on is no longer necessary since it is on by default (but Proxmox did not update the manual yet).
If you are having issues with IOMMU, then try intel_iommu=off (and lose the ability to do PCI(e) passthrough) or try adding iommu=pt if you have trouble with devices that are NOT passed through. Or maybe disable the device in the motherboard BIOS or remove the add-in card or update the motherboard BIOS.

 

[jeffgreg]

Thanks for your reply, the intel_iommu=on was actually in the /etc/kernel/cmdline, I didn't actually add that. I did add the modules which were not in the /etc/modules file (I realized I forgot that in the above posting so I added it).

The problem isn't that things don't work, it's just the log file messages. Trying to make them go away.

-Jeff