After last update: Cannot open iommu_group: No such file or directory

U

Urbaman

Member
Jan 1, 2022
137
12
23
49
Anzola dell'Emilia, Bologna, Italy
www.dreamlord.it
Hi,

After last update, just one of my nodes throws the
Code: 
Cannot open iommu_group: No such file or directory
error.

The nodes are different Intel nodes, with integrated gpu, the other two are working fine:

Working:
Intel(R) Core(TM) i9-13900H
Intel(R) Core(TM) i7-8700T

Not working anymore:
Intel(R) Core(TM) i7-6700T

Is there any way to make it work again without rolling back all of the cluster?

Thank you.
 
I'm always updated with the kernel, just upgraded with the latest 6.8.12-4-pve

I do find these lines with dmesg | grep IOMMU on the failing node:

Code: 
[    0.259159] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.429491] pci 0000:00:02.0: DMAR: Disabling IOMMU for graphics on this chipset
 
Can you please re-check if the iommu parameter is correct in the /etc/default/grub?

I would also try test boot from an older kernel to narrow down the issue.
 
/etc/default/grub

Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

and /etc/kernel/cmdline

Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs

Are the very same on all three nodes (it should be enabled by default right?)

I'll try reboot on older kernel as soon as I'll get to the physical machine, but it worked till this we, and the VM to which the GPU was passed through didn't come up after the reboot for the new kernel version with the error in the title, meaning IOMMU was not working for the GPU anymore:

IOMMU for the failing node:
Code: 
dmesg | grep -e DMAR -e IOMMU
[    0.010128] ACPI: DMAR 0x00000000C9EC9000 0000A8 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.010160] ACPI: Reserving DMAR table memory at [mem 0xc9ec9000-0xc9ec90a7]
[    0.259125] DMAR: Host address width 39
[    0.259128] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.259138] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.259143] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.259148] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.259152] DMAR: RMRR base: 0x000000c9c38000 end: 0x000000c9c57fff
[    0.259156] DMAR: RMRR base: 0x000000cc000000 end: 0x000000ce7fffff
[    0.259159] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.259162] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.259165] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.260757] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.429491] pci 0000:00:02.0: DMAR: Disabling IOMMU for graphics on this chipset
[    0.483972] DMAR: No ATSR found
[    0.483975] DMAR: No SATC found
[    0.483979] DMAR: dmar1: Using Queued invalidation
[    0.485035] DMAR: Intel(R) Virtualization Technology for Directed I/O

pvesh get /nodes/pve2/hardware/pci --pci-class-blacklist ""
[...]
│ 0x030000 │ 0x1912 │ 0000:00:02.0 │         -1 │ 0x8086 │ HD Graphics 530                                                         │      │ 0x8055           │                                                 │ 0x103c           │ Hewle
[...]

IOMMU for a working node:
Code: 
dmesg | grep -e DMAR -e IOMMU
[    0.011826] ACPI: DMAR 0x000000009AD914A0 0000A8 (v01 LENOVO TC-M1U   00001710      01000013)
[    0.011885] ACPI: Reserving DMAR table memory at [mem 0x9ad914a0-0x9ad91547]
[    0.288800] DMAR: Host address width 39
[    0.288803] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.288815] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.288821] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.288827] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.288832] DMAR: RMRR base: 0x00000099aae000 end: 0x00000099acdfff
[    0.288836] DMAR: RMRR base: 0x0000009d000000 end: 0x0000009f7fffff
[    0.288840] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.288844] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.288847] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.290890] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.627127] DMAR: No ATSR found
[    0.627131] DMAR: No SATC found
[    0.627134] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.627136] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.627140] DMAR: IOMMU feature nwfs inconsistent
[    0.627144] DMAR: IOMMU feature pasid inconsistent
[    0.627147] DMAR: IOMMU feature eafs inconsistent
[    0.627151] DMAR: IOMMU feature prs inconsistent
[    0.627154] DMAR: IOMMU feature nest inconsistent
[    0.627157] DMAR: IOMMU feature mts inconsistent
[    0.627161] DMAR: IOMMU feature sc_support inconsistent
[    0.627164] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.627168] DMAR: dmar0: Using Queued invalidation
[    0.627177] DMAR: dmar1: Using Queued invalidation
[    0.629303] DMAR: Intel(R) Virtualization Technology for Directed I/O

pvesh get /nodes/pve1/hardware/pci --pci-class-blacklist ""
[...]
 0x030000 │ 0xa7a0 │ 0000:00:02.0 │          0 │ 0x8086 │ Raptor Lake-P [Iris Xe Graphics]                     │      │ 0x0000           │                                         │ 0x0000           │

[...]
 
Last edited:
Hi @Moayad

Can confirm that 6.8.12-3-pve still manages to make the IOMMU work for the GPU on the "older" node, while 6.8.12-4-pve does not recognize the GPU as compatible.

Code: 
pvesh get /nodes/pve2/hardware/pci --pci-class-blacklist ""
[...]
 0x030000 │ 0x1912 │ 0000:00:02.0 │          0 │ 0x8086 │ HD Graphics 530                                                         │      │ 0x8055           │                                                 │ 0x103c           │
[...]
 
Last edited:
same here with 6.8.12-4-pve

Code: 
root@proxmox:~# dmesg | grep -e DMAR -e IOMMU
[    0.011454] ACPI: DMAR 0x00000000D9FC2000 0000A8 (v01 INTEL  KBL      00000001 INTL 00000001)
[    0.011497] ACPI: Reserving DMAR table memory at [mem 0xd9fc2000-0xd9fc20a7]
[    0.034656] DMAR: IOMMU enabled
[    0.103698] DMAR: Host address width 39
[    0.103699] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.103711] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.103713] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.103718] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.103720] DMAR: RMRR base: 0x000000d9c7a000 end: 0x000000d9c99fff
[    0.103723] DMAR: RMRR base: 0x000000dc000000 end: 0x000000de7fffff
[    0.103726] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.103728] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.103730] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.105645] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.294754] pci 0000:00:02.0: DMAR: Disabling IOMMU for graphics on this chipset
[    0.350753] DMAR: No ATSR found
[    0.350755] DMAR: No SATC found
[    0.350757] DMAR: dmar1: Using Queued invalidation
[    0.351431] DMAR: Intel(R) Virtualization Technology for Directed I/O

changed grub to 6.8.12-2
Code: 
#GRUB_DEFAULT=0
GRUB_DEFAULT="gnulinux-advanced-48953669-7334-4ed5-a6e4-9c688122ac55>gnulinux-6.8.12-2-pve-advanced-48953669-7334-4ed5-a6e4-9c688122ac55"

and now I get:
Code: 
root@proxmox:~# dmesg | grep -e DMAR -e IOMMU
[    0.011821] ACPI: DMAR 0x00000000D9FC2000 0000A8 (v01 INTEL  KBL      00000001 INTL 00000001)
[    0.011864] ACPI: Reserving DMAR table memory at [mem 0xd9fc2000-0xd9fc20a7]
[    0.034972] DMAR: IOMMU enabled
[    0.102362] DMAR: Host address width 39
[    0.102363] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.102373] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.102376] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.102380] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.102382] DMAR: RMRR base: 0x000000d9c7a000 end: 0x000000d9c99fff
[    0.102384] DMAR: RMRR base: 0x000000dc000000 end: 0x000000de7fffff
[    0.102387] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.102388] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.102390] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.104309] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.347182] DMAR: No ATSR found
[    0.347183] DMAR: No SATC found
[    0.347185] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.347186] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.347188] DMAR: IOMMU feature nwfs inconsistent
[    0.347189] DMAR: IOMMU feature eafs inconsistent
[    0.347190] DMAR: IOMMU feature prs inconsistent
[    0.347191] DMAR: IOMMU feature nest inconsistent
[    0.347192] DMAR: IOMMU feature mts inconsistent
[    0.347193] DMAR: IOMMU feature sc_support inconsistent
[    0.347194] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.347196] DMAR: dmar0: Using Queued invalidation
[    0.347200] DMAR: dmar1: Using Queued invalidation
[    0.348845] DMAR: Intel(R) Virtualization Technology for Directed I/O
[   14.329658] DMAR: DRHD: handling fault status reg 2
[   14.329664] DMAR: [DMA Write NO_PASID] Request device [00:02.0] fault addr 0x0 [fault reason 0x02] Present bit in context entry is clear
 
Same here. I'm using an iGPU HD graphics 530 from a i5-6600 CPU.
It has always worked with the GURB line
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=sysfb_init video=simplefb:off video=vesafb:off video=efifb:off video=vesa:off disable_vga=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=i915"
Reverting to an older kernel (6.8.12-3-pve) everyting goes back to normal.
 
Last edited:
Same here - i7-6700 Intel HD Graphics - reverted kernel but wondering if there is long-term fix for supporting this chipset?
 
Easy enough to revert to previous kernel to fix.. Looking for long term solution though.
 
Hi, same here, Intel i7-6700t.

Kind of scared me, googling only brought up a kernel patch disabling the graphics of the chipset. Hopefully only for Broadwell processors, that was reassuring.
Still no idea why on kernel 6.8.12-4 DMAR is disabling graphics.
 
Also present on Intel i7-6770HQ.

Code: 
cat /etc/default/grub

[...]
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""
[...]

Code: 
pvesh get /nodes/prox1/hardware/pci --pci-class-blacklist ""

[...]
0x030000 │ 0x193b │ 0000:00:02.0 │          0 │ 0x8086 │ Iris Pro Graphics 580
[...]

`proxmox-boot-tool kernel pin 6.8.12-3-pve` works to re-enable.
Is it safe to update the rest of the packages after rolling back and pinning the older kernel?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back