IOMMU enabled but not detected

Jun 3, 2022
4
0
6
Hello all. I just started tinkering with proxmox, and am trying to pass the integrated graphics to an Ubuntu VM.
I can't quite get it to work, so I'd like some help troubleshooting what's wrong.

This is running on a Minisforum TH50 (cpu: i5-11320h).
Essentially, I've tried every method I can for enabling the IOMMU, but adding a pci device to a VM through the management interface gives me the "No IOMMU detected, please activate it.See Documentation for further information" message.
I should note that my BIOS does not have a IOMMU or VT-d option, but Minisforum has said for other products that this feature is forced on and hidden in the BIOS. I've sent support an email asking if that is true for the TH50 as well, but I figured I'd still post this here in case I missed something else.

This is my kernel command line:
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet vfio-pci.ids=8086:9a49 intel_iommu=on "
I haven't changed any other line in /etc/default/grub

I'm pretty sure I can get rid of the vfio-pci.ids=8086:9a49 part of the kernel command line, since it doesn't seem to apply that driver anyway (as below)

The output of lspci -nnkv -s 00:02:
Code:
00:02.0 VGA compatible controller [0300]: Intel Corporation TigerLake GT2 [Iris Xe Graphics] [8086:9a49] (rev 03) (prog-if 00 [VGA controller])
        Subsystem: Device [1e50:800d]
        Flags: bus master, fast devsel, latency 0, IRQ 255
        Memory at 607d000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 3000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
        Kernel modules: i915

and the output of dmesg | grep -i iommu:
Code:
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.30-2-pve root=/dev/mapper/pve-root ro quiet vfio-pci.ids=8086:9a49 intel_iommu=on
[    0.040428] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.30-2-pve root=/dev/mapper/pve-root ro quiet vfio-pci.ids=8086:9a49 intel_iommu=on
[    0.040467] DMAR: IOMMU enabled
[    0.685431] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.723207] iommu: Default domain type: Translated
[    0.723207] iommu: DMA domain TLB invalidation policy: lazy mode

The "Skip IOMMU disabling for graphics" seems important, but I don't know what it means.
I also installed kvm-ok, which gives this result:
Code:
INFO: /dev/kvm exists
KVM acceleration can be used

Have I made any glaring mistakes? If not, what should my next steps be?
Thanks
 
Please post the full output from the PVE-host in code-tags of:
Bash:
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
(Thanks to avw for the command!)
 
Here it is
Code:
IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 11th Gen Core Processor Host Bridge/DRAM Registers [8086:9a14] (rev 02)
00:02.0 VGA compatible controller [0300]: Intel Corporation TigerLake GT2 [Iris Xe Graphics] [8086:9a49] (rev 03)
00:06.0 PCI bridge [0604]: Intel Corporation 11th Gen Core Processor PCIe Controller [8086:9a09] (rev 02)
00:07.0 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #0 [8086:9a23] (rev 02)
00:07.1 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #1 [8086:9a25] (rev 02)
00:07.2 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #2 [8086:9a27] (rev 02)
00:07.3 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #3 [8086:9a29] (rev 02)
00:0d.0 USB controller [0c03]: Intel Corporation Tiger Lake-LP Thunderbolt 4 USB Controller [8086:9a13] (rev 02)
00:0d.2 USB controller [0c03]: Intel Corporation Tiger Lake-LP Thunderbolt 4 NHI #0 [8086:9a1b] (rev 02)
00:0d.3 USB controller [0c03]: Intel Corporation Tiger Lake-LP Thunderbolt 4 NHI #1 [8086:9a1d] (rev 02)
00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-LP USB 3.2 Gen 2x1 xHCI Host Controller [8086:a0ed] (rev 30)
00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-LP Shared SRAM [8086:a0ef] (rev 30)
00:15.0 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #0 [8086:a0e8] (rev 30)
00:15.1 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #1 [8086:a0e9] (rev 30)
00:15.2 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #2 [8086:a0ea] (rev 30)
00:15.3 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #3 [8086:a0eb] (rev 30)
00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-LP Management Engine Interface [8086:a0e0] (rev 30)
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a0d3] (rev 30)
00:19.0 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #4 [8086:a0c5] (rev 30)
00:19.1 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #5 [8086:a0c6] (rev 30)
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a0b8] (rev 30)
00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a0bc] (rev 30)
00:1c.5 PCI bridge [0604]: Intel Corporation Tigerlake PCH-LP PCI Express Root Port #6 [8086:a0bd] (rev 30)
00:1d.0 PCI bridge [0604]: Intel Corporation Tiger Lake-LP PCI Express Root Port #9 [8086:a0b0] (rev 30)
00:1e.0 Communication controller [0780]: Intel Corporation Tiger Lake-LP Serial IO UART Controller #0 [8086:a0a8] (rev 30)
00:1e.3 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO SPI Controller #1 [8086:a0ab] (rev 30)
00:1f.0 ISA bridge [0601]: Intel Corporation Tiger Lake-LP LPC Controller [8086:a082] (rev 30)
00:1f.3 Multimedia audio controller [0401]: Intel Corporation Tiger Lake-LP Smart Sound Technology Audio Controller [8086:a0c8] (rev 30)
00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-LP SMBus Controller [8086:a0a3] (rev 30)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP SPI Controller [8086:a0a4] (rev 30)
01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a809]
af:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
b0:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
b1:00.0 Network controller [0280]: MEDIATEK Corp. Device [14c3:0608
 
IOMMU is not enabled (because it is all in one group *). Check with cat /proc/cmdline whether intel_iommu=on is actually there. If that's the case then it's a motherboard/BIOS issue. If it is not there, then maybe you are booting in UEFI mode with ZFS or you did not run update-grub?

EDIT: Turns out that DMAR: IOMMU enabled already (and only!) signals that the setting is active. I didn't know that before.
 
Last edited:
IOMMU is not enabled (because it is all in one group *). Check with cat /proc/cmdline whether intel_iommu=on is actually there. If that's the case then it's a motherboard/BIOS issue. If it is not there, then maybe you are booting in UEFI mode with ZFS or you did not run update-grub?

EDIT: Turns out that DMAR: IOMMU enabled already (and only!) signals that the setting is active. I didn't know that before.

What do you mean by "maybe you are booting in UEFI mode with ZFS" ?

I am booting Proxmox with UEFI and ZFS (for boot drive). Could this be why I am having trouble getting IOMMU groups activated?

Machine is a Dell Precision Rack 7910 with Intel Virtualization turned on in BIOS
... but Proxmox is saying "No IOMMU detected" and /sys/kernel/iommu_groups is empty.

$ cat /proc/cmdline
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

Any help appreciated
 
  • Like
Reactions: semanticbeeng
What do you mean by "maybe you are booting in UEFI mode with ZFS" ?
Maybe you are not using GRUB but systemd-boot instead, which you can check according to the manual (to which I provided a link).
I am booting Proxmox with UEFI and ZFS (for boot drive). Could this be why I am having trouble getting IOMMU groups activated?
Yes because then you need to put kernel parameters in /etc/kernel/cmdline and not /etc/default/grub, because your system using systemd-boot instead of GRUB.
Machine is a Dell Precision Rack 7910 with Intel Virtualization turned on in BIOS
... but Proxmox is saying "No IOMMU detected" and /sys/kernel/iommu_groups is empty.
This is because kernel parameters are in the wrong configuration file.
$ cat /proc/cmdline
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
As you can see intel_iommu=on is missing and that line looks nothing like your GRUB_CMDLINE_LINUX_DEFAULT="quiet vfio-pci.ids=8086:9a49 intel_iommu=on ".
Any help appreciated
Add kernel parameters to the first (and only!) line in /etc/kernel/cmdline and run proxmox-boot-tool refresh (instead of /etc/default/grub and update-grub). See the manual for more information about this.
 
  • Like
Reactions: AgingOrange
Thanks... this thread allowed me to (finally) track down this problem.

For future readers:
.. because I am using a mirror zfs as my boot drive, the update-grub command will throw some errors
.. instead of editing /etc/default/grub with "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
.. you need to add "intel_iommu=on" to the end of existing line on /etc/kernel/cmdline (all on one line)
.. then run "proxmox-boot-tool refresh"
.. reboot & I have my IOMMU groups :)
 
Thanks... this thread allowed me to (finally) track down this problem.

For future readers:
.. because I am using a mirror zfs as my boot drive, the update-grub command will throw some errors
.. instead of editing /etc/default/grub with "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
.. you need to add "intel_iommu=on" to the end of existing line on /etc/kernel/cmdline (all on one line)
.. then run "proxmox-boot-tool refresh"
.. reboot & I have my IOMMU groups :)

This↓ helped so much thanks!!!
"instead of editing /etc/default/grub with "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
.. you need to add "intel_iommu=on" to the end of existing line on /etc/kernel/cmdline (all on one line)"
 
Thanks... this thread allowed me to (finally) track down this problem.

For future readers:
.. because I am using a mirror zfs as my boot drive, the update-grub command will throw some errors
.. instead of editing /etc/default/grub with "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
.. you need to add "intel_iommu=on" to the end of existing line on /etc/kernel/cmdline (all on one line)
.. then run "proxmox-boot-tool refresh"
.. reboot & I have my IOMMU groups :)
This is exactly my setup, and the Proxmox manual doesn't do a great job of indicating that the configuration we have (assuming yours is similar), which looks as below, is systemd and not grub:

Code:
Boot0005* UEFI OS       HD(2,GPT,<UUID>)/File(\EFI\BOOT\BOOTX64.EFI)
Boot0006* UEFI OS       HD(2,GPT,<UUID>)/File(\EFI\BOOT\BOOTX64.EFI)

Anyway, thanks for getting me on the right track.
 
Thanks... this thread allowed me to (finally) track down this problem.

For future readers:
.. because I am using a mirror zfs as my boot drive, the update-grub command will throw some errors
.. instead of editing /etc/default/grub with "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
.. you need to add "intel_iommu=on" to the end of existing line on /etc/kernel/cmdline (all on one line)
.. then run "proxmox-boot-tool refresh"
.. reboot & I have my IOMMU groups :)
Hello,

I've tried this, but still IOMMU is not detected. My device is a NUC8i3BEH.

root@proxmox1:~# cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs
GRUB_CMDLINE_LINUX_DEFAULT=“quiet intel_iommu=on”

root@proxmox1:~# cat /proc/cmdline
initrd=\EFI\proxmox\6.5.11-6-pve\initrd.img-6.5.11-6-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

I could not figure out what I did wrong
 
I could not figure out what I did wrong
root@proxmox1:~# cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs GRUB_CMDLINE_LINUX_DEFAULT=“quiet intel_iommu=on”
This is wrong and your also mixing GRUB with systemd-boot. Did you read the manual?
The contents of /etc/kernel/cmdline should be only one line: root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on
 
  • Like
Reactions: kifeo
.. because I am using a mirror zfs as my boot drive, the update-grub command will throw some errors
.. instead of editing /etc/default/grub with "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
Can you elaborate on what the ZFS error is and how it interferes with the kernel command line?

context: I am working through some setup and my `iommu` seems detected even with the kernel command line in screenshot attached.
My current issue is GPU not detected and wip on it.

Code:
# dmesg | grep -e DMAR -e IOMMU

...
DMAR: Intel-IOMMU force enabled due to platform opt-in
DMAR: Intel(R) Virtualization Technology for Directed I/O
# "When you see this, you're pretty much past all the failure points of initializing VT-d"

# dmesg | grep 'remapping'

...
DMAR-IR: Enabled IRQ remapping in x2apic mode
 

Attachments

  • iommu.jpg
    iommu.jpg
    196.8 KB · Views: 40
Last edited:
I had the issue of the IOMMU enable in BIOS but not detected.
I did change the /etc/default/grub with:
sudo nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt

update-grub
BUT > The IOMMU was not detected

What solved it for me was:
:cool: apt full-upgrade


After the change perform "update-grub"
reboot
then check if the parameters in /etc/default/grub are de-facto used in grub with:
cat /proc/cmdline
You will obtain:
BOOT_IMAGE=/boot/<THEBOOTIMAGE> root=/dev/mapper/pve-root ro quiet <other params>

If you do not see your changes
apt full-upgrade
and try to make cat /proc/cmdline output the right params.

Hope this helps
 
Hi everybody,

This post is giving me some hope since I am running on the very same hardware Minisforum TH50 (cpu: i5-11320h).

So starting from the beginning, determining which bootloader is used:
efibootmgr -v
Code:
BootCurrent: 0003
Timeout: 1 seconds
BootOrder: 0003,0008,0000
Boot0000  Windows Boot Manager  VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}....M...............
Boot0003* proxmox       HD(2,GPT,682e7d0f-9ca7-4511-88e5-28812d83ffae,0x800,0x100000)/File(\EFI\PROXMOX\GRUBX64.EFI)
Boot0008* UEFI OS       HD(2,GPT,682e7d0f-9ca7-4511-88e5-28812d83ffae,0x800,0x100000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

So it's well GRUB.

Checking /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
is well present ... and
update-grubwas done.


Code:
# cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-6.8.12-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on

Code:
# dmesg | grep -e DMAR -e IOMMU
[    0.039901] DMAR: IOMMU enabled
[    0.675678] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics

Bash:
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Code:
IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 11th Gen Core Processor Host Bridge/DRAM Registers [8086:9a14] (rev 02)
00:02.0 VGA compatible controller [0300]: Intel Corporation TigerLake-LP GT2 [Iris Xe Graphics] [8086:9a49] (rev 03)
00:06.0 PCI bridge [0604]: Intel Corporation 11th Gen Core Processor PCIe Controller [8086:9a09] (rev 02)
00:07.0 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #0 [8086:9a23] (rev 02)
00:07.1 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #1 [8086:9a25] (rev 02)
00:07.2 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #2 [8086:9a27] (rev 02)
00:07.3 PCI bridge [0604]: Intel Corporation Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #3 [8086:9a29] (rev 02)
00:0d.0 USB controller [0c03]: Intel Corporation Tiger Lake-LP Thunderbolt 4 USB Controller [8086:9a13] (rev 02)
00:0d.2 USB controller [0c03]: Intel Corporation Tiger Lake-LP Thunderbolt 4 NHI #0 [8086:9a1b] (rev 02)
00:0d.3 USB controller [0c03]: Intel Corporation Tiger Lake-LP Thunderbolt 4 NHI #1 [8086:9a1d] (rev 02)
00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-LP USB 3.2 Gen 2x1 xHCI Host Controller [8086:a0ed] (rev 30)
00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-LP Shared SRAM [8086:a0ef] (rev 30)
00:15.0 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #0 [8086:a0e8] (rev 30)
00:15.1 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #1 [8086:a0e9] (rev 30)
00:15.2 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #2 [8086:a0ea] (rev 30)
00:15.3 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #3 [8086:a0eb] (rev 30)
00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-LP Management Engine Interface [8086:a0e0] (rev 30)
00:17.0 SATA controller [0106]: Intel Corporation Tiger Lake-LP SATA Controller [8086:a0d3] (rev 30)
00:19.0 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #4 [8086:a0c5] (rev 30)
00:19.1 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #5 [8086:a0c6] (rev 30)
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a0b8] (rev 30)
00:1c.4 PCI bridge [0604]: Intel Corporation Tiger Lake-LP PCI Express Root Port [8086:a0bc] (rev 30)
00:1c.5 PCI bridge [0604]: Intel Corporation Tigerlake PCH-LP PCI Express Root Port #6 [8086:a0bd] (rev 30)
00:1d.0 PCI bridge [0604]: Intel Corporation Tiger Lake-LP PCI Express Root Port #9 [8086:a0b0] (rev 30)
00:1e.0 Communication controller [0780]: Intel Corporation Tiger Lake-LP Serial IO UART Controller #0 [8086:a0a8] (rev 30)
00:1e.3 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP Serial IO SPI Controller #1 [8086:a0ab] (rev 30)
00:1f.0 ISA bridge [0601]: Intel Corporation Tiger Lake-LP LPC Controller [8086:a082] (rev 30)
00:1f.3 Multimedia audio controller [0401]: Intel Corporation Tiger Lake-LP Smart Sound Technology Audio Controller [8086:a0c8] (rev 30)
00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-LP SMBus Controller [8086:a0a3] (rev 30)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-LP SPI Controller [8086:a0a4] (rev 30)
01:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 660P Series [8086:f1a8] (rev 03)
af:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
b0:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
b1:00.0 Network controller [0280]: MEDIATEK Corp. MT7921K (RZ608) Wi-Fi 6E 80MHz [14c3:0608]
So, if i am following well, IOMMU groups are missing... but I have no clue how to get it.... HELP...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!