Problems with IOMMU being detected

[waxy]

Hello everyone,

I'm new to PVE and I have been struggling with using IOMMU. I have been trying everything I have seen on the internet, but I can't get IOMMU to be detected even though it's enabled. I have reinstalled PVE a couple of times but still it was not working, I tried booting with ZFS and without ZFS and I get the same result. Am I doing something wrong or does my motherboard not support it? and what could be the problem.

Hardware:
- CPU: i3-8100
- Mobo: Asus TUF Z390-Plus Gaming
- RAM: Corsair Vengeance DDR4 16x2GB 2400MT/s

Kernel cmdline:
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
dmsg | gerp -i iommu output:

[    0.000000] Command line: initrd=\EFI\proxmox\6.8.8-4-pve\initrd.img-6.8.8-4-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
[    0.048739] Kernel command line: initrd=\EFI\proxmox\6.8.8-4-pve\initrd.img-6.8.8-4-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
[    0.048802] DMAR: IOMMU enabled
[    0.373662] iommu: Default domain type: Translated
[    0.373662] iommu: DMA domain TLB invalidation policy: lazy mode

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; doneoutput:

IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e1f] (rev 08)
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 08)
00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
00:1c.5 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 [8086:a33d] (rev f0)
00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Z390 Chipset LPC/eSPI Controller [8086:a305] (rev 10)
00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)

cat /proc/cmdline output:

initrd=\EFI\proxmox\6.8.8-4-pve\initrd.img-6.8.8-4-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

cat /etc/default/grub output:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Thanks in advance!

 

[dcsapak]

Hi,

maybe a dumb question: did you enable iommu in the bios? (you did not explicitly say so)
it may have a different name, e.g. VT-d depending on the bios/motherboard

also, could you post the complete output of the 'dmesg' command? maybe we can see there what might go wrong

 

[leesteken]

It looks like the Intel CPU and motherboard chipset support it. intel_iommu=on is not needed in kernel version 6.8 (since it is on by default), so that cannot be the problem. You need to check if VT-d is enabled in the motherboard BIOS and/or maybe ask Asus how to enable it.

 

[waxy]

Sorry I didn't mention this but the option VT-d is not in the BIOS but there is VMX which should enable VT-x as per Asus's guides. I have already enabled it before I did all the other steps in this guide https://pve.proxmox.com/wiki/PCI(e)_Passthrough I have also followed other guides on the internet but nothing worked. I cannot provide the full dmesgoutput because there is a post character limit.

 

[leesteken]

Sorry I didn't mention this but the option VT-d is not in the BIOS but there is VMX which should enable VT-x as per Asus's guides.

VT-x is for just running virtual machines. For IOMMU and device passthrough, you need to have the VT-d (or IOMMU) option in the BIOS (and enabled). Contact Asus for support, as only they can do something about it. There is nothing Proxmox or Linux kernel can do if it's not enabled by the motherboard BIOS.

 

[waxy]

Thank you very much for your help!

I got confused when following this guide from asus https://www.asus.com/support/faq/1043786/. I thought VT-x is the same technology as VT-d. Turned out I need to enable it from the "System Agent (SA) Configuration" tab instead of "CPU Configuration" tab in the BIOS.