[SOLVED] Unable to Activate IOMMU - despite active VT-X

[DavidKahl]

Hi All,

over the last few days i've been trying to activate PCI Passthrough using this the provided guide in the Proxmox Knowledge base. My system is on Proxmox 7.0-11

My CPU supports VT-x and its been activated in the BIOS however whaterver i try i'm unable to activate IOMMU for my INTEL CPU.

Running efibootmgr -v

BootCurrent: 0003
Timeout: 0 seconds
BootOrder: 0001,0002,0003,0004,0005,0006,0007,0008
Boot0001* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000001)AMBO
Boot0002* USB Hard Drive        VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0200000001)AMBO
Boot0003* UEFI: CT500MX500SSD1  PciRoot(0x0)/Pci(0x1f,0x2)/Sata(1,65535,0)/HD(2,GPT,c558e10c-311a-4525-b442-a20bd0aceddc,0x800,0x100000)AMBO
Boot0004* ATAPI CD-ROM Drive    VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0300000001)AMBO
Boot0005* CD/DVD Drive  BBS(CDROM,,0x0)AMGOAMNO........o.h.p. . . . . . . .C.D.D.V.D.W. .S.H.-.2.1.6.A.L.N....................A...........................>..Gd-.;.A..MQ..L.8.R.H.L.G.6.C.C.0.4.4.3.D.M. . . . . . ......AMBO
Boot0006* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000000)AMBO
Boot0007* Hard Drive    BBS(HD,,0x0)AMGOAMNO........o.H.i.t.a.c.h.i. .H.D.S.7.2.3.0.2.0.B.L.A.6.4.2....................A...........................>..Gd-.;.A..MQ..L. . . . . . .N.M.2.1.0.4.3.F.W.3.V.1.D.6......AMBOAMNO........o.C.T.5.0.0.M.X.5.0.0.S.S.D.1....................A...........................>..Gd-.;.A..MQ..L.1.2.5.0.C.2.5.D.B.8.A.9. . . . . . . . ......AMBO
Boot0008* Realtek PXE B03 D00   BBS(Network,,0x0)AMBO

I've tried both GRUB and systemd-boot but neither seems to be able to get this working.

I'm open to any help and recommendations.

Thanks

David

 

[oguz]

hi,

VT-x is not enough for IOMMU, you will also need to enable VT-d, otherwise your IOMMU groups most likely won't show up.

which motherboard are you using? there should be an option for that in the BIOS.

 

[DavidKahl]

Its an HP Motherboard - but i'll check the BIOS Setting there and revert. But thats a great hint. Thanks.

 

[leesteken]

In both GRUB (selection top-left of screen) and systemd-boot (selection in center of screen), you can press e to change the boot line and add intel_iommu=on (to the correct line). This way you can make sure it is applied (until a reboot) and verify it with cat /proc/cmdline.
Once VT-d/IOMMU is enabled in your BIOS (if supported, and supported by your specific CPU) and it is clear which boot loader is used, we can help making the setting permanent.

 

[DavidKahl]

So I had a chance to work on the device today and VT-x/Vt-d is the same Setting in the BIOS. After turning it to disabled and back to enabled the server gave me the following output on: dmesg | grep -e DMAR -e IOMMU

[ 0.000000] DMAR: IOMMU enabled [ 0.000000] DMAR: IOMMU enabled

Strangely despite it being active when you add a PCI passthrough it tells you IOMMU isn't enabled. I'm a loss :S

 

[leesteken]

The definitive test whether IOMMU is working and passthrough is possible is by listing the groups using something like for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done. This will also tell us which devices are isolated properly for passthrough.
Please show cat /proc/cmdline to check if intel_iommu=on is present and lsmod | grep vfio to check that the required kernel modules are loaded.
Maybe you can tell us which CPU and motherboard you are using exactly? Maybe we can verify VT-d support or get some pointers via the motherboard manual?

 

[DavidKahl]

So the following was yielded by:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done

IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [8086:0150] (rev 09)
        Subsystem: Hewlett-Packard Company Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [103c:2ada]
        Kernel driver in use: ivb_uncore
        Kernel modules: ie31200_edac
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
        Kernel driver in use: pcieport
00:02.0 Display controller [0380]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0152] (rev 09)
        DeviceName:  Onboard IGD
        Subsystem: Hewlett-Packard Company Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [103c:2ada]
        Kernel driver in use: i915
        Kernel modules: i915
00:16.0 Communication controller [0780]: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 [8086:1c3a] (rev 04)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family MEI Controller [103c:2ada]
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family USB Enhanced Host Controller [103c:2ada]
        Kernel driver in use: ehci-pci
        Kernel modules: ehci_pci
00:1b.0 Audio device [0403]: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller [8086:1c20] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family High Definition Audio Controller [103c:2ada]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b5)
        Kernel driver in use: pcieport
00:1c.5 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 [8086:1c1a] (rev b5)
        Kernel driver in use: pcieport
00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family USB Enhanced Host Controller [103c:2ada]
        Kernel driver in use: ehci-pci
        Kernel modules: ehci_pci
00:1f.0 ISA bridge [0601]: Intel Corporation H61 Express Chipset LPC Controller [8086:1c5c] (rev 05)
        Subsystem: Hewlett-Packard Company H61 Express Chipset LPC Controller [103c:2ada]
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [8086:1c02] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:2ada]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1f.3 SMBus [0c05]: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller [8086:1c22] (rev 05)
        Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family SMBus Controller [103c:2ada]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GT 630 OEM] [10de:0fc2] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GK107 [GeForce GT 630 OEM] [1462:2750]
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GK107 HDMI Audio Controller [1462:2750]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
02:00.0 USB controller [0c03]: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller [104c:8241] (rev 02)
        Subsystem: Hewlett-Packard Company TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller [103c:2ada]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 07)
        Subsystem: Hewlett-Packard Company RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [103c:2ada]
        Kernel driver in use: r8169
        Kernel modules: r8169

cat /proc/cmdline gave me the confirmation that Intel IOMMU was on:

BOOT_IMAGE=/boot/vmlinuz-5.11.22-4-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on

and finally lsmod | grep vfio

vfio_pci               57344  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  10 vfio_pci,kvm
vfio_iommu_type1       36864  0
vfio                   36864  2 vfio_iommu_type1,vfio_pci

The Processor is a

4 x Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz

Checking the Intel Ref page its stated that the CPU support VT-x/VT-d.

Board im not 100% sure which model it is and will need to revert on that.

Really appreciate the help.

 

[leesteken]

Since there is only one IOMMU group (numbered *), the IOMMU is not enabled according to the kernel. /proc/cmdline correctly contains intel_iommu=on to turn it on (if available). The VFIO kernel modules (vfio_pci etc.) are correctly loaded.
However if i5-3450 is your processor, it does not support VT-d (IOMMU) according to Intel's product specification. It states No after Intel® Virtualization Technology for Directed I/O (VT-d). Or did I make a mistake, read it wrong or is this not your processor?

 

[DavidKahl]

I just rechecked and seems youre right - I was looking at the VT-x option on the page. Hmm looks like it'll need an upgrade.

Thanks for your support - much appreciated.

BR

D