[SOLVED] Yet Another IOMMU Not Detected Thread

hchasens

New Member
Feb 2, 2021
2
0
1
21
Yet Another IOMMU Not Detected Thread

This is solved. I was going to post this as a question but I solved it so now I'm posting it so that others will not follow in my footsteps. Here is my story (or just skip to the end for my solution).

Hardware:
  • Dell Poweredge R620
    • 48Gigs of registered ECC
    • CPU(s) 32 x Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2 Sockets)
      • (Yes the CPU supports Virtualisations and It's been turned on in the BIOS) (and yes I've made sure its been turned on the in the bios twice)
Software
  • Kernel-Version (I installed it last night (Feb 1st 2021)
    • Linux 5.4.73-1-pve #1 SMP PVE 5.4.73-1

The following line is in my Grub config
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
I ran
Code:
update-grub
after.

My /etc/modules looks like
Code:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

and I can confirm they're loaded

Code:
root@nike:~# lsmod | grep vfio
vfio_pci               53248  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 vfio_pci,kvm
vfio_iommu_type1       32768  0
vfio                   32768  2 vfio_iommu_type1,vfio_pci

Then I rebooted the system and got this when I went to check if IOMMU was working.

Code:
root@nike:~# dmesg | grep -e DMAR -e IOMMU
[    0.014651] ACPI: DMAR 0x00000000AD3346F4 000130 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.387126] DMAR: Host address width 46
[    0.387128] DMAR: DRHD base: 0x000000bb000000 flags: 0x0
[    0.387134] DMAR: dmar0: reg_base_addr bb000000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.387137] DMAR: DRHD base: 0x000000dc900000 flags: 0x1
[    0.387142] DMAR: dmar1: reg_base_addr dc900000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.387145] DMAR: RMRR base: 0x000000af458000 end: 0x000000af46ffff
[    0.387147] DMAR: RMRR base: 0x000000af450000 end: 0x000000af450fff
[    0.387148] DMAR: RMRR base: 0x000000af452000 end: 0x000000af452fff
[    0.387150] DMAR: ATSR flags: 0x0
[    0.387154] DMAR-IR: IOAPIC id 2 under DRHD base  0xbb000000 IOMMU 0
[    0.387156] DMAR-IR: IOAPIC id 0 under DRHD base  0xdc900000 IOMMU 1
[    0.387158] DMAR-IR: IOAPIC id 1 under DRHD base  0xdc900000 IOMMU 1
[    0.387160] DMAR-IR: HPET id 0 under DRHD base 0xdc900000
[    0.387162] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.387162] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.387672] DMAR-IR: Enabled IRQ remapping in xapic mode

Nothing showed up in iommu_groups either.

Code:
root@nike:~# find /sys/kernel/iommu_groups/ -type l
root@nike:~#

I then wondered if it was bug or error in my motherboard and ran.

Code:
root@nike:~# dmesg | grep "Virtualization Technology for Directed I/O"
root@nike:~#

This returned nothing so then I ran

Code:
root@nike:~# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       46 bits physical, 48 bits virtual
CPU(s):              32
On-line CPU(s) list: 0-31
Thread(s) per core:  2
Core(s) per socket:  8
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               45
Model name:          Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
Stepping:            7
CPU MHz:             3018.247
CPU max MHz:         3000.0000
CPU min MHz:         1200.0000
BogoMIPS:            4399.99
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            20480K
NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

So now I know virtualization is active and it's not my hardware.

I then download libvirt-clients.


Code:
root@nike:~# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'freezer' controller support                     : PASS
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

It's like the kernel isn't getting the flags


I've already checked and read the following resources:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
https://pve.proxmox.com/wiki/Pci_passthrough#Intel_CPU
https://forum.proxmox.com/threads/iommu-broke-solved.53188/
https://heiko-sieger.info/iommu-groups-what-you-need-to-consider/
https://bbs.archlinux.org/viewtopic.php?id=259967
https://wiki.archlinux.org/index.php/Intel_GVT-g#Prerequisite
https://forum.proxmox.com/threads/pci-passthrough-no-iommu-detected.67504/
https://www.reddit.com/r/Proxmox/comments/d8e714/no_iommu_detected_pcie_passthrough/



Solution:

Turns out Proxmox doesn't install with GRUB by default on all installs. I've never actually seen my system boot because I control it via iDrac so I never noticed. 90% of the documentation is for GRUB too. I solved it by trying the systemd-boot config. (brain fart)
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!