[SOLVED] Yet Another IOMMU Not Detected Thread

[hchasens]

Yet Another IOMMU Not Detected Thread

This is solved. I was going to post this as a question but I solved it so now I'm posting it so that others will not follow in my footsteps. Here is my story (or just skip to the end for my solution).

Hardware:

• Dell Poweredge R620
  • 48Gigs of registered ECC
  • CPU(s) 32 x Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2 Sockets)
    • (Yes the CPU supports Virtualisations and It's been turned on in the BIOS) (and yes I've made sure its been turned on the in the bios twice)

Software

• Kernel-Version (I installed it last night (Feb 1st 2021)
  • Linux 5.4.73-1-pve #1 SMP PVE 5.4.73-1

The following line is in my Grub config

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

I ran

update-grub

after.

My /etc/modules looks like

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

and I can confirm they're loaded

root@nike:~# lsmod | grep vfio
vfio_pci               53248  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 vfio_pci,kvm
vfio_iommu_type1       32768  0
vfio                   32768  2 vfio_iommu_type1,vfio_pci

Then I rebooted the system and got this when I went to check if IOMMU was working.

root@nike:~# dmesg | grep -e DMAR -e IOMMU
[    0.014651] ACPI: DMAR 0x00000000AD3346F4 000130 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.387126] DMAR: Host address width 46
[    0.387128] DMAR: DRHD base: 0x000000bb000000 flags: 0x0
[    0.387134] DMAR: dmar0: reg_base_addr bb000000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.387137] DMAR: DRHD base: 0x000000dc900000 flags: 0x1
[    0.387142] DMAR: dmar1: reg_base_addr dc900000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.387145] DMAR: RMRR base: 0x000000af458000 end: 0x000000af46ffff
[    0.387147] DMAR: RMRR base: 0x000000af450000 end: 0x000000af450fff
[    0.387148] DMAR: RMRR base: 0x000000af452000 end: 0x000000af452fff
[    0.387150] DMAR: ATSR flags: 0x0
[    0.387154] DMAR-IR: IOAPIC id 2 under DRHD base  0xbb000000 IOMMU 0
[    0.387156] DMAR-IR: IOAPIC id 0 under DRHD base  0xdc900000 IOMMU 1
[    0.387158] DMAR-IR: IOAPIC id 1 under DRHD base  0xdc900000 IOMMU 1
[    0.387160] DMAR-IR: HPET id 0 under DRHD base 0xdc900000
[    0.387162] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.387162] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.387672] DMAR-IR: Enabled IRQ remapping in xapic mode

Nothing showed up in iommu_groups either.

root@nike:~# find /sys/kernel/iommu_groups/ -type l
root@nike:~#

I then wondered if it was bug or error in my motherboard and ran.

root@nike:~# dmesg | grep "Virtualization Technology for Directed I/O"
root@nike:~#

This returned nothing so then I ran

root@nike:~# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       46 bits physical, 48 bits virtual
CPU(s):              32
On-line CPU(s) list: 0-31
Thread(s) per core:  2
Core(s) per socket:  8
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               45
Model name:          Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
Stepping:            7
CPU MHz:             3018.247
CPU max MHz:         3000.0000
CPU min MHz:         1200.0000
BogoMIPS:            4399.99
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            20480K
NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

So now I know virtualization is active and it's not my hardware.

I then download libvirt-clients.

root@nike:~# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'freezer' controller support                     : PASS
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

It's like the kernel isn't getting the flags


I've already checked and read the following resources:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
https://pve.proxmox.com/wiki/Pci_passthrough#Intel_CPU
https://forum.proxmox.com/threads/iommu-broke-solved.53188/
https://heiko-sieger.info/iommu-groups-what-you-need-to-consider/
https://bbs.archlinux.org/viewtopic.php?id=259967
https://wiki.archlinux.org/index.php/Intel_GVT-g#Prerequisite
https://forum.proxmox.com/threads/pci-passthrough-no-iommu-detected.67504/
https://www.reddit.com/r/Proxmox/comments/d8e714/no_iommu_detected_pcie_passthrough/



Solution:

Turns out Proxmox doesn't install with GRUB by default on all installs. I've never actually seen my system boot because I control it via iDrac so I never noticed. 90% of the documentation is for GRUB too. I solved it by trying the systemd-boot config. (brain fart)

 

[Moayad]

Glad that you solved your problem yourself!

Thanks for sharing your solution have a nice day

 

[selfsufficient]

Thanks for sharing. I have a Lenovo X 3550 M5 that I cannot get IMMU to work. I see similar results as the above, however when I follow the grub or the systemd-boot process, I brick proxmox. Basically it cannot boot up after a reboot. Messages like this: ""pve" not found"
Anyone have any suggestions or maybe I just need to switch back to ESX for pci passthrough applications.

 

[hchasens]

Thanks for sharing. I have a Lenovo X 3550 M5 that I cannot get IMMU to work. I see similar results as the above, however when I follow the grub or the systemd-boot process, I brick proxmox. Basically it cannot boot up after a reboot. Messages like this: ""pve" not found"
Anyone have any suggestions or maybe I just need to switch back to ESX for pci passthrough applications.

ESXi is a more viable solution with its licensing changes (I still like Proxmox more though). Let's try to work this out. The first step I think would be to figured out which boot loader is being installed by default and stick with that. You mentioned:

`however when I follow the grub or the systemd-boot process, I brick proxmox.`

To clarify, are you reinstalling the default boot loader or just configuring for one or the other? If the former you should definitely stick with the default boot loader. Changing this can brick your system easily and in most server applications there really isn't a huge difference between the two excluding configuration differences. If the later - If you're configuring your system to boot from systemd-boot when it runs grub by default that could also cause issues or vice versa.

Can you post the output of bootctl status to see if systemd-boot is installed.

And also ls /boot or ls /efi (if the EFI directory exists).

This information should help us find which boot loader is being used.

 

[selfsufficient]

ESXi is a more viable solution with its licensing changes (I still like Proxmox more though). Let's try to work this out. The first step I think would be to figured out which boot loader is being installed by default and stick with that. You mentioned:



To clarify, are you reinstalling the default boot loader or just configuring for one or the other? If the former you should definitely stick with the default boot loader. Changing this can brick your system easily and in most server applications there really isn't a huge difference between the two excluding configuration differences. If the later - If you're configuring your system to boot from systemd-boot when it runs grub by default that could also cause issues or vice versa.

Can you post the output of bootctl status to see if systemd-boot is installed.

And also ls /boot or ls /efi (if the EFI directory exists).

This information should help us find which boot loader is being used.

Thanks for the fast response ! Ironically I swapped the hard drive 5 minutes ago, reinstalled the OS and made the changes for GRUB. Now it is working.
Appreciate the response. Sorry I kind of wasted some time over a hardware issue.

 

[hchasens]

Thanks for the fast response ! Ironically I swapped the hard drive 5 minutes ago, reinstalled the OS and made the changes for GRUB. Now it is working.
Appreciate the response. Sorry I kind of wasted some time over a hardware issue.

No problem! I'm glad you solved your issue!