[SOLVED] No configuration will enable IOMMU

[dellicious]

I have a dell r630 and I can't get proxmox to use iommu.
Proxmox keeps giving this error: "TASK ERROR: cannot prepare PCI pass-through, IOMMU not present"

I've followed the advice of multiple threads, yet this is the result of
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet

And this is dmesg:
# dmesg | grep -e DMAR -e IOMMU
[ 0.008870] ACPI: DMAR 0x000000007BAFE000 0000F8 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.008900] ACPI: Reserving DMAR table memory at [mem 0x7bafe000-0x7bafe0f7]
[ 0.775576] DMAR: Host address width 46
[ 0.775577] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 0.775583] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.775586] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[ 0.775590] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.775592] DMAR: RMRR base: 0x0000007ae07000 end: 0x0000007af06fff
[ 0.775594] DMAR: ATSR flags: 0x0
[ 0.775596] DMAR: ATSR flags: 0x0
[ 0.775598] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0
[ 0.775600] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1
[ 0.775602] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1
[ 0.775603] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[ 0.775604] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.775881] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode
[ 0.776148] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode
[ 0.776273] DMAR-IR: Enabled IRQ remapping in x2apic mode


Here's what I've done:

• I've enabled everything associated with virtualization in the BIOS
• I've updated grub:
  # cat /etc/default/grub
  [~snip~]
  GRUB_DEFAULT=0
  GRUB_TIMEOUT=5
  GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
  GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
  GRUB_CMDLINE_LINUX=""
  
  # update-grub
  Generating grub configuration file ...
  Found linux image: /boot/vmlinuz-6.2.16-15-pve
  Found initrd image: /boot/initrd.img-6.2.16-15-pve
  Found linux image: /boot/vmlinuz-6.2.16-3-pve
  Found initrd image: /boot/initrd.img-6.2.16-3-pve
  Found memtest86+x64 image: /boot/memtest86+x64.bin
  done
• It's updating entries in /boot/grub/grub.cfg:
  linux /boot/vmlinuz-6.2.16-15-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on
• Grub is running:
  # efibootmgr -v
  EFI variables are not supported on this system.
• I've added the kernel modules:
  # cat /etc/modules
  [~snip~]
  vfio
  vfio_iommu_type1
  vfio_pci
  vfio_virqfd
• They are running:
  # lsmod | grep vfio
  vfio_pci 16384 0
  vfio_pci_core 94208 1 vfio_pci
  irqbypass 16384 2 vfio_pci_core,kvm
  vfio_iommu_type1 49152 0
  vfio 57344 3 vfio_pci_core,vfio_iommu_type1,vfio_pci
  iommufd 73728 1 vfio
• I've changed /etc/kernel/cmdline:
  # cat /etc/kernel/cmdline
  intel_iommu=on
  
  # proxmox-boot-tool refresh
  Running hook script 'proxmox-auto-removal'..
  Running hook script 'zz-proxmox-boot'..
  Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
  No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
  
  # pve-efiboot-tool refresh
  Running hook script 'proxmox-auto-removal'..
  Running hook script 'zz-proxmox-boot'..
  Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
  No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
  
  I have no clue what "No /etc/kernel/proxmox-boot-uuids" means, but I don't think using /etc/kernel/cmdline applies when using grub anyways.

I'm not sure where to go from here, from my understanding adding intel_iommu=on to grub should've changed /proc/cmdline

 

[dellicious]

It seems that even though /boot/grub/grub.cfg is being updated, when loading into grub and editing the main boot entry there, "iommu=on" is missing.
Adding "iommu=on" through grub's editor works, but I'd have to manually do that after each reboot.

 

[leesteken]

It seems that even though /boot/grub/grub.cfg is being updated, when loading into grub and editing the main boot entry there, "iommu=on" is missing.
Adding "iommu=on" through grub's editor works, but I'd have to manually do that after each reboot.

Please check the manual: maybe your Proxmox uses systemd-boot instead?

 

[dellicious]

Turned out that the grub from an old install of proxmox was being booted into, but it was then somehow using the new install.

 

[jrivam]

Turned out that the grub from an old install of proxmox was being booted into, but it was then somehow using the new install.

hi, im having the same problem. Please can you elaborate what you did in here to diagnose this? Thank you.

 

[dellicious]

hi, im having the same problem. Please can you elaborate what you did in here to diagnose this? Thank you.

My issue was very specific, but the grub config I was editing wasn't the same grub config being used. I installed proxmox twice and forgot to switch the boot drive. I'd try going down the list of things I did in my main post for yourself, but what tipped me off to it was that by manually changing the grub config at boot time (by pressing 'e' on the proxmox entry) I could get intel_iommu=on to take effect

 

[ilium007]

So does IOMMU work with the R630?