Cannot activate IOMMU under proxmox - works in Unraid

[WoRie]

Hi,

I'm currently in the process of creating a proxmox cluster.

The system was used with unraid before and as a first step, I want to virtualize Unraid (Which I managed to do correctly in the past) and progress from there.

To do so, I need to passthrough my HBA to the to be created VM. However, proxmox tells me that IOMMU is disabled. But I don't get why that is the case:

- The board is a very recent asrock W680D4U-2L2T/G5 with the most recent bios
- in BIOS I've got settings for SR-IOV as well as Intel Vt-d, both of which are enabled.
- CPU is a 14700k
- the bootloader is GRUB, since the OS is on ext4 not zfs. I plan on employing CEPH so I don't care much for local VMs on this host apart from the Unraid VM
- I appended "iommu_enabled=on" as well as "iommu=pt" all on a single line
- the install is from yesterday and the kernel is 6.5.11-7-pve

but nevertheless, the output for "dmesg | grep -e DMAR -e IOMMU" is completly empty.

Any ideas what I could have overlooked?

 

[cheiss]

Hi,

I appended "iommu_enabled=on" as well as "iommu=pt" all on a single line

Where did you append them exactly? Did you run update-grub afterwards?

Have you also checked whether the system was actually booted with these kernel arguments? E.g. whether they appear in the output of cat /proc/cmdline.

 

[WoRie]

Hi,


Where did you append them exactly? Did you run update-grub afterwards?

in /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"


and I ran update-grub afterwards

Have you also checked whether the system was actually booted with these kernel arguments? E.g. whether they appear in the output of cat /proc/cmdline.

root@pvenode1:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-7-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

 

[cheiss]

Hm, everything looks good there AFAICS.

Can you post the whole kernel log, i.e. the output of dmesg -H?
Please attach it as a file to your post since it can be quite big. You can redirect the log directly to a file using e.g. dmesg -H >kern.log.

 

[WoRie]

Here you go

 

[cheiss]

First of, the kernel log is unfortunately spammed with an error from the i40e module and only a portion remains. What's important here is the log from the boot itself.
I can see that this thread seems to be related to that: https://forum.proxmox.com/threads/error-i40e_aq_rc_enospc-forcing-overflow-promiscuous-on-pf.62875/

I'd first advise you to fix that, as outlined here. Probably not related to you IOMMU problem, but worth fixing anyway.

But the systemd journal should kept all entries. You can view that using journalctl -b, and again redirect into a file in the same manner.

 

[WoRie]

Attached you'll find the journalctl output. Thanks for your support btw

Concerning the log spam: I tried adding the offload-rx-vlan-filter off to each bridge in the network config, but this didn't solve the issue. I therefore deleted all entries in the log concerning this error, since the log was too large to upload it here

Also when reading through that thread, others reported as well, that they were unable to fix this with appending that flag.

 

[WoRie]

I just looked through the log myself and found, that IOMMU is now enabled. Seemingly the

dmesg |grep -e DMAR -e IOMMU -e AMD-Vi

returns empty due to the log spam.

I don't know what fixed it, but I'm happy it works. Now I have to somehow get rid of the log spam.