IOMMU unsafe interrupts enabled, still error message

[Feni]

Hi there,

I've used Proxmox for years on an old HP DL380 G6 which needs a patch to disable IOMMU RMRR checking and allow unsafe interrupts in order to passthrough GPU's to VM's. This combination has worked all that time with some rewrites of my patch whenever another kernel is used in Proxmox (https://forum.proxmox.com/threads/c...ntel-iommu-driver-to-remove-rmrr-check.36374/).

After updating to Proxmox 6.1.8 (Focal) however, I get the following output in dmesg:

[vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

A quick check shows that my modprobe.d conf files are still there:

root@alpha:/etc/modprobe.d# ls
blacklist.conf  iommu_unsafe_interrupts.conf  kvm.conf  pve-blacklist.conf  vfio.conf  zfs.conf

And unsafe interrupts should be enabled:

options vfio_iommu_type1 allow_unsafe_interrupts=1

Any guru here that can show me what I'm missing? Or tell me if the latest kernel has a new way (or no way) of dealing with unsafe iommu interrupts? I'm reading something about lockdown in v5.4, could that affect me?

 

[t.lamprecht]

After updating to Proxmox 6.1.8 (Focal) however, I get the following output in dmesg:

There's no version named focal, and strictly seen the most current pve-manager version (at time of writing) in Proxmox VE is 6.1-8
I guess that with "focal" you mean that you installed the experimental pve-kernel-5.4 series? Or does this happen with the default 5.3 one: pveversion -v

And unsafe interrupts should be enabled:

You can check the current active parameter value as root by looking in sysfs:
cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

I'm reading something about lockdown in v5.4, could that affect me?

No, that specific thing should not affect you - we do not enable kernel lock down.

It rather seems that the "vfio_iommu_type1" is not a module by default in 5.4 anymore, i.e., compare:

ls -l /lib/modules/5.3.18-3-pve/kernel/drivers/vfio/
# with
ls -l /lib/modules/5.4.24-1-pve/kernel/drivers/vfio/

It switched from "build as module" to "build always in kernel", so modprobe doesn't loads that module (and cannot apply params) as it is already loaded due to being compiled directly into the kernel.

You should be able to set the option via the kernel commandline ( https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline )

Adding something like vfio_iommu_type1.allow_unsafe_interrupts=1 should do the trick.

 

[Feni]

There's no version named focal, and strictly seen the most current pve-manager version (at time of writing) in Proxmox VE is 6.1-8
I guess that with "focal" you mean that you installed the experimental pve-kernel-5.4 series? Or does this happen with the default 5.3 one: pveversion -v



You can check the current active parameter value as root by looking in sysfs:
cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts



No, that specific thing should not affect you - we do not enable kernel lock down.

It rather seems that the "vfio_iommu_type1" is not a module by default in 5.4 anymore, i.e., compare:

ls -l /lib/modules/5.3.18-3-pve/kernel/drivers/vfio/
# with
ls -l /lib/modules/5.4.24-1-pve/kernel/drivers/vfio/

It switched from "build as module" to "build always in kernel", so modprobe doesn't loads that module (and cannot apply params) as it is already loaded due to being compiled directly into the kernel.

You should be able to set the option via the kernel commandline ( https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline )

Adding something like vfio_iommu_type1.allow_unsafe_interrupts=1 should do the trick.

Thanks for the great support Thomas, editing the kernel commandline worked indeed. My Linux fu is not that strong, my major was business economics.

5.4 (Focal Fossa) was installed after cloning the master pve-kernel.git and compiling a patch to remove RMRR check (see pve-kernel appendix *-removermrr below). This change will make many PCIe passthrough tutorials obsolete, including these two:
https://pve.proxmox.com/wiki/Pci_passthrough
https://pve.proxmox.com/wiki/PCI(e)_Passthrough

pveversion -v

proxmox-ve: 6.1-2 (running kernel: 5.4.24-1-pve-removermrr)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-5
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.24-1-pve-removermrr: 5.4.24-1
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.13-1-pve-removermrr: 5.3.13-1
pve-kernel-5.3.10-1-pve-removermrr: 5.3.10-1
pve-kernel-5.3.10-1-pve-removermrr-v2: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-1-pve-removermrr: 5.0.21-2
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-22
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[t.lamprecht]

I think your fu is not bad, glad you could get it going againg

 

[davidthejr]

You should be able to set the option via the kernel commandline ( https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline )

Adding something like vfio_iommu_type1.allow_unsafe_interrupts=1 should do the trick.

Hello! I have just ran into an issue with iommu not working after running updates on my proxmox box. where did you add this line to allow unsafe interrupts again? is it on the same line as the "intel_iommu=on" option? on its own line?

Thanks!

 

[t.lamprecht]

Hi,

It should be added all in one line to the kernel command line, see the linked documentation article at the bottom:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline

But actually we moved the vfio-pci from builtin to module again, together with the xhci and similar modules (as they were they reason vfio-pci got switched builtin in the first place, as they could already bind some devices this way before vfio-pci could.)
https://pve.proxmox.com/pipermail/pve-devel/2020-May/043538.html

So with the next kernel release this change should not be required anymore, but if you change it now it will still stay valid with the (new) old way too - hope I didn't made to point to complicated