No IOMMU detected, please activate it

[dudenell]

Running a intel S2600CW, with 2x E5-2690 v4 and 2x RMS25KB080 raid controllers, I have 11x HDDs and 1x SSD on one of the two RMS25KB080.

Promox is installed the SSD for the raid controller.

I'm trying to pass the raid controllers to a VM for management of the other 11 HDDs.

The system came with many default values off (IE UEFI, VT-X).

Now I'm trying to create a VM, and pass through the raw HDDs to the VM, I'm getting the above error. I can select the PCI controllers in the hardware section of the VM (but that error shows).

VT-d was turned off before the proxmox install, I turned it on but I'm not seeing it when I run the command:

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet

I'm not sure where to go from here to get it running.

 

[leesteken]

Please follow the steps from the Proxmox manual:

IOMMU
First, you have to enable IOMMU support in your BIOS/UEFI. Usually the corresponding setting is called IOMMU or VT-d,but you should find the exact option name in the manual of your motherboard.
For Intel CPUs, you also need to enable the IOMMU on the kernel command line by adding:
intel_iommu=on
For AMD CPUs it should be enabled automatically.

What is the output of cat /proc/cmdline? Is VT-d enabled in the motherboard BIOS? Did you add intel_iommu=on in the right place, depending on the boot loader?

 

[Ventority]

Maybe a dumb question but is iommu enabled in the bios?

 

[dudenell]

Please follow the steps from the Proxmox manual:

What is the output of cat /proc/cmdline? Is VT-d enabled in the motherboard BIOS? Did you add intel_iommu=on in the right place, depending on the boot loader?

I think that fixed the issue, it appears to be "working", but my next assumption is that I can't use a disk on the same raid controller as what I'm trying to pass through to the VM? As soon as I start the VM proxmox crashes.

 

[leesteken]

I think that fixed the issue, it appears to be "working", but my next assumption is that I can't use a disk on the same raid controller as what I'm trying to pass through to the VM? As soon as I start the VM proxmox crashes.

You can only PCI(e) passthrough whole PCI(e) devices. Also, you cannot share devices (and everything connected to it, like drives) from the same iOMMU group with other VMs or with the Proxmox host. You can use this command to get a nice overview of the IOMMU groups: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done.
You can passthrough physical disks, but it's different from PCI(e) passthrough.

 

[Zetto]

Hey dudenell,

did you ever resolve this? I have the same CPU and a similar issue. Even though vt-x and VTd is enabled in the BIOS and I turned on IOMMU in the proxmox grub file I only have one IOMMU group. And proxmox gives me an error when I try to add a PCI device. Been tinkering around with this for a few hours now and kind of run out of ideas.

 

[dudenell]

Hey dudenell,

did you ever resolve this? I have the same CPU and a similar issue. Even though vt-x and VTd is enabled in the BIOS and I turned on IOMMU in the proxmox grub file I only have one IOMMU group. And proxmox gives me an error when I try to add a PCI device. Been tinkering around with this for a few hours now and kind of run out of ideas.

So sorry if this isn't much help, I went to my grub file and added

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

then ran update-grub and it seemed to have worked, I'm unsure what other settings I played with as it's been a while. I apologize for not posting the answer when I figured it out.

 

[leesteken]

did you ever resolve this? I have the same CPU and a similar issue. Even though vt-x and VTd is enabled in the BIOS and I turned on IOMMU in the proxmox grub file I only have one IOMMU group. And proxmox gives me an error when I try to add a PCI device. Been tinkering around with this for a few hours now and kind of run out of ideas.

Does your Proxmox use GRUB or systemd-boot? What is the output of cat /proc/cmdline?

 

[Zetto]

Does your Proxmox use GRUB or systemd-boot? What is the output of cat /proc/cmdline?

Uses systemd-boot now. This is the output.

initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset initcall_blacklist=sysfb_init

I added everything from quit onwards yesterday and after I restarted I had everything split up into IOMMU groups....perfect. Today however everything is back to a single group. I haven't changed anything on the line above in between. All I have done is add more PCIe devices.

 

[leesteken]

initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset initcall_blacklist=sysfb_init
I added everything from quit onwards yesterday and after I restarted I had everything split up into IOMMU groups....perfect. Today however everything is back to a single group. I haven't changed anything on the line above in between. All I have done is add more PCIe devices.

It is active on the kernel command line, so it must be the VT-d setting in the motherboard BIOS. Maybe it went back to default settings due to the hardware changes? Or maybe the motherboard battery is low and it forgets when the power is turned off?

 

[Zetto]

It is active on the kernel command line, so it must be the VT-d setting in the motherboard BIOS. Maybe it went back to default settings due to the hardware changes? Or maybe the motherboard battery is low and it forgets when the power is turned off?

Just hooked up a Monitor to confirm the BIOS and VT-d is still enabled. That's not the reason, unfortunately.

 

[leesteken]

Just hooked up a Monitor to confirm the BIOS and VT-d is still enabled. That's not the reason, unfortunately.

And intel_iommu=on is still present in cat /proc/cmdline? Then it's still a motherboard/BIOS issue with the added hardware or something like that.

 

[Zetto]

yep,

intel_iommu=on

is still there. Started ripping out the PCIe devices one by one and found the corporate. A Dual 1Gbyte Ethernet card was causing it. With the card removed I have IOMMU groups again.