IOMMU not present (vt-d broken?)

[ExactXmpl]

I have HP ProLiant se1220 board with xeon x5670 cpus. All virtualization options enabled in bios and ve configured by tutorial. Here some terminal outputs i got

root@homelab:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.11.22-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction

root@homelab:~# lsmod | grep -e vfio -e kvm
kvm_intel             282624  0
kvm                   823296  1 kvm_intel
vfio_pci               57344  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 vfio_pci,kvm
vfio_iommu_type1       36864  0
vfio                   36864  2 vfio_iommu_type1,vfio_pci
root@homelab:~#

root@homelab:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.009123] ACPI: DMAR 0x00000000C7622E80 000176 (v01 HP     ProLiant 00000001 \xd2?   0000162E)
[    0.009161] ACPI: Reserving DMAR table memory at [mem 0xc7622e80-0xc7622ff5]
[    0.151977] DMAR: IOMMU enabled
[    0.365319] DMAR-IR: This system BIOS has enabled interrupt remapping
[    2.198032] DMAR: Host address width 39
[    2.198034] DMAR: DRHD base: 0x000000cfffe000 flags: 0x1
[    2.198063] DMAR: dmar0: reg_base_addr cfffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[    2.198067] DMAR: RMRR base: 0x000000c77fb000 end: 0x000000c77fdfff
[    2.198072] DMAR: RMRR base: 0x000000c77f4000 end: 0x000000c77f9fff
[    2.198073] DMAR: RMRR base: 0x000000c763e000 end: 0x000000c763ffff
[    2.198074] DMAR: ATSR flags: 0x0
[    2.198095] DMAR: Device scope type does not match for 0000:00:14.0
[    2.198097] DMAR: Device scope type does not match for 0000:00:14.1
[    2.198099] DMAR: Device scope type does not match for 0000:00:14.2
[    3.712365] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    3.712368] AMD-Vi: AMD IOMMUv2 functionality not available on this system

root@homelab:~# find /sys/kernel/iommu_groups/ -type l
root@homelab:~#

I think there some problems with vt-d as i don`t have something like 'Enabled IRQ remapping in xapic mode' in dmesg.
Have no idea what to do about it so any help is appreciated.

 

[leesteken]

If there are not IOMMU groups, then IOMMU (VT-d) is not fully enabled and passthrough won't work (and pcie_acs_override cannot change that). Your VFIO configuration appears to be correct, so it is probably a BIOS setting (or problem).
Some HP systems are known to not enable IOMMU properly. Sometimes patching the kernel appears to help. I don't know how to check if this applies to your particular system sorry.

 

[ExactXmpl]

If there are not IOMMU groups, then IOMMU (VT-d) is not fully enabled and passthrough won't work (and pcie_acs_override cannot change that). Your VFIO configuration appears to be correct, so it is probably a BIOS setting (or problem).
Some HP systems are known to not enable IOMMU properly. Sometimes patching the kernel appears to help. I don't know how to check if this applies to your particular system sorry.

It appears to be some hp bios issue. Seems this patch is to deal with RMRR issue but seems i don`t exactly have any issues with it.

root@homelab:~# dmesg | grep RMRR
[    2.194008] DMAR: RMRR base: 0x000000c77fb000 end: 0x000000c77fdfff
[    2.194013] DMAR: RMRR base: 0x000000c77f4000 end: 0x000000c77f9fff
[    2.194015] DMAR: RMRR base: 0x000000c763e000 end: 0x000000c763ffff

Anyways thanks for reply. I will continue investigating it

 

[ExactXmpl]

Ok,something i found here

Output of that check command:

root@homelab:~# lspci -nn | grep -qE '8086:(340[36].*rev 13|3405.*rev (12|13|22))' && echo "Interrupt remapping is broken"
Interrupt remapping is broken

Looks i have one of that faulty chipsets so interrupt remapping doesn`t work (getting automatically disabled according to dmesg) and from here iommu refuses to work as well.
So i must try to enable it in any ways (probably not an good idea but still)

 

[DropZII]

Ok,something i found here

Output of that check command:

root@homelab:~# lspci -nn | grep -qE '8086:(340[36].*rev 13|3405.*rev (12|13|22))' && echo "Interrupt remapping is broken"
Interrupt remapping is broken

Looks i have one of that faulty chipsets so interrupt remapping doesn`t work (getting automatically disabled according to dmesg) and from here iommu refuses to work as well.
So i must try to enable it in any ways (probably not an good idea but still)

D'après la documentation proxmox, on peut peut-être contourner le problème :


If your system doesn't support interrupt remapping, you can allow unsafe interrupts with:

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

 

[ExactXmpl]

D'après la documentation proxmox, on peut peut-être contourner le problème :


If your system doesn't support interrupt remapping, you can allow unsafe interrupts with:

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

No luck because iommu itself doesn't start properly

 

[furmansk]

Did you set your boot options in /etc/default/grub or did you do it in /etc/kernel/cmdline? From the guide, it mentioned to use the latter since it boots systemd-boot instead of grub. Whats weird is, I had to set in the grub config file rootdelay=10 for my zfs partition to load correctly. Yet it shouldn't work cause I shouldn't be on grub since it boots with systemd-boot. I am lost lol. I have had to edit the file twice in a rescue disk and I'd rather not anymore and find out the answers.

Any luck on your end? I'm pretty sure my bios has the VT-d set.

 

[leesteken]

The manual describes how to determine which boot loader is used.

 

[ExactXmpl]

Did you set your boot options in /etc/default/grub or did you do it in /etc/kernel/cmdline? From the guide, it mentioned to use the latter since it boots systemd-boot instead of grub. Whats weird is, I had to set in the grub config file rootdelay=10 for my zfs partition to load correctly. Yet it shouldn't work cause I shouldn't be on grub since it boots with systemd-boot. I am lost lol. I have had to edit the file twice in a rescue disk and I'd rather not anymore and find out the answers.

Any luck on your end? I'm pretty sure my bios has the VT-d set.

No luck. Iommu just refuses to start for some reason. I use grub config for boot options and it works for sure. But all that i can see in dmesg that iommu enabled but not something like 'Virtualization technology for directed I/O' as it should be. Tried everything bios-related also.

 

[furmansk]

The manual describes how to determine which boot loader is used.

Yes I am aware and which is why I figured it's systemd-boot, based on the screenshot it shows that's the same menu I have. But as I said, I changed things in the grub config and it took the rootdelay=10 boot argument. So unless I'm not understanding how the boot works and it still uses grub even though its sytemd-boot then I'm at a loss. Any other recommendations?

No luck. Iommu just refuses to start for some reason. I use grub config for boot options and it works for sure. But all that i can see in dmesg that iommu enabled but not something like 'Virtualization technology for directed I/O' as it should be. Tried everything bios-related also.

Dang, yeah I tried couple times in my systemd-boot but both times it wouldn't boot. Would just sit there saying its scanning for drives and will take awhile. I left it the first time for over an hour and knew it wasn't going to boot. See I don't even see the "IOMMU is enabled" in my dmesg output. So something is wrong on the boot up command side. I'm going to just need to make a separate post I guess.