IOMMU broke [SOLVED]

[hlucid]

My homelab machine is as follows:

Motherboard: Supermicro X8DTI-F
CPU        : 2x Xeon E5645
RAM        : 64GB ECC

The last 2 years I have tried Hyper-V, Xen server and ESXi on which I settled for about a year. I was able to eventually enable and use pass-through on all of them.

For the last couple of weeks I started experimenting with Proxmox.

I installed Proxmox 5.3 and successfully enabled IOMMU.

I created two VMs, one FreeNAS and one Windows 7. The FreeNAS has a sata controller pass-through and worked as expected.

I was able to pass almost any device I cared for to any VM.

The system was rock stable for days up until the point that I created a new VM for pfSense.

My motherboard has 2 Gibabit NICs:

06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

the network interfaces are named enp6s0f0 and enp6s0f1 respectively.

I was using enp6s0f0 for connecting with my lan and for proxmox web administration.

I though to try and pass the other NIC (the one corresponding to enp6s0f1) to the newly created pfSense VM to act as it's WAN interface.

By starting the pfSense VM I lost connection to the hypervisor. So, I went and manually rebooted it.

I still had no connection to the server after the reboot. The network interfaces (both) were no longer showing with "ip link show" but were still showing with lspci.

I deleted the pfSense VM and rebooted. My connection to the server got restored and now "ip link show" showed them correctly.

From that point on. IOMMU seems to be disabled.

The VMs with pci pass-through devices fail to start with the error "IOMMU not present"

When editing any of the VMs pci pass-through hardware I now get "No IOMMU detected, please activate it. See Documentation for further information.".

In the same dialog the "Device" drop down menu shows all IOMMU groups set to "-1" while the
IDs do still contain the letters (a,d,f) that indicate IOMMU capability (if I'm not mistaken).

Even though:

1) grub boot option is still set to iommu=on:

GRUB_CMD_LINE_LINUX_DEFAULT="quiet intel_iommu=on"

2) /etc/modules does still contain the modules needed:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

3) the modules still do load:

lsmod | grep vfio

vfio_pci               45056  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 vfio_pci,kvm
vfio_iommu_type1       24576  0
vfio                   28672  2 vfio_iommu_type1,vfio_pci

4) relating kernel messages are as follows:

dmesg | grep -E 'IOMMU|DMAR'

[    0.000000] ACPI: DMAR 0x00000000BF75E0D0 000128 (v01 AMI    OEMDMAR  00000001 MSFT 00000097)
[    0.000000] DMAR: Host address width 40
[    0.000000] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[    0.000000] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[    0.000000] DMAR: RMRR base: 0x000000000e6000 end: 0x000000000e9fff
[    0.000000] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
[    0.000000] DMAR: ATSR flags: 0x0
[    0.000000] DMAR-IR: IOAPIC id 6 under DRHD base  0xfbffe000 IOMMU 0
[    0.000000] DMAR-IR: IOAPIC id 7 under DRHD base  0xfbffe000 IOMMU 0
[    0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode

Any ideas? Thanks in advance.

 

[lhorace]

Hello there,

The problem might have to do with your IOMMU grouping, normally you can't simply pass one device in the same IOMMU group, read more here https://heiko-sieger.info/iommu-groups-what-you-need-to-consider/.

Cheers.

 

[hlucid]

Hi,
Thanks for taking the time. Well, yes I already knew they were in the same group but I expected the system to demand from me that I passed both through or none at all. Instead of that, everything blew up in my face. The thing is that I 'm no longer trying to pass through that device. I deleted that whole VM. The NICs are showing and are available to proxmox. The system seems to be just like before I created the VM but IOMMU is still unusable by all other VMs that had other devices passing through. I'll try to read thoroughly the link you suggested.

 

[lhorace]

Well, yes I already knew they were in the same group but I expected the system to demand from me that I passed both through or none at all.

In my experience, I've never seen that, though I could be wrong. I'm not aware of the WebUI/Backend have such protections or who would be responsible to appropriately prompt the end user. The expectation would be having a good grasp of the technology before hand. IOMMU, in my experience is diverse and experiences are different between motherboards and peripherals.

IOMMU is still unusable

The system was rock stable for days up until the point that I created a new VM for pfSense.

Confused.

 

[hlucid]

I used pass-through for a sata controller to my FreeNAS VM. That worked rock stable for days.

Some days later I tried to pass-through a NIC to a new VM. That wrecked the havoc I mentioned above (lost connection etc).

After deleting that new VM (the one with the NIC) I can no longer use IOMMU for my sata (or any other device) on my FreeNAS VM (or any other VM for that).

The system appears as if IOMMU is enabled (configuration wise - as far I can tell). But, I'm getting error messages as if it's not.

 

[hlucid]

dmesg | grep "Virtualization Technology for Directed I/O"

indeed yields nothing when it should yield something like:

[ 0.902214] DMAR: Intel(R) Virtualization Technology for Directed I/O

The article you pointed to me says for that:
(well, actually another article that is being referenced by the one in the link you gave)

"If you do not get this output, then VT-d or AMD-V is not working – you need to fix that before you continue! Most likely it means that your hardware (CPU) doesn’t support IOMMU, in which case there is no point continuing this tutorial . Check again to make sure your CPU supports IOMMU. If yes, the cause may be a faulty motherboard BIOS. See if you find a newer version and update your motherboard BIOS."

But I know it's not a hardware/BIOS failure. I can switch HDDs, boot my ESXi without issues and use passthrough.

 

[lhorace]

But I know it's not a hardware/BIOS failure. I can switch HDDs, boot my ESXi without issues and use passthrough.

I think you are amateur and you are blaming Proxmox?
I've used Proxmox for 3 years, and I figure it out.

 

[BobhWasatch]

What does "cat /proc/cmdline" say? Maybe grub isn't booting the configuration you think it is. The grub boot option you posted is only evaluated when the grub configuration is updated.

 

[dcsapak]

What does "cat /proc/cmdline" say?

yes this would be interesting, also the output of 'dmesg' (in full) would be interesting

 

[hlucid]

I think you are amateur and you are blaming Proxmox?
I've used Proxmox for 3 years, and I figure it out.

I’m sorry you felt so offended. I think you completely misunderstood my intentions. No one is blaming anyone for anything here. having an expectation is not the same as having an opinion and having an opinion doesn't always equate to a value judgment. I was just sharing my experience in order to further narrow down the search in the correct direction and eventually make sense of the situation. I don’t see how your comments are constructive to that end.

I’ am actually impressed with Proxmox and is by far the easiest to install and configure. It has the most features and no limitations. The web administration is exceptional. The whole system feels more robust and easy. The debian heritage is also welcome. I had the least trouble setting up ups safe shutdown with nut. The list goes on and on. So, maybe you jumped the gun on this one.

 

[hlucid]

What does "cat /proc/cmdline" say? Maybe grub isn't booting the configuration you think it is. The grub boot option you posted is only evaluated when the grub configuration is updated.

yes this would be interesting, also the output of 'dmesg' (in full) would be interesting

Thank you guys so much. You nailed it. It wasn't being updated by update-grub. Maybe a rookie mistake but, I never had update-grub fail to update before. I tried to update with grub-mkconfig and to reinstall grub via grub-install. None of them worked. What worked was what was suggested in this link https://serverfault.com/questions/648468/modifying-kernel-cmdline-in-ubuntu .

create the directory /etc/default/grub.d

create a file ending with “.cfg” in that directory.

add the following to it:

GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} intel_iommu=on"

run update-grub.

Rebooted and everything works. Thank you!