[SOLVED] Problem with HP DL380P G8 after enabling IOMMU

[Kanze]

Hello all,

I'm having a really strange behavior on my HP server after enabling IOMMU on grub file and the related modules.
I have installed the HP H220 HBA card and connected the 4 SAS disks on the second bay, after that on Proxmox GUI I can see these 4 HDDs. So I enabled IOMMU modifying the grub file and enabled the modules, so I updated the grub config and then rebooted the server. After that I can't log anymore on the server using ssh external tools or log to the GUI, since it responds always connection refused, I can only login using the embedded HTML5 console from ILO management.
Is there anything I can do to fix this problem?

Thanks!

 

[Moayad]

Weird, can you provide us with the `GRUB_CMDLINE_LINUX_DEFAULT` line you edited? In the /etc/default/grub. did you see anything interesting in the output of update-grub?

Regarding the network, can you maybe check if the network interface names got changed by issuing the `ip a` and compare the result in the `/etc/network/interfaces`

 

[Kanze]

Hello @Moayad,

attached you can find the output of the grub file and the output of the network commands you asked. For the last two, no changes for the name of the interface.
For output of

update-grub

nothing stange, unfortunatelly I haven't the logs since on ILO I can't save them.

 

[Moayad]

Thank you for the screenshots!

Regarding the network interface, can you see if the eno1 is printed on the output of ip a command? Sometimes the NIC got change when enable the IOMMU (This had happened in my case).


Regarding the GRUB command: may I ask you what/why you are using the "interemap=off"? I would use the following Kernel command with their parameter, as mentioned in our wiki guide [0].

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

[0] https://pve.proxmox.com/wiki/PCI(e)_Passthrough

 

[Kanze]

No problem for the screen!
Yes, eno1 is printed as you can see from the screen attached and I'm using intremap=off since I had a really laggy CLI and many times the HTML5 ILO not accepting ILO virtual keyboard inputs anymore, using this command has improved this aspect since this is the only method that I have to connect to the server for now.
Ok, I try to insert also that command on grub file.

 

[Kanze]

@Moayad really strange, I don't know if is related but I have added the kernel command iommu=pt and now I'm able to log with external SSH tools and in the GUI too.

 

[Kanze]

Ok, seems working now, I mark the post as SOLVED

 

[Moayad]

Hi,

Glad to read that you fix the issue yourself!

Maybe the `intel_iommu=on` parameter alone caused the issue, and by adding the `iommu=pt` fixed that issue? In general, hard to say why/what happened without more Syslog.

 

[leesteken]

Maybe the `intel_iommu=on` parameter alone caused the issue, and by adding the `iommu=pt` fixed that issue?

Could very well be right. iommu=pt makes sure that the IOMMU uses no mapping (or identity mapping) for devices that are not passed through. If you have devices that cannot handle IOMMU (enabled by intel_iommu=on) properly, then the no/identity mapping can be a work-around.
IMHO, the Proxmox manual and Wiki are not fully correct about what iommu=pt does and other guides on the internet also don't really explain what it does.

 

[Darkhand]

Regarding the GRUB command: may I ask you what/why you are using the "interemap=off"? I would use the following Kernel command with their parameter, as mentioned in our wiki guide [0].

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

[0] https://pve.proxmox.com/wiki/PCI(e)_Passthrough

This thread is older, but I can add some clarity here as I'm working through the same issue.

OP mentioned that he's on an HP DL380P G8, which uses the Intel 5520 chipset. That chipset has a documented problem with interrupt remapping:

https://support.citrix.com/article/CTX136517/intel-55x0-chipset-errata-interrupt-remapping-issue

For Proxmox and pretty much any linux install I've tried, adding intremap=off as a boot parameter is required to prevent DMAR errors in the logs/main console, and to allow use of the iLo remote management console. Using the regular quiet intel_iommu=on iommu=pt is usually a no-go on this generation of HP server, or anything that uses the same chipset.

Unfortunately this seems to break device mapping in general, especially mapped passthrough in Proxmox.

There is a way to enable specific PCI slots for passthrough but it requires configuration with HP-specific utilities:

https://web.archive.org/web/2022081...pesc/public/docDisplay?docId=emr_na-c04781229
https://www.jimmdenton.com/proliant-intel-dpdk/