IOMMU Pass-through w/ UEFI boot config not working

[vetvetter]

I have the IOMMU pass-through issue with what I am seeing is the infamous HP Gen9's. Love the servers, hate the isssue HP has caused at hand here.


I tried to follow a few things I found in here and various places. I just cannot get IOMMU to detect on my HP DL360/380 G9's. Running latest Proxmox on all nodes 8.1.4

I'm getting “No IOMMU detected, please activate it. See Documentation for further information” trying to add the pass-through in a VM. The VM of course won't boot if I try it.

The steps I have seen have mostly been for a GRUB setup. However I am using UEFI to boot my machines. I tried to edit the steps and where I am updating things for my setup, but I think I am either missing something or doing it wrong passing the commands. The machines boot off USB NVME, another NVME(s) is installed along with other various Sata and SaS drives.

Found this thread and other various things talking about this issue:

https://forum.proxmox.com/threads/iommu-grub-config-not-working.145115/#post-654104

I am trying to pass through a NVME to Xpenology for NVME cache'ing. I also may pass off my entire HBA for all of the HDD's on one of them. if I can get this working. Everything else seems OK with them and I can build out everything without it OK. Just don't get the pass-through that I want to take advantage of here.


Things I have tried:


# nano /etc/kernel/cmdline

Added Lines to file:

intel_iommu=on
intell_iommu=relax_rmrr
iommu=pt
intremap=no_x2apic_optout

# proxmox-boot-tool refresh

Cmds run OK
reboot - same issue


# nano /etc/default/grub
Editing the GRUB_CMDLINE_LINUX_DEFAULT to include the option:
new line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr iommu=pt intremap=no_x2apic_optout"
update-grub

Cmds run OK
Rebooted - same issue.


Last ditch effort:
# echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

Rebooted - same thing.


Checks returned:

# pvesh get /nodes/localhost/hardware/pci --pci-class-blacklist ""
This returns a list of normal device mappings OK

# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
This returns:

root@DL380g9-LFF1:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.021615] ACPI: DMAR 0x000000007B7E7000 000300 (v01 HP ProLiant 00000001 HP 00000001)
[ 0.021698] ACPI: Reserving DMAR table memory at [mem 0x7b7e7000-0x7b7e72ff]
[ 2.630680] DMAR: Host address width 46
[ 2.630686] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 2.630711] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
[ 2.630723] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[ 2.630735] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
[ 2.630744] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff
[ 2.630752] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff
[ 2.630761] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff
[ 2.630768] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff
[ 2.630774] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff
[ 2.630781] DMAR: ATSR flags: 0x0
[ 2.630787] DMAR: ATSR flags: 0x0
[ 2.630794] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0
[ 2.630802] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1
[ 2.630809] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1
[ 2.630815] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[ 2.630822] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 2.632321] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@DL380g9-LFF1:~#

 

[leesteken]

# nano /etc/kernel/cmdline

Added Lines to file:

intel_iommu=on
intell_iommu=relax_rmrr
iommu=pt
intremap=no_x2apic_optout

All lines except the very first one are ignored. You need to put everything on the first line, according to the manual:

Systemd-boot
The kernel commandline needs to be placed as one line in /etc/kernel/cmdline.To apply your changes, run proxmox-boot-tool refresh, which sets it as the option line for all config files in loader/entries/proxmox-*.conf.

 

[vetvetter]

I ended up fixing my own issue. Thought posting it might help others running into the same issue:

Fix for issue:

Must have Proxmox 8+ (I believe the kernel is pre-patched on all ver 8 and up) / Or patched kernel per links in my first post above.

# nano /etc/kernel/cmdline


*** Important Note*** - Add following Line to first line of file. All must be on the same line already there with a space between each cmd:

<whats already in the file> intel_iommu=on intell_iommu=relax_rmrr iommu=pt intremap=no_x2apic_optout

Save file

# proxmox-boot-tool refresh

reboot host


Check if its good:

# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi



Maybe someone from the Proxmox team could add this to the wiki instructions on IOMMU pass-through as a note related to HP servers. Having this info layed out like this would have saved me a TON of time.

 

[vetvetter]

All lines except the very first one are ignored. You need to put everything on the first line, according to the manual:

Yep! I figured that out and was posting the fix I found right as you replied. Something so simple. Thanks for chiming in!