Pci passthrough problems

[Discolai]

HI, I am having trouble setting up IOMMU for pci passthrough on my system. The server is a HP ProLiant DL380 Gen9 running proxmox 6.0-4. I have enabled all the virtualization options within the bios including vt-d.

I followed the instructions on the wiki to set up iommu support, but proxmox still reports that it cannot detect IOMMU when I try to add a new pci device to a vm. In addition all the devices in the dropdown show up belonging to the same IOMMU group "-1".

The output from "dmesg | grep -e DMAR -e IOMMU" is:

[ 0.027294] ACPI: DMAR 0x000000007B7E7000 00032A (v01 HP ProLiant 00000001 HP 00000001) [ 3.746102] DMAR: Host address width 46 [ 3.746106] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 3.746118] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746123] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 3.746132] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746137] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff [ 3.746141] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff [ 3.746145] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff [ 3.746149] DMAR: RMRR base: 0x000000634a1000 end: 0x000000674a0fff [ 3.746153] DMAR: RMRR base: 0x000000674a1000 end: 0x0000006b4a0fff [ 3.746156] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff [ 3.746160] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff [ 3.746166] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0 [ 3.746171] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746174] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746178] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 3.746182] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 3.747344] DMAR-IR: Enabled IRQ remapping in x2apic mode

 

[lhorace]

According to https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM and looking at my kernel config on my HP DL360e Gen8. CONFIG_DMAR_DEFAULT_ON doesn't appear to be set, you might need to set the below on the kernel command line for iommu to function:

intel_iommu=on

Additionally, you might want to give this code a try:

if compgen -G "/sys/kernel/iommu_groups/*/devices/*" > /dev/null; then
    echo "AMD's IOMMU / Intel's VT-D is enabled in the BIOS/UEFI."
else
    echo "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI"
fi

 

[User5665]

HI, I am having trouble setting up IOMMU for pci passthrough on my system. The server is a HP ProLiant DL380 Gen9 running proxmox 6.0-4. I have enabled all the virtualization options within the bios including vt-d.

I followed the instructions on the wiki to set up iommu support, but proxmox still reports that it cannot detect IOMMU when I try to add a new pci device to a vm. In addition all the devices in the dropdown show up belonging to the same IOMMU group "-1".

The output from "dmesg | grep -e DMAR -e IOMMU" is:

[ 0.027294] ACPI: DMAR 0x000000007B7E7000 00032A (v01 HP ProLiant 00000001 HP 00000001) [ 3.746102] DMAR: Host address width 46 [ 3.746106] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 3.746118] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746123] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 3.746132] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746137] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff [ 3.746141] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff [ 3.746145] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff [ 3.746149] DMAR: RMRR base: 0x000000634a1000 end: 0x000000674a0fff [ 3.746153] DMAR: RMRR base: 0x000000674a1000 end: 0x0000006b4a0fff [ 3.746156] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff [ 3.746160] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff [ 3.746166] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0 [ 3.746171] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746174] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746178] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 3.746182] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 3.747344] DMAR-IR: Enabled IRQ remapping in x2apic mode

Hi Discolai,

I am having the exact same issues with my ProLiant DL380 gen9 when trying to enable IOMMU in Proxmox v7.1. I followed the official guide on how to enable PIC Passtrough but the output of the command "dmesg | grep -e DMAR -e IOMMU" is the same as yours, my output is:

root@proliant:~# dmesg | grep -e DMAR -e IOMMU
[    0.023801] ACPI: DMAR 0x000000007B7E7000 000294 (v01 HP     ProLiant 00000001 HP   00000001)
[    0.023902] ACPI: Reserving DMAR table memory at [mem 0x7b7e7000-0x7b7e7293]
[    1.404101] DMAR: Host address width 46
[    1.404105] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.404119] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    1.404128] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.404137] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    1.404144] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff
[    1.404151] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff
[    1.404157] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff
[    1.404163] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff
[    1.404170] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff
[    1.404178] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.404185] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.404190] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.404195] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.404200] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.405227] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@proliant:~#

I do not understand the output and what it might hint at.
I checked the settings in the BIOS of the Server. "vt-d", which was the only setting I was able to find, is enabled (see attached screenshot of the BIOS Settings).

If I insert the command provided by Ihorace:

if compgen -G "/sys/kernel/iommu_groups/*/devices/*" > /dev/null; then
    echo "AMD's IOMMU / Intel's VT-D is enabled in the BIOS/UEFI."
else
    echo "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI"
fi

the output is:

AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI

although I did enable it (see attached screenshot of the BIOS Settings)!

The command:

find /sys/kernel/iommu_groups/ -type l

shows no output at all.

I am out of ideas... what was your solution?
I checked an the CPUs E5-2650 v3 from Intel do support IOMMU. I was not able to check the Motherboard because I did not know how (could not find any Model Number) but it does not make any sense to me that this server grade motherboard would not support it.

I checked which Boot Mode the Server is using and it is UEFI according to the BIOS information. I edited
- /etc/default/grub
and
- /etc/kernel/cmdline

and used the commands:
-update-grub
and
-proxmox-boot-tool refresh

accordingly, because it did not work at first when only updating grub. I configured the modules in /etc/modules as well and ran "update-initramfs" to update kernel modules.

I rebooted after each change with "reboot" in the Proxmox CLI.v
Thanks in Advance to anybody for any help!

 

[User5665]

According to https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM and looking at my kernel config on my HP DL360e Gen8. CONFIG_DMAR_DEFAULT_ON doesn't appear to be set, you might need to set the below on the kernel command line for iommu to function:

intel_iommu=on

Additionally, you might want to give this code a try:

if compgen -G "/sys/kernel/iommu_groups/*/devices/*" > /dev/null; then
    echo "AMD's IOMMU / Intel's VT-D is enabled in the BIOS/UEFI."
else
    echo "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI"
fi

Hi Ihorace, I just commented on the initial question of Discolai.
I tried you code which gave the output "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI" even though I configured Intel's VT-D in the BIOS. Do you have any ideas?
Your help is much appreciated!

 

[D0doooh]

did any of you get this to work?
@User5665 @lhorace or @Discolai ?
stand also with the same problem here...

 

[User5665]

did any of you get this to work?
@User5665 @lhorace or @Discolai ?
stand also with the same problem here...

Hi @D0doooh,

in my case, the PCI Card (SAS HBA Controller Card) was bad. I bought a new one and as far as I can remember (it was a few months ago), followed the official documentation.

 

[D0doooh]

Hi @D0doooh,

in my case, the PCI Card (SAS HBA Controller Card) was bad. I bought a new one and as far as I can remember (it was a few months ago), followed the official documentation.

Hi, thank you for your replay!

that means you haven't done anything special? and you are on the latest proxmox version 7.3?

No other kernel or anything? I just can't activate IOMMU completely. I have already tried various things but I just can't get any further.

 

[leesteken]

I just can't activate IOMMU completely. I have already tried various things but I just can't get any further.

What is the make and model of your motherboard? What is the output of cat /proc/cmdline. What is the output of for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done (without using pcie_acs_overide)?

 

[D0doooh]

What is the make and model of your motherboard? What is the output of cat /proc/cmdline. What is the output of for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done (without using pcie_acs_overide)?

Hi @leesteken

Here my infos:
Base Board Information
Manufacturer: HP
Product Name: ProLiant DL380 Gen9
Version: Not Specified

do you need any more info here?

currently i'm on a custum kernel, and will otherwise completely reinstall it again, to the current proxmox version, because currently everything is a bit messed up, because i've turned every knob to get it running.

cat /proc/cmdline

initrd=\EFI\proxmox\5.19.7-2-pve-relaxablermrr\initrd.img-5.19.7-2-pve-relaxablermrr root=ZFS=rpool/ROOT/pve-1 boot=zfs

I had to upload here because too long:
https://pastebin.com/raw/GktafBUe

Thanks a lot!

 

[leesteken]

ince you have an HP you might have RMRR issues that might be fixed by a newer BIOS, but that's another thing to solve after enabling IOMMU in Proxmox.

cat /proc/cmdline
initrd=\EFI\proxmox\5.19.7-2-pve-relaxablermrr\initrd.img-5.19.7-2-pve-relaxablermrr root=ZFS=rpool/ROOT/pve-1 boot=zfs

You have an Intel system and there is no intel_iommu=on in there. Therefore IOMMU is most likely off (in Proxmox).

I had to upload here because too long:
https://pastebin.com/raw/GktafBUe

There appears to be missing a small part at the start (something like IOMMU group *) but it seems to confirm that IOMMU is off (in Proxmox).

Your system probably uses systemd-boot instead of GRUB but make sure to check as shown in the manual. Check cat /etc/kernel/cmdline for intel_iommu=on. If it's not there, add it (on the same line and don't add any other lines) (and run proxmox-boot-tool refresh) and reboot.

 

[D0doooh]

okay, is that a thing I should rather use GRUB? I mean I have not set anything else only a standard installation?

however, i have now adjusted it in system d and this is my output after the reboot:

initrd=\EFI\proxmox\5.19.7-2-pve-relaxablermrr\initrd.img-5.19.7-2-pve-relaxablermrr root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on relax_rmrr iommu=pt

There appears to be missing a small part at the start (something like IOMMU group *) but it seems to confirm that IOMMU is off (in Proxmox).

after the adjustments it displays it.
https://pastebin.com/raw/sDDqQ8RS


otherwise I do now really again
- a reinstallation, with Proxmox 7.3
- set GRUB as bootloader with the correct parameters,
- and update the BIOS.

then everything should work properly, right?

 

[User5665]

Hi, thank you for your replay!

that means you haven't done anything special? and you are on the latest proxmox version 7.3?

No other kernel or anything? I just can't activate IOMMU completely. I have already tried various things but I just can't get any further.

Hi @D0doooh,

because I configured it a long time ago, I really can not tell anymore if I did something special. I remember following the official Proxmox Docs and using this thread here: https://forum.proxmox.com/threads/i...fter-following-guide.93756/page-2#post-466710 -> Mr/Mrs. leesteken helped me

Hope this thread might give some further information. I am sorry I can not tell you anything else.

 

[leesteken]

okay, is that a thing I should rather use GRUB? I mean I have not set anything else only a standard installation?

systemd-boot is normal for Proxmox installations that boot in UEFI mode from a ZFS pool. It's not your choice, it's the choice of the Proxmox developers. You can read about it in the Proxmox manual.

however, i have now adjusted it in system d and this is my output after the reboot:

initrd=\EFI\proxmox\5.19.7-2-pve-relaxablermrr\initrd.img-5.19.7-2-pve-relaxablermrr root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on relax_rmrr iommu=pt

after the adjustments it displays it.
https://pastebin.com/raw/sDDqQ8RS

IOMMU is enabled because, as you can see now, you have multiple IOMMU groups. I assume that solves your problem?

otherwise I do now really again
- a reinstallation, with Proxmox 7.3
- set GRUB as bootloader with the correct parameters,
- and update the BIOS.

then everything should work properly, right?

I don't know, you might still have the RMRR problem that some HP systems have. If you really want GRUB, you need to reinstall in Legacy mode instead of UEFI but Proxmox might move to systemd-boot for all systems in the future anyway, so why bother. You might want to try the normal PVE kernel first and experiment with your now working IOMMU first, because starting all over again.

 

[D0doooh]

systemd-boot is normal for Proxmox installations that boot in UEFI mode from a ZFS pool. It's not your choice, it's the choice of the Proxmox developers. You can read about it in the Proxmox manual.

IOMMU is enabled because, as you can see now, you have multiple IOMMU groups. I assume that solves your problem?

I don't know, you might still have the RMRR problem that some HP systems have. If you really want GRUB, you need to reinstall in Legacy mode instead of UEFI but Proxmox might move to systemd-boot for all systems in the future anyway, so why bother. You might want to try the normal PVE kernel first and experiment with your now working IOMMU first, because starting all over again.

okay that makes sense.

I think a BIOS update certainly does not hurt, I will now make that.

Would like to update to the official kernel, as well as the current version. I have manipulated so much on the current system I would like to reinstall anyway. not that some dependency later ruins anything...

I thank you very much again I am now quite well optimistic and hope this will work

 

[D0doooh]

Hi @D0doooh,

because I configured it a long time ago, I really can not tell anymore if I did something special. I remember following the official Proxmox Docs and using this thread here: https://forum.proxmox.com/threads/i...fter-following-guide.93756/page-2#post-466710 -> Mr/Mrs. leesteken helped me

Hope this thread might give some further information. I am sorry I can not tell you anything else.

thank you for your answer, yes I think that now comes to work, thanks to this threat and the help of @leesteken

 

[D0doooh]

I don't know, you might still have the RMRR problem that some HP systems have.

I think everything is working, I can assign the network card to the VM and have no problems.

I can't finalize it until I'm on site, because I just set it up remotely, and unfortunately I don't have a robot arm to plug in the cables... writes todo list - build robot arm....

My last question would be how would this RMRR problem occur or how can I check this?

 

[leesteken]

My last question would be how would this RMRR problem occur or how can I check this?

VMs with passthrough will fail to start and the error message (somewhere) will contain RMRR and tell you to contact your vendor (or something like that).

 

[fbeye]

Hate to bring up new inquiry on older topic, but I am seeming to have same issue.
It is an HPE Proliant DL380 Gen 9; I verified, even now, Every Virtual option enabled.
No matter what I do, IOMMU will not enabled;

dmesg | grep -e DMAR -e IOMMU
[ 0.021214] ACPI: DMAR 0x000000007B7E7000 0002CE (v01 HP ProLiant 00000001 HP 00000001)
[ 0.021299] ACPI: Reserving DMAR table memory at [mem 0x7b7e7000-0x7b7e72cd]
[ 1.335138] DMAR: Host address width 46
[ 1.335143] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 1.335158] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[ 1.335169] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[ 1.335180] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[ 1.335188] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff
[ 1.335196] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff
[ 1.335205] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff
[ 1.335211] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff
[ 1.335218] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff
[ 1.335224] DMAR: ATSR flags: 0x0
[ 1.335229] DMAR: ATSR flags: 0x0
[ 1.335237] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0
[ 1.335245] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1
[ 1.335252] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1
[ 1.335258] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[ 1.335264] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 1.336482] DMAR-IR: Enabled IRQ remapping in x2apic mode

if compgen -G "/sys/kernel/iommu_groups/*/devices/*" > /dev/null; then echo "AMD's IOMMU / Intel's VT-D is enabled in the BIOS/UEFI."; else echo "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI"; fi
AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI

But it is

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-8-pve root=/dev/mapper/pve-root ro ”quiet

And my GRUB file looks like this that I added;

GRUB_CMDLINE_LINUX_DEFAULT=”quiet mitigations=off intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction pci=nommconf”

 

[leesteken]

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-8-pve root=/dev/mapper/pve-root ro ”quiet

intel_iommu=on is missing and IOMMU won't be enabled in Proxmox. The manual describes how to add it, depending on the bootloader (little mistake in the manual: you still need it on Intel after kernel 5.14).

And my GRUB file looks like this that I added;
GRUB_CMDLINE_LINUX_DEFAULT=”quiet mitigations=off intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction pci=nommconf”

You used the wrong double-quotes. Don't use ” or ” but use the correct ". Did you copy them from a website or office document?
Don't use pcie_acs_override=downstream,multifunction except for a last (and very insecure) resort. Similar to mitigations=off, VMs can steal data from other VMs and the Proxmox host.

 

[fbeye]

That is what is throwing me off... When I edit the /etc/kernel/cmdline, it is empty! I do not even know where cat /proc/cmdline gets it's information.
Yes, I did copy/paste the line in the /etc/default/grub.
I mean, crap, I am pretty sure it is using grub!! When I installed, I did not specify anything, unless it does not use grub by default... Maybe I am not.