Pci passthrough problems

Discolai

New Member
Feb 16, 2020
1
0
1
23
HI, I am having trouble setting up IOMMU for pci passthrough on my system. The server is a HP ProLiant DL380 Gen9 running proxmox 6.0-4. I have enabled all the virtualization options within the bios including vt-d.

I followed the instructions on the wiki to set up iommu support, but proxmox still reports that it cannot detect IOMMU when I try to add a new pci device to a vm. In addition all the devices in the dropdown show up belonging to the same IOMMU group "-1".

The output from "dmesg | grep -e DMAR -e IOMMU" is:

[ 0.027294] ACPI: DMAR 0x000000007B7E7000 00032A (v01 HP ProLiant 00000001 HP 00000001) [ 3.746102] DMAR: Host address width 46 [ 3.746106] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 3.746118] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746123] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 3.746132] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746137] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff [ 3.746141] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff [ 3.746145] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff [ 3.746149] DMAR: RMRR base: 0x000000634a1000 end: 0x000000674a0fff [ 3.746153] DMAR: RMRR base: 0x000000674a1000 end: 0x0000006b4a0fff [ 3.746156] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff [ 3.746160] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff [ 3.746166] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0 [ 3.746171] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746174] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746178] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 3.746182] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 3.747344] DMAR-IR: Enabled IRQ remapping in x2apic mode
 

lhorace

Active Member
Oct 17, 2015
186
24
38
According to https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM and looking at my kernel config on my HP DL360e Gen8. CONFIG_DMAR_DEFAULT_ON doesn't appear to be set, you might need to set the below on the kernel command line for iommu to function:

Code:
intel_iommu=on

Additionally, you might want to give this code a try:

Bash:
if compgen -G "/sys/kernel/iommu_groups/*/devices/*" > /dev/null; then
    echo "AMD's IOMMU / Intel's VT-D is enabled in the BIOS/UEFI."
else
    echo "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI"
fi
 

User5665

New Member
Apr 26, 2022
10
1
3
HI, I am having trouble setting up IOMMU for pci passthrough on my system. The server is a HP ProLiant DL380 Gen9 running proxmox 6.0-4. I have enabled all the virtualization options within the bios including vt-d.

I followed the instructions on the wiki to set up iommu support, but proxmox still reports that it cannot detect IOMMU when I try to add a new pci device to a vm. In addition all the devices in the dropdown show up belonging to the same IOMMU group "-1".

The output from "dmesg | grep -e DMAR -e IOMMU" is:

[ 0.027294] ACPI: DMAR 0x000000007B7E7000 00032A (v01 HP ProLiant 00000001 HP 00000001) [ 3.746102] DMAR: Host address width 46 [ 3.746106] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 3.746118] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746123] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 3.746132] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 3.746137] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff [ 3.746141] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff [ 3.746145] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff [ 3.746149] DMAR: RMRR base: 0x000000634a1000 end: 0x000000674a0fff [ 3.746153] DMAR: RMRR base: 0x000000674a1000 end: 0x0000006b4a0fff [ 3.746156] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff [ 3.746160] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff [ 3.746166] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0 [ 3.746171] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746174] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1 [ 3.746178] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 3.746182] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 3.747344] DMAR-IR: Enabled IRQ remapping in x2apic mode

Hi Discolai,

I am having the exact same issues with my ProLiant DL380 gen9 when trying to enable IOMMU in Proxmox v7.1. I followed the official guide on how to enable PIC Passtrough but the output of the command "dmesg | grep -e DMAR -e IOMMU" is the same as yours, my output is:

Code:
root@proliant:~# dmesg | grep -e DMAR -e IOMMU
[    0.023801] ACPI: DMAR 0x000000007B7E7000 000294 (v01 HP     ProLiant 00000001 HP   00000001)
[    0.023902] ACPI: Reserving DMAR table memory at [mem 0x7b7e7000-0x7b7e7293]
[    1.404101] DMAR: Host address width 46
[    1.404105] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.404119] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    1.404128] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.404137] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    1.404144] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff
[    1.404151] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff
[    1.404157] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff
[    1.404163] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff
[    1.404170] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff
[    1.404178] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.404185] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.404190] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.404195] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.404200] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.405227] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@proliant:~#

I do not understand the output and what it might hint at.
I checked the settings in the BIOS of the Server. "vt-d", which was the only setting I was able to find, is enabled (see attached screenshot of the BIOS Settings).

If I insert the command provided by Ihorace:
Code:
if compgen -G "/sys/kernel/iommu_groups/*/devices/*" > /dev/null; then
    echo "AMD's IOMMU / Intel's VT-D is enabled in the BIOS/UEFI."
else
    echo "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI"
fi

the output is:

Code:
AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI

although I did enable it (see attached screenshot of the BIOS Settings)!

The command:

Code:
find /sys/kernel/iommu_groups/ -type l

shows no output at all.

I am out of ideas... what was your solution?
I checked an the CPUs E5-2650 v3 from Intel do support IOMMU. I was not able to check the Motherboard because I did not know how (could not find any Model Number) but it does not make any sense to me that this server grade motherboard would not support it.

I checked which Boot Mode the Server is using and it is UEFI according to the BIOS information. I edited
- /etc/default/grub
and
- /etc/kernel/cmdline

and used the commands:
-update-grub
and
-proxmox-boot-tool refresh

accordingly, because it did not work at first when only updating grub. I configured the modules in /etc/modules as well and ran "update-initramfs" to update kernel modules.

I rebooted after each change with "reboot" in the Proxmox CLI.v
Thanks in Advance to anybody for any help!
 

Attachments

  • Captura de Pantalla 2022-04-26 a las 16.33.40.png
    Captura de Pantalla 2022-04-26 a las 16.33.40.png
    237.7 KB · Views: 3
Last edited:

User5665

New Member
Apr 26, 2022
10
1
3
According to https://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM and looking at my kernel config on my HP DL360e Gen8. CONFIG_DMAR_DEFAULT_ON doesn't appear to be set, you might need to set the below on the kernel command line for iommu to function:

Code:
intel_iommu=on

Additionally, you might want to give this code a try:

Bash:
if compgen -G "/sys/kernel/iommu_groups/*/devices/*" > /dev/null; then
    echo "AMD's IOMMU / Intel's VT-D is enabled in the BIOS/UEFI."
else
    echo "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI"
fi
Hi Ihorace, I just commented on the initial question of Discolai.
I tried you code which gave the output "AMD's IOMMU / Intel's VT-D is not enabled in the BIOS/UEFI" even though I configured Intel's VT-D in the BIOS. Do you have any ideas?
Your help is much appreciated!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!