HP Server - is pcie passthrough possible?

[senadman]

Hello all,

I am running Proxmox 7.2 on a HP DL380 G9 server and have done so for a while without any major issues except one - pcie passthrough.

All steps have been followed (wiki etc). No matter what I try task log complains with "Error: start failed: QEMU exited with code 1".

I am trying to passthrough 10gbe NIC to a VM

root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.023143] ACPI: DMAR 0x000000007B7E7000 0002D0 (v01 HP     ProLiant 00000001 HP   00000001)
[    0.023233] ACPI: Reserving DMAR table memory at [mem 0x7b7e7000-0x7b7e72cf]
[    1.267872] DMAR: IOMMU enabled
[    3.003073] DMAR: Host address width 46
[    3.003076] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    3.003087] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    3.003092] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    3.003099] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[    3.003103] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff
[    3.003106] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff
[    3.003111] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff
[    3.003114] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff
[    3.003116] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff
[    3.003120] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    3.003124] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    3.003127] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    3.003130] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    3.003133] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    3.004368] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    3.387604] DMAR: No ATSR found
[    3.387606] DMAR: No SATC found
[    3.387609] DMAR: dmar0: Using Queued invalidation
[    3.387615] DMAR: dmar1: Using Queued invalidation
[    3.400411] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 2405.845061] vfio-pci 0000:0b:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[ 2500.728124] vfio-pci 0000:0b:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[ 3822.881991] vfio-pci 0000:0b:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

The last three lines to me seem to be an issue, does anyone have any guidance?

 

[leesteken]

Maybe you can update the BIOS or find a work-around in the other RMRR topics here? Sorry, I have no experience with this but it appears to be a common issue on HP.

 

[Urbanovits]

Same issue on HP DL380 G9 with LSI MegaRAid

syslog:

Oct 29 21:43:39 pve03 kernel: [ 118.555080] vfio-pci 0000:01:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
Oct 29 21:43:40 pve03 kernel: [ 118.737813] vmbr0: port 9(tap150i0) entered disabled state
Oct 29 21:43:40 pve03 pvedaemon[4219]: VM 150 qmp command failed - VM 150 not running
Oct 29 21:43:40 pve03 pvestatd[4025]: VM 150 qmp command failed - VM 150 not running
Oct 29 21:43:40 pve03 systemd[1]: 150.scope: Succeeded.
Oct 29 21:43:40 pve03 pvedaemon[7263]: start failed: QEMU exited with code 1


dmesg |grep -e DMAR -e IOMMU -e AMD-Vi


[ 118.555080] vfio-pci 0000:01:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.


Any working solution?

 

[GastonJ]

I did try run up my DL380G7 last weekend, and hit the same problem. There is a "fix" for this but it involves compiling the kernel. However I've not successfully managed this as yet.

You probably want the thread titled

Compile Proxmox VE with patched intel-iommu driver to remove RMRR check​

HTH - I'd sooner have something that didn't break every time a new kernel was released by Proxmox.

 

[Urbanovits]

HI,

At first most of mentioned issue appearing on G9 servers. I have G1 to G10, G9 is the problematic part only.

Yes I did a test on a custom kernel (github as source), which is fixed the "DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement" error. But another issue appeared> Some memory allocation error (I don't remember exactly what was it and how it was look like) and PCIe passthrue does not work exempt on first boot only. All settings on SCSI card does not have any effect and boot failed.

Last but not least maybe a server BIOS update AND/OR SCSI card firmware copy&manage on ProxMox would help, but I do not have access to download updates from HP support site.

Project of the PCIe passthrue postponed here.

 

[LnxBil]

At first most of mentioned issue appearing on G9 servers. I have G1 to G10, G9 is the problematic part only.

Really? I have a G6 here and in recent PVE's I'm unable to get stable pcie passthrough to work. It worked fine before (as the hardware was a little bit younger).

 

[Urbanovits]

Really? I have a G6 here and in recent PVE's I'm unable to get stable pcie passthrough to work. It worked fine before (as the hardware was a little bit younger).

Instead of fighting, I suppose we have to focus on to find solution.
What I shared here is my thoughts and experiences. Not a demand or whatever like that.

Still in question focus: PCIe passthure.

 

[D0doooh]

I also have the same problem, are there any successes? I'm getting desperate...

 

[Azabua]

I have this problem as well with my HP DL380p G8 and i found an artikel that shoud fix it but i didnt tried it yet.

DPDK and RMRR Compatibility Issues on the HP Proliant DL360e G8​

 

[D0doooh]

I have this problem as well with my HP DL380p G8 and i found an artikel that shoud fix it but i didnt tried it yet.

DPDK and RMRR Compatibility Issues on the HP Proliant DL360e G8​

have a look here:
https://forum.proxmox.com/threads/pci-passthrough-problems.65485/#post-519449

it helped me and everything is running now