Is it impossible to setup pcie passthrough for Proliant gen8 servers ?

L

Lancer

New Member
Sep 10, 2023
10
1
1
Hi
My server is DL380p gen8 server, and I did read about and test various solutions but ALL of them failed.
I must admit that most of them are about GPU passthrough, and I just need to passthrough my p420i controller from proxmox to truenas (and a sas hba later).

So is there any proven way to do it? or we have to deal with it as is ?!
Thank you
 
  • Like
Reactions: gjani
Lancer said:
My server is DL380p gen8 server, and I did read about and test various solutions but ALL of them failed.
Click to expand...
Did you enable PCIe passthrough as per the Proxmox manual? Without a clearer description of the failures, it might just be a configuration mistake or maybe you need a motherboard BIOS update.
Lancer said:
I must admit that most of them are about GPU passthrough, and I just need to passthrough my p420i controller from proxmox to truenas (and a sas hba later).
Click to expand...
Did passthrough of the device give you trouble or could you not enabled VT-d? Do you known that Proxmox has two possible bootloaders?
Lancer said:
So is there any proven way to do it? or we have to deal with it as is ?!
Click to expand...
I don't know your particular hardware, so maybe someone else here can comment. There a some threads about DL380 and passthrough. Your system might have "the RMRR problem" and require an involved work-around or fix from HP.
 
  • Like
Reactions: Lancer
leesteken said:
Did you enable PCIe passthrough as per the Proxmox manual? Without a clearer description of the failures, it might just be a configuration mistake or maybe you need a motherboard BIOS update.

Did passthrough of the device give you trouble or could you not enabled VT-d? Do you known that Proxmox has two possible bootloaders?

I don't know your particular hardware, so maybe someone else here can comment. There a some threads about DL380 and passthrough. Your system might have "the RMRR problem" and require an involved work-around or fix from HP.
Click to expand...
Thank you so much

Yes I did All the instructions, I just didn't do the kernel edit since I don't see that it will work in the long run and I kinda understand that it got included in proxmox kernel.

As in the manual I did verfiy with (lspci -nnk) and I do get (Kernel driver in use: vfio-pci) !

But I still couldn't run the vm, and got this message:
kvm: -device vfio-pci,host=0000:02:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:02:00.0: failed to setup container for group 49: Failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: QEMU exited with code 1

So what to do now?
 
Just read this:
https://forum.proxmox.com/threads/hpe-ml-dl-server-series-pci-gpu-passthrough-pve8.131361/

kvm: -device vfio-pci,host=0000:02:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:02:00.0: failed to setup container for group 49: Failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: QEMU exited with code 1
Click to expand...

HPE IOMMU/passthrough is not enabled, its need to enable per PCI-E slot using the "conrep" util from HPE, after you enabled on the specific PCI-E slot, there will be no more error.
 
Last edited:
  • Like
Reactions: Lancer
Lancer said:
Thank you so much

Yes I did All the instructions, I just didn't do the kernel edit since I don't see that it will work in the long run and I kinda understand that it got included in proxmox kernel.

As in the manual I did verfiy with (lspci -nnk) and I do get (Kernel driver in use: vfio-pci) !

But I still couldn't run the vm, and got this message:
kvm: -device vfio-pci,host=0000:02:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:02:00.0: failed to setup container for group 49: Failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: QEMU exited with code 1

So what to do now?
Click to expand...
Sounds like the RMRR problem. I have no experience with that but maybe this can provide a work-around? Looks like you selected hardware that does not work well with passthrough.
 
  • Like
Reactions: Lancer
emunt6 said:
Just read this:
https://forum.proxmox.com/threads/hpe-ml-dl-server-series-pci-gpu-passthrough-pve8.131361/



HPE IOMMU/passthrough is not enabled, its need to enable per PCI-E slot using the "conrep" util from HPE, after you enabled on the specific PCI-E slot, there will be no more error.
Click to expand...
Thank you
I think it's enabled as I check with this command:
Code: 
# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.007107] ACPI: DMAR 0x00000000BDDAD200 0006A8 (v01 HP     ProLiant 00000001 \xd2?   0000162E)
[    0.007164] ACPI: Reserving DMAR table memory at [mem 0xbddad200-0xbddad8a7]
[    1.162964] DMAR: IOMMU enabled
[    2.597420] DMAR: Host address width 46
[    2.597422] DMAR: DRHD base: 0x000000fbefe000 flags: 0x0
[    2.597431] DMAR: dmar0: reg_base_addr fbefe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    2.597434] DMAR: DRHD base: 0x000000eaffe000 flags: 0x1
[    2.597445] DMAR: dmar1: reg_base_addr eaffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    2.597447] DMAR: RMRR base: 0x000000bdffd000 end: 0x000000bdffffff
[    2.597451] DMAR: RMRR base: 0x000000bdff6000 end: 0x000000bdffcfff
[    2.597452] DMAR: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff
[    2.597454] DMAR: RMRR base: 0x000000bdf7f000 end: 0x000000bdf82fff
[    2.597455] DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff
[    2.597456] DMAR: RMRR base: 0x000000bdf6e000 end: 0x000000bdf6efff
[    2.597457] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
[    2.597458] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
[    2.597460] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
[    2.597530] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000000e8000-0x00000000000e8fff]
[    2.597532] DMAR: RMRR base: 0x000000bddde000 end: 0x000000bdddefff
[    2.597533] DMAR: ATSR flags: 0x0
[    2.597541] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbefe000 IOMMU 0
[    2.597542] DMAR-IR: IOAPIC id 8 under DRHD base  0xeaffe000 IOMMU 1
[    2.597544] DMAR-IR: IOAPIC id 0 under DRHD base  0xeaffe000 IOMMU 1
[    2.597546] DMAR-IR: HPET id 0 under DRHD base 0xeaffe000
[    2.598376] DMAR-IR: Enabled IRQ remapping in xapic mode
[    4.051504] DMAR: No SATC found
[    4.051510] DMAR: dmar0: Using Queued invalidation
[    4.051540] DMAR: dmar1: Using Queued invalidation
[    4.063053] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    5.808849] DMAR: DRHD: handling fault status reg 2
[    5.808944] DMAR: [INTR-REMAP] Request device [01:00.0] fault index 0x23 [fault reason 0x26] Blocked an interrupt request due to source-id verification failure

Anyway I did the instruction in your link, but the dead end was in this command:
lspci -s 0c:00.0 -vvv | grep 'Physical Slot'
Because I got nothing from it ! maybe because p420i is embedded ?

Hope you can help with that, thanks
 
emunt6 said:
He don't need any "hack/patch", just use the HPE "conrep" util to enable the IOMMU on specific PCI-E slot (disabled by default on all PCI-E slot).
Click to expand...
Please see my previous reply. I couldn't find a 'Physical Slot' for my p420i.
Can I do it without this step ?
 
Lancer said:
Please see my previous reply. I couldn't find a 'Physical Slot' for my p420i.
Can I do it without this step ?
Click to expand...
I checked, it has no "busid" as the other hardware:
Code: 
$> ssacli ctrl all show
    Smart Array P420i in Slot 0 (Embedded)
You can try the "Slot0" with conrep - I do not know if this works.

Also, need to blacklist the "hpsa" kernel-module to prevent load the driver on the HOST.

Example:

Code: 
Example:
/etc/modprobe.d/hba.conf
   blacklist hpsa

$> update-initramfs -c -d -u

If the conrep fails, then its not possible to passthrough.
 
Last edited:
  • Like
Reactions: Lancer
emunt6 said:
I checked, it has no "busid" as the other hardware:
Code: 
$> ssacli ctrl all show
    Smart Array P420i in Slot 0 (Embedded)
You can try the "Slot0" with conrep - I do not know if this works.

Also, need to blacklist the "hpsa" kernel-module to prevent load the driver on the HOST.

Example:

Code: 
Example:
/etc/modprobe.d/hba.conf
   blacklist hpsa

$> update-initramfs -c -d -u

If the conrep fails, then its not possible to passthrough.
Click to expand...
Thanks
I did block hpsa.
And I have to indecate two points:
1- Suddenly my server goes to 100% fan speed, then iLO told me about a storage controller failure (I think), not once but multiple times ! and I think it was triggered by my passthrough changes.
2- in the server console I got this message: "dmar device is ineligible for iommu domain attach due to platform rmrr requirement..." and others you can see in the attachment.
Also earlier I got this message "L1TF SPU bug present and SMT on, data leak possible. ...".
 

Attachments

  • dmar.png
    dmar.png
    73.2 KB · Views: 6
Last edited:
Was anyone able to use the P420i in a vm with IOMMU? Is this possible? I'm trying on an HP Proliant DL380p Gen8.

Or did anyone met with the Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping. Contact your platform vendor. error?
I get this error when trying to start a VM that I added the P420i Storage Controller to as a PCI device.

I can't find many references to this error except the kernel source code itself which I'm not very familiar with: https://github.com/torvalds/linux/blob/master/drivers/iommu/iommu.c#L2247
EDIT: I got the same error message for an NVME drive when I tried to start a VM with that drive added as a PCIe device before excluding the drive's PCIe slot by conrep. So maybe there is a connection here? Maybe the P420i only gives this error message because it is used by iLo but it can't be excluded by conrep because it has no physical slot (or I just don't know of it)?


When I open the iLo web UI and navigate to "Information" > "System Information" > "Storage" > "Controller on System Board" > "Physical View" I can see the drives attached to the storage card, so I guess iLo uses the P420i Storage Controller. So maybe iLo is using the PCIe interface and that's why it isn't usable by a VM?

When I do lspci -vvv I can see Kernel modules: hpsa for the P420i (even though I tried blacklisting the hpsa kernel module by creating a file named /etc/modprobe.d/blacklist.con with the content blacklist hpsa and then running update-initramfs -u -k all and rebooting)



FWIW
  • I can pass through other PCIe devices to VMs (I tried passing through an NVME drive to a TrueNAS vm and it works fine)
  • I changed the firmware of the P420i to work as an HBA (NOT a bunch of RAID0s) by following this guide, so the drives I attach to it show up in the proxmox host when I run lsblk : https://www.reddit.com/r/homelab/comments/yjkx2w/comment/jmhn7pt/
  • I use GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" in /etc/default/grub
  • pveversion output is: pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.8.4-3-pve)
  • I did not play with conrep yet (because there is no Physical Slot displayed for P420i by lspci)

The output of the below command includes "Your BIOS is broken; bad RMRR" but I'm not sure if that has anything to do with the "Firmware has requested this device have a 1:1 IOMMU mapping..." error I get when starting the VM
Code: 
root@borzos:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.000000] ACPI: DMAR 0x00000000BDDAD200 00060C (v01 HP     ProLiant 00000001 \xd2?   0000162E)
[    0.000000] ACPI: Reserving DMAR table memory at [mem 0xbddad200-0xbddad80b]
[    0.000000] DMAR: IOMMU enabled
[    0.000000] DMAR: Host address width 46
[    0.000000] DMAR: DRHD base: 0x000000fb8fe000 flags: 0x0
[    0.000000] DMAR: dmar0: reg_base_addr fb8fe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.000000] DMAR: DRHD base: 0x000000f1ffe000 flags: 0x1
[    0.000000] DMAR: dmar1: reg_base_addr f1ffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.000000] DMAR: RMRR base: 0x000000bdffd000 end: 0x000000bdffffff
[    0.000000] DMAR: RMRR base: 0x000000bdff6000 end: 0x000000bdffcfff
[    0.000000] DMAR: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff
[    0.000000] DMAR: RMRR base: 0x000000bdf7f000 end: 0x000000bdf82fff
[    0.000000] DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff
[    0.000000] DMAR: RMRR base: 0x000000bdf6e000 end: 0x000000bdf6efff
[    0.000000] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
[    0.000000] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
[    0.000000] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
[    0.000000] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000000e8000-0x00000000000e8fff]
[    0.000000] DMAR: RMRR base: 0x000000bddde000 end: 0x000000bdddefff
[    0.000000] DMAR: ATSR flags: 0x0
[    0.000000] DMAR-IR: IOAPIC id 10 under DRHD base  0xfb8fe000 IOMMU 0
[    0.000000] DMAR-IR: IOAPIC id 8 under DRHD base  0xf1ffe000 IOMMU 1
[    0.000000] DMAR-IR: IOAPIC id 0 under DRHD base  0xf1ffe000 IOMMU 1
[    0.000000] DMAR-IR: HPET id 0 under DRHD base 0xf1ffe000
[    0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.001000] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.360304] DMAR: No SATC found
[    0.360310] DMAR: dmar0: Using Queued invalidation
[    0.360326] DMAR: dmar1: Using Queued invalidation
[    0.373044] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    2.460670] DMAR: DRHD: handling fault status reg 2
[    2.460683] DMAR: [INTR-REMAP] Request device [01:00.0] fault index 0x1e [fault reason 0x26] Blocked an interrupt request due to source-id verification failure

Code: 
root@borzos:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro nvme_core.default_ps_max_latency_us=0 quiet intel_iommu=on iommu=pt



When I run update-initramfs -u -k all I get the below messages. Is this a problem?
Code: 
update-initramfs: Generating /boot/initrd.img-6.8.4-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.152-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.143-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.39-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-6-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
 
Last edited:
gjani said:
When I run update-initramfs -u -k all I get the below messages. Is this a problem?
Code: 
update-initramfs: Generating /boot/initrd.img-6.8.4-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.152-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.143-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.39-3-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-6-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Click to expand...
This could be a problem, in the very least, it is not working at it supposedly should be. I found a solution in this thread:
https://forum.proxmox.com/threads/p...h-cannot-change-initramfs.126627/#post-631013

Short version:
  1. Verify the cause of the above symptoms. Run
    Code: 
    # proxmox-boot-tool status
    and check the output. If it says
    Code: 
    Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.
    then we found the most likely cause to @gjani post above, and can fix that easily:
  2. Find out which partition holds your /boot drive, watch for /boot and vfat file system. The example belows shows the relevant part on my system, with sdg2 being the device name and /boot/efi being the mount point:
    Code: 
    lsblk -o +FSTYPE
  NAME                                                               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS FSTYPE
  (...)
  ├─sdg2                                                               8:98   0     1G  0 part /boot/efi   vfat
  (...)
  3. Unmount the boot partition:
    Code: 
    umount /boot/efi
  4. Link the vfat partiton with proxmox-boot-tool (use the device name that you determined in step 2 (here /dev/sdg2):
    Code: 
    proxmox-boot-tool init /dev/sdg2
    This should then add the missing ID and generate output comparable to this one:
    Code: 
    Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..    UUID="5D00-9C32" SIZE="1073741824" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="sdg" MOUNTPOINT=""Mounting '/dev/sdg2' on '/var/tmp/espmounts/5D00-9C32'.
Installing grub x86_64 target..
Installing for x86_64-efi platform.
Installation finished. No error reported.
Installing grub x86_64 target (removable)..
Installing for x86_64-efi platform.
Installation finished. No error reported.
Unmounting '/dev/sdg2'.
Adding '/dev/sdg2' to list of synced ESPs..
Refreshing kernels and initrds..
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/5D00-9C32
After that,
Code: 
update-initramfs -u -k all
should run smoothly.
 
Last edited:
Thanks @timm_e_r, interesting enough I had two proxmox nodes, (one with UEFI, and one with Legacy BIOS) and both had the same issue for update-initramfs -u -k all right after fresh installs. But after running your solution they seem to be fixed.

For the node with Legacy BIOS I didn't run the unmount part because the vfat partition was not mounted. I it just a UEFI thing maybe and the update-initramfs errors were meaningless on a system with BIOS? I didn't bother getting into why because I didn't mind accidentally breaking this server since it's just part of a lab env - if you have info on it though, don't hold it back :D

For anyone in the future reading this, this was not a blocker for IOMMU though. IOMMU worked fine for me (on the DL380p Gen8) even with the seemingly broken update-initramfs -u -k all.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back