I have posted this information to reddit as well... but I have been having some issues getting PCI-E Passthough working correctly on my older gear. I wasn't sure if cross-posting there or somewhere else would be effective or not, but either way I was hoping to get some assistance in getting PCI-E passthrough working on my HP DL380 Gen7.
Short description: PCI-E Pass-through not working on older hardware with "known" firmware issue (known in quotes because it is documented but actually getting the info was fairly exhaustive). I have run out of options and I'm hoping for some assistance. You can skip the full description, though it does add context
Hardware:
- HP DL380 Gen7 with "firmware bug"
- BIOS: P67 05/21/2018
- NVidia Quadro P620 to be passed through (for nvenc on Plex VM)
- PVE 7.3-6 (fully updated)
Full description (for context):
This server was running ESXi 6.7U3 with no issues other than I did have some odd failures here an there (hardware mostly), and the fact that 6.7U3 was a bit long at the tooth. The backup solution that I was running (ghettoVCB), was okay, it took a while to run, and since I had some issues with boot media, it was kind of a pain to get back up and running (gladly I saved my configs), but I have another server that has been running really well with proxmox, and I wanted to give it a go on this machine and maybe experiment with some clustering. I also really like the backup setup on proxmox. After dealing with some issues on installation (video related), I got the install working with 7.1 and upgraded to 7.3 without issues. I only moved one VM back onto the machine for testing... the one I need for PCI-E Passthrough (I figured this would be the hardest thing).
The other server I have running proxmox (which is a supermicro machine from a similar generation) I also run PCI-E Passthrough on, and it seems to be flawless (TrueNAS Core and a Windows VM running Veeam connected to an external SAS LTO-4 drive).
I seem to have hit the same wall that a lot of people have on this... only I cannot seem to actually fix the issue. The problem with most of the information that I have found is that it is for Gen8 servers, which although close are definitely not the same.
Troubleshooting Steps:
- Verified that all of the virtualization related settings were on in BIOS (they were, but always good to double check). Hit Ctrl+A in BIOS, no shared memory settings available.
- Verified BIOS was up to date, it initially was not (!), I updated to the latest available version. then re-checked all settings, still also no shared memory settings.
- Followed steps in the following links:
https://forum.level1techs.com/t/dl380p-proxmox-pci-passthrough-operation-not-permitted/159031
https://www.jimmdenton.com/proliant-intel-dpdk/
Also found some decent info here (blacklisting and such): https://old.reddit.com/r/Proxmox/comments/lcnn5w/proxmox_pcie_passthrough_in_2_minutes/
No change, I can get "conrep" to work, I had some issues initially, but got the HPhealth package installed and things seem to work properly. But the RMRDS_SlotX related things do NOT seem to take with my machine (again, gen 7, not 8... that's probably at least part of it).
- Gone through all of the PVE docs (https://pve.proxmox.com/wiki/Pci_passthrough)
I've tried a bung of different things, on and off... none appear to make any difference. Here is my current GRUB commandline:
I've tried the allow unsafe interrupts in both modprobe as well as on the command line. I will be removing the acs override, as it didn't change anything except the group numbers. Both the video and audio device on the NVidia card belonged to the same iommu group, and there was nothing else in that group. Below is the output from demsg related to DMAR
This other user here (https://www.reddit.com/r/Proxmox/comments/lhmlew/gpu_passthrough_interrupt_remapping_issue/) apparently nailed it some how... but I can't seem to connect the dots here. I have no doubt there is something going on with the audio side (that's the .1 at the end of the pci device ident.)... but there's nothing on how to fix it other than just passing the video, not ALL functions (which I have tried).
I really could use some help, and I'm happy to provide any additional info needed. There's probably something stupid I missed somewhere... or buried in some archive... at least that's the hope. I don't want to get rid of this machine quite yet, and I really don't want to crawl back to VMware... because nothing newer than 6.7U3 is supported.
Thank you!
Short description: PCI-E Pass-through not working on older hardware with "known" firmware issue (known in quotes because it is documented but actually getting the info was fairly exhaustive). I have run out of options and I'm hoping for some assistance. You can skip the full description, though it does add context
Hardware:
- HP DL380 Gen7 with "firmware bug"
- BIOS: P67 05/21/2018
- NVidia Quadro P620 to be passed through (for nvenc on Plex VM)
- PVE 7.3-6 (fully updated)
Full description (for context):
This server was running ESXi 6.7U3 with no issues other than I did have some odd failures here an there (hardware mostly), and the fact that 6.7U3 was a bit long at the tooth. The backup solution that I was running (ghettoVCB), was okay, it took a while to run, and since I had some issues with boot media, it was kind of a pain to get back up and running (gladly I saved my configs), but I have another server that has been running really well with proxmox, and I wanted to give it a go on this machine and maybe experiment with some clustering. I also really like the backup setup on proxmox. After dealing with some issues on installation (video related), I got the install working with 7.1 and upgraded to 7.3 without issues. I only moved one VM back onto the machine for testing... the one I need for PCI-E Passthrough (I figured this would be the hardest thing).
The other server I have running proxmox (which is a supermicro machine from a similar generation) I also run PCI-E Passthrough on, and it seems to be flawless (TrueNAS Core and a Windows VM running Veeam connected to an external SAS LTO-4 drive).
I seem to have hit the same wall that a lot of people have on this... only I cannot seem to actually fix the issue. The problem with most of the information that I have found is that it is for Gen8 servers, which although close are definitely not the same.
Troubleshooting Steps:
- Verified that all of the virtualization related settings were on in BIOS (they were, but always good to double check). Hit Ctrl+A in BIOS, no shared memory settings available.
- Verified BIOS was up to date, it initially was not (!), I updated to the latest available version. then re-checked all settings, still also no shared memory settings.
- Followed steps in the following links:
https://forum.level1techs.com/t/dl380p-proxmox-pci-passthrough-operation-not-permitted/159031
https://www.jimmdenton.com/proliant-intel-dpdk/
Also found some decent info here (blacklisting and such): https://old.reddit.com/r/Proxmox/comments/lcnn5w/proxmox_pcie_passthrough_in_2_minutes/
No change, I can get "conrep" to work, I had some issues initially, but got the HPhealth package installed and things seem to work properly. But the RMRDS_SlotX related things do NOT seem to take with my machine (again, gen 7, not 8... that's probably at least part of it).
- Gone through all of the PVE docs (https://pve.proxmox.com/wiki/Pci_passthrough)
I've tried a bung of different things, on and off... none appear to make any difference. Here is my current GRUB commandline:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"I've tried the allow unsafe interrupts in both modprobe as well as on the command line. I will be removing the acs override, as it didn't change anything except the group numbers. Both the video and audio device on the NVidia card belonged to the same iommu group, and there was nothing else in that group. Below is the output from demsg related to DMAR
Code:
dmesg | grep DMAR
[ 0.010249] ACPI: DMAR 0x00000000C762FE80 000186 (v01 HP ProLiant 00000001 \xd2? 0000162E)
[ 0.010307] ACPI: Reserving DMAR table memory at [mem 0xc762fe80-0xc7630005]
[ 0.660546] DMAR: IOMMU enabled
[ 1.522106] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 2.057124] DMAR: Host address width 39
[ 2.057127] DMAR: DRHD base: 0x000000cfffe000 flags: 0x1
[ 2.057138] DMAR: dmar0: reg_base_addr cfffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[ 2.057141] DMAR: RMRR base: 0x000000c77fc000 end: 0x000000c77fdfff
[ 2.057144] DMAR: RMRR base: 0x000000c77f5000 end: 0x000000c77fafff
[ 2.057146] DMAR: RMRR base: 0x000000c763e000 end: 0x000000c763ffff
[ 2.057147] DMAR: ATSR flags: 0x0
[ 2.057190] DMAR: No SATC found
[ 2.057198] DMAR: dmar0: Using Queued invalidation
[ 2.061500] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 151.091447] vfio-pci 0000:11:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.This other user here (https://www.reddit.com/r/Proxmox/comments/lhmlew/gpu_passthrough_interrupt_remapping_issue/) apparently nailed it some how... but I can't seem to connect the dots here. I have no doubt there is something going on with the audio side (that's the .1 at the end of the pci device ident.)... but there's nothing on how to fix it other than just passing the video, not ALL functions (which I have tried).
I really could use some help, and I'm happy to provide any additional info needed. There's probably something stupid I missed somewhere... or buried in some archive... at least that's the hope. I don't want to get rid of this machine quite yet, and I really don't want to crawl back to VMware... because nothing newer than 6.7U3 is supported.
Thank you!
