[TUTORIAL] Compile Proxmox VE with patched intel-iommu driver to remove RMRR check

Hello,
I have managed to follow the tutorial through successfully through to the point of the command "make" I get the following:
"make: *** No targets specified and no makefile found. Stop."
I'm sorry because I know the reason of failure is my own incompetence. I would be very grateful if you could help me.
Thanks
Arthur
(Idk if important but this is on HP Proliant DL380 G7)
 
Hey Guys, might have found a better way of doing this, and it's partly supported by HPe!

It let's you choose individual slots to disable RMRR and enables some pretty sweat developer things!

going to be trying it with a Pop_OS test drive with Qemu (the platform that proxmox uses for virtualization! )

https://www.jimmdenton.com/proliant-intel-dpdk/

Essentially, the bios has a the native ability to exclude individual slots from RMRR, but this isn't exposed by the rom. this process uses OS level tools and doesn't need to recompile the kernel!
 
  • Like
Reactions: Whitterquick
FelixCLC said:
Hey Guys, might have found a better way of doing this, and it's partly supported by HPe!

It let's you choose individual slots to disable RMRR and enables some pretty sweat developer things!

going to be trying it with a Pop_OS test drive with Qemu (the platform that proxmox uses for virtualization! )

https://www.jimmdenton.com/proliant-intel-dpdk/

Essentially, the bios has a the native ability to exclude individual slots from RMRR, but this isn't exposed by the rom. this process uses OS level tools and doesn't need to recompile the kernel!
Click to expand...

Interested to hear about the results of this! I feel like I have come across this before and no joy, but let us know if it works for you. The kernel seems to have an effect on the ability to passthrough as it’s the only part changed when it works or not. MicroServer Gen8 seems to have different BIOS to other Gen8 servers too...
 
This does have to be done for RMRR to be able to work, but then on top of this you also need to patch the Kernel afaik.
 
Whitterquick said:
This does have to be done for RMRR to be able to work, but then on top of this you also need to patch the Kernel afaik.
Click to expand...
Won't have a chance to test until this weekend, but I don't think that's correct- Since without RMRR, this should function as a relatively standard Iommu/slot passthrough to the VFIO driver, which is then passed through to the QEMU server
 
FelixCLC said:
Won't have a chance to test until this weekend, but I don't think that's correct- Since without RMRR, this should function as a relatively standard Iommu/slot passthrough to the VFIO driver, which is then passed through to the QEMU server
Click to expand...
Gosh I hope u are right! I have done it on mine and it worked only after patching an older version of the kernel. The latest still doesn’t work as some stuff has moved to and from the kernel and I’m not experienced enough with Linux to work it all out. I had never got passthrough working before I had done this. Hopefully u can make some discoveries here! :D
 
Alright, update. Looked through the Conrep file that is being written to rom (basically bios). I've attached the output of that file and a secondary file from the verification step.

What get's me is that, although the shortened/verification of conrep shows excluded for slots 4, 5 ,6, I can't tell any real changes in the Conrep file.

Need's more investigation- but I think I'm on to something. Might have to manually edit the Conrep, things tbd
 

Attachments

  • onrep_rmrds.xml.txt
    5.3 KB · Views: 3
  • verify.dat.txt
    1.4 KB · Views: 5
  • Like
Reactions: Whitterquick
Next update. Having excluded slots 4, 5 and 6 (each one has to be done independently via conrep) and running dmesg I got the following:

Bash: 
dmesg | grep -e DMAR -e IOMMU
[    0.008216] ACPI: DMAR 0x00000000C761FE80 000184 (v01 HP     ProLiant 00000001 \xd2?   0000162E)
[    0.716473] DMAR: IOMMU enabled
[    1.369892] DMAR-IR: This system BIOS has enabled interrupt remapping
[    2.745835] DMAR: Host address width 39
[    2.745837] DMAR: DRHD base: 0x000000cfffe000 flags: 0x1
[    2.745860] DMAR: dmar0: reg_base_addr cfffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[    2.745861] DMAR: RMRR base: 0x000000c77fc000 end: 0x000000c77fdfff
[    2.745862] DMAR: RMRR base: 0x000000c77f5000 end: 0x000000c77fafff
[    2.745863] DMAR: RMRR base: 0x000000c763e000 end: 0x000000c763ffff
[    2.745864] DMAR: ATSR flags: 0x0
[    2.746209] DMAR: dmar0: Using Queued invalidation
[    2.764765] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    4.126828] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    4.126829] AMD-Vi: AMD IOMMUv2 functionality not available on this system
As almost anyone who's been in this thread knows, that's a pretty darn good sign. Worth noting is that I'm trying this on an UNPATCHED version of PVe, so if this works, we can deprecate the patch process and point users to this process instead!
 
  • Like
Reactions: Whitterquick
Update 2:

tried to start a VM with my LSI sas controller passed through via VFIO (could use SR-IOV, but for this sort of testing it doesn't matter)

What's REALLY weird to me is that, after a reboot (following instructions per https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04781229)

it now shows that those PCI-e slots are included from RMRR, even tho conrep reported that they were Excluded prior to reboot.

Update 3: per https://forums.unraid.net/topic/530...for-iommu-domain-attach-due-to-platform-rmrr/

it seems it might be possible to use a downgraded bios (poster at the end had it working with a bios from 2009) and then enabling ACS_override. Granted this was on unraid, but being as they're both using mostly the same virtualization mechanisms (VFIO for starters) this might work
 
Last edited:
FelixCLC said:
Update 2:

tried to start a VM with my LSI sas controller passed through via VFIO (could use SR-IOV, but for this sort of testing it doesn't matter)

What's REALLY weird to me is that, after a reboot (following instructions per https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04781229)

it now shows that those PCI-e slots are included from RMRR, even tho conrep reported that they were Excluded prior to reboot.

Update 3: per https://forums.unraid.net/topic/530...for-iommu-domain-attach-due-to-platform-rmrr/

it seems it might be possible to use a downgraded bios (poster at the end had it working with a bios from 2009) and then enabling ACS_override. Granted this was on unraid, but being as they're both using mostly the same virtualization mechanisms (VFIO for starters) this might work
Click to expand...

Interesting... did you have a password on your BIOS?
 
Question is, would you rather an old BIOS and new software, or newer BIOS and slightly older kernel...? :rolleyes::D
 
Whitterquick said:
Question is, would you rather an old BIOS and new software, or newer BIOS and slightly older kernel...? :rolleyes::D
Click to expand...
definitely the newer kernel- Haven't done kernel dev in a while, but the guys over at the linux foundation are nothing short of wizards

Also, on topic, I wonder if it's an issue with a version of Conrep- might make a quick bootable windows image and try running this: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_c0f92d3d3688417d91f0755b58#tab3 to try and get this working
 
FelixCLC said:
definitely the newer kernel- Haven't done kernel dev in a while, but the guys over at the linux foundation are nothing short of wizards

Also, on topic, I wonder if it's an issue with a version of Conrep- might make a quick bootable windows image and try running this: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_c0f92d3d3688417d91f0755b58#tab3 to try and get this working
Click to expand...
I really hope you can get this working. Nothing I have tried has worked and I do see tons of threads on here regarding the issue, but again my Linux knowledge is very limited.
 
In step 1 of the kernel patching process we install all the following packages:
Code: 
apt-get install git nano screen patch fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev asciidoc xmlto gnupg gnupg2 rsync lintian debhelper libdw-dev libnuma-dev libslang2-dev sphinx-common asciidoc-base automake cpio dh-python file gcc kmod libiberty-dev libpve-common-perl libtool perl-modules python-minimal sed tar zlib1g-dev lz4

Are these all necessary? It seems like a lot of extras installed to the host, when installing extra packages should be kept to a minimum when necessary.
 
Whitterquick said:
In step 1 of the kernel patching process we install all the following packages:
Code: 
apt-get install git nano screen patch fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev asciidoc xmlto gnupg gnupg2 rsync lintian debhelper libdw-dev libnuma-dev libslang2-dev sphinx-common asciidoc-base automake cpio dh-python file gcc kmod libiberty-dev libpve-common-perl libtool perl-modules python-minimal sed tar zlib1g-dev lz4

Are these all necessary? It seems like a lot of extras installed to the host, when installing extra packages should be kept to a minimum when necessary.
Click to expand...
Yeah, I covered this in a thread over on the Level1techs forum (https://forum.level1techs.com/t/lin...ding-over-60gb-and-counting-to-compile/160009)

I'm considering moving to standard debian install with KVM instead of PVE for a related reason. it seems all those packages are needed because of the way PVE likes to build everything itself instead of running a smaller kernel and adding in modules. I'm more of a FreeBSD guy, so the linux world is taking some getting used to, but this seams like a decent way of going about it.

Another consideration is simply to grab the .deb that get compiled and loading them as a kernel module-not sure if it'll work, but of so, would mean we could use this process on pretty much any debian based distribution, including ubuntu, and ideally even Pop_OS!
 
FelixCLC said:
Yeah, I covered this in a thread over on the Level1techs forum (https://forum.level1techs.com/t/lin...ding-over-60gb-and-counting-to-compile/160009)

I'm considering moving to standard debian install with KVM instead of PVE for a related reason. it seems all those packages are needed because of the way PVE likes to build everything itself instead of running a smaller kernel and adding in modules. I'm more of a FreeBSD guy, so the linux world is taking some getting used to, but this seams like a decent way of going about it.

Another consideration is simply to grab the .deb that get compiled and loading them as a kernel module-not sure if it'll work, but of so, would mean we could use this process on pretty much any debian based distribution, including ubuntu, and ideally even Pop_OS!
Click to expand...

This is very interesting...that Level1techs thread is a good read, still going through it. I did consider Debian/KVM/Libvirt but thought it may be too much to learn at this stage, and figured Proxmox would be lighter (maybe not so!) as it wouldn’t have all the default stuff that comes with Debian. Again, I’m coming from Windows so mostly new to all this:)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back