Tutorial:
Well it took a bit more time but here it is, a tutorial to disable the RMRR check. There are other tutorials but those aren't complete, don't work or are for v4.4 and earlier. If you've been affected by RMRR on HP Proliant G7 or earlier, other solutions will not work (excluding RMRR with conrep, acs_override, etc. etc.).
You have probably read this already as well, otherwise you wouldn't be here. ;-)
https://forum.proxmox.com/threads/gpu-passthrough-tutorial-reference.34303/
You should only use this tutorial if you have old hardware that cannot disable RMRR (Reserved Memory Region Reporting). Red Hat has a good whitepaper on the subject, please read it before continuing:
https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf
Anyway, if you're aware of the security risk of accepting RMRR in a virtualized environment, you accept the inherent risk of compiling a Linux kernel with a self-made patch (typos in the patch may kill your system) and still want to use passthrough: here's a how to for Proxmox VE
v5.1 v5.2 v5.3 v5.4 v6.0 v6.2. I'm pretty new to all this stuff (first time compiling ever) so don't expect this to be the prettiest tutorial ever and there's probably a better or quicker way to do things. This however did work for me where all other suggestions failed.
If you're trying to passthrough a GPU but can't get it to work and get a message like this in the GUI:
Code:
vfio: failed to set iommu for container: Operation not permitted
RMRR exclusion might be in effect, you can check this like this:
Code:
dmesg | grep -e DMAR -e IOMMU
If that returns
Code:
vfio-pci 0000:0X:YY.Z: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
RMRR exclusion is indeed in effect and passthrough for that device has been disabled.
Still on board? Okay then, we're going to make a patch to disable the RMRR check in the offending intel-iommu driver. Sidenote: this patch does NOT survive a kernel update. You'll need to patch again using this tutorial (works as long as kernel =
Zesty, Artful Aardvark,
Bionic Beaver, Eoan Ermine).
Also, the required patch has changed since Eoan Ermine because Intel changed their intel-iommu driver. Make sure to use the v2 patch if you are on Eoan.
First, download and install required packages. I'm not sure if you need all of them, but this is what other users used and I've been adding to the list whenever I got an error stating I was missing a package:
Code:
apt-get update
apt-get install git nano screen patch fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev asciidoc xmlto gnupg gnupg2 rsync lintian debhelper libdw-dev libnuma-dev libslang2-dev sphinx-common asciidoc-base automake cpio dh-python file gcc kmod libiberty-dev libpve-common-perl libtool perl-modules python-minimal sed tar zlib1g-dev lz4
Then download the pve-kernel git:
Code:
cd /usr/src/
git clone git://git.proxmox.com/git/pve-kernel.git
This makes a map called pve-kernel in /usr/src/ (the preferred folder for compiling stuff) where all the magic is going to happen.
Next you can either create the patch yourself (option A) or download my variant (option B).
OPTION A:
If you want to build the patch yourself, download the Ubuntu kernel, this is needed to make a patch for the driver:
Code:
cd /usr/src/
git clone git://git.proxmox.com/git/mirror_ubuntu-focal-kernel.git
mv mirror_ubuntu-focal-kernel ubuntu-focal
Then, copy the driver file so we can edit the copy and make a patch out of the differences between the copy and original:
Code:
cp ubuntu-focal/drivers/iommu/intel-iommu.c ubuntu-focal/drivers/iommu/intel-iommu_new.c
And use your preferred editor (mine is Nano) to edit the intel.iommu_new.c driver file:
Code:
nano ubuntu-focal/drivers/iommu/intel-iommu_new.c
Find this section (ctrl-w in Nano):
Code:
if (domain->type == IOMMU_DOMAIN_UNMANAGED &&
device_is_rmrr_locked(dev)) {
dev_warn(dev, "Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.\n");
return -EPERM;
}
Edit out the offending "return -EPERM" line, modify the text for your convenience and close with ctrl-o and ctrl-x, it should look like this:
Code:
if (domain->type == IOMMU_DOMAIN_UNMANAGED &&
device_is_rmrr_locked(dev)) {
dev_warn(dev, "Device was ineligible for IOMMU domain attach due to platform RMRR requirement. Patch is in effect.\n");
}
Then, make a patch file in the pve-kernel directory out of the original and new driver file:
Code:
cd /usr/src/pve-kernel
diff -u /usr/src/ubuntu-focal/drivers/iommu/intel-iommu.c /usr/src/ubuntu-focal/drivers/iommu/intel-iommu_new.c > remove_rmrr_check_v2.patch
Open your new patch file, you need to edit the first two lines to co-align with the necessary filenames and structure when compiling.
Code:
nano remove_rmrr_check.patch
The first two lines should look like this (don't mind the dates, just edit the directories en filenames):
Code:
--- a/drivers/iommu/intel-iommu.c 2019-11-24 17:47:46.031359158 +0200
+++ b/drivers/iommu/intel-iommu.c 2019-11-24 17:50:59.904823799 +0200
Close and save with ctrl-o, ctrl-x (or ctrl-x and y)
Find your newly created (or downloaded) patch file in the pve-kernel directory and place it in the patches directory.
Code:
cd /usr/src/pve-kernel
mv remote_rmrr_check.patch ./patches/kernel
OPTION B
If you want to use my patch, download the attachment (remove_rmrr_check for Disco and earliers, remove_rmrr_check_v2 for Eoan and further), unzip it and place it in the following directory with WinSCP, FTP or another program of choice:
Code:
/usr/src/pve-kernel/patches/kernel/
PROCESS AFTER BOTH OPTION A AND B
There is a script that checks for strange dir names that needs to be disabled in order for us to have a custom kernel version name so we can update systems that are already up to date:
Code:
nano /usr/src/pve-kernel/debian/scripts/find-firmware.pl
Comment out the fourth line from above with a # so that it looks as such:
Code:
#die "strange directory name" if..
Save with ctrl-x and y.
Then edit the control.in file located at /usr/src/pve-kernel so that the generated package is named differently. This prevents errors when updating an up to date vanilla install. Add -removermrr to the pve-kernel article so the line lookes like this:
Code:
pve-kernel-@KVNAME@-removermrr
To finish up and give your system a nice identifier, edit the Makefile:
Code:
nano /usr/src/pve-kernel/Makefile
Edit the EXTRAVERSION line near the top of the Makefile and add this to the end:
Again, save with ctrl-x and y.
Now, you've made a patch to modify the intel-iommu driver and set everything to successfully compile the new kernel. All that's left is to compile! Since the Makefile does all the thinking, you don't need to pass arguments like -j:
Now watch the magic, it will take a while and put quite a load on your system. It took 30 minutes on my HP DL380 G6 with twin X5670's. When it's finished, you should find a couple of new .deb files in your /usr/src/pve-kernel directory. Install them:
The Makefile also updates GRUB and initramfs, so no need to update those manually. When finished, reboot and check again:
Code:
dmesg | grep -e DMAR -e IOMMU
If you still get an error about unsafe interrupts, note that since 6.1 the allow_unsafe_interrupts method has changed:
https://forum.proxmox.com/threads/i...nabled-still-error-message.67341/#post-312870
You should see your newly modified line about RMRR patch being in effect and passthrough should now work. If not, something else might be wrong, you made a typo somewhere or I forgot something (reply below) or this tutorial is outdated already because Proxmox uses a different kernel than
Zesty (updated already to Artful after writing this tutorial, had to rewrite. Thanks to and3l12 for providing info on the new makefile).Bionic already. ;-). And on to Disco. And it's Eoan now. Focal.
Happy compiling and don't forget to like this post if it helped you.
