[TUTORIAL] Compile Proxmox VE with patched intel-iommu driver to remove RMRR check

[Feni]

I came over to this thread because I have seen this on reddit:
https://www.reddit.com/r/VFIO/comments/fovu39/iommu_avic_in_linux_kernel_56_boosts_pci_device/
Because I hardly lack of time and I want to update the kernel of my 6.1.8 proxmox to at least kernel 5.6-rc6, maybe someone can post his build instructions.
Way better interrupt handling sounds verry nice!

Looks nice indeed! Reduced latency is always nice. Unfortunately, AVIC is only available on AMD Zen processors.

 

[efeu]

Build mainline kernel 5.6 with standard ubuntu patches and zfs 0.8.3 built in [-[ with 5.5/5.6 patches (until zfs 0.8.4 is out) ]-]

apt install git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache flex bison  libelf-dev build-essential autoconf libtool gawk alien fakeroot zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libdevmapper-dev
mkdir ~/build && cd ~/build
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ubuntu_kernel
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.6/0001-base-packaging.patch
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.6/0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.6/0003-UBUNTU-SAUCE-tools-hv-lsvmbus-add-manual-page.patch
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.6/0004-debian-changelog.patch
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.6/0005-configs-based-on-Ubuntu-5.6.0-6.6.patch
cd ubuntu_kernel/
git checkout tags/v5.6
cd ..
git clone https://github.com/zfsonlinux/zfs.git
cd zfs
git checkout tags/zfs-0.8.3
cd ../ubuntu_kernel/
patch -p1 < ../0001-base-packaging.patch
patch -p1 < ../0001-base-packaging.patch
patch -p1 < ../0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch
patch -p1 < ../0003
patch -p1 < ../0003-UBUNTU-SAUCE-tools-hv-lsvmbus-add-manual-page.patch
patch -p1 < ../0004-debian-changelog.patch
patch -p1 < ../0005-configs-based-on-Ubuntu-5.6.0-6.6.patch
cp /boot/config-"$(uname -r)" .config
yes '' | make oldconfig
make prepare scripts
cd ../
wget https://gist.githubusercontent.com/satmandu/67cbae9c4d461be0e64428a1707aef1c/raw/ba0fb65f17ccce5b710e4ce86a095de577f7dfe1/k5.6.3.patch
cd zfs
patch -p1 < ../k5.6.3.patch
sh autogen.sh
./configure --prefix=/ --libdir=/lib --includedir=/usr/include --datarootdir=/usr/share --enable-linux-builtin=yes --with-linux=$HOME/build/ubuntu_kernel --with-linux-obj=$HOME/build/ubuntu_kernel
./copy-builtin $HOME/build/ubuntu_kernel
make -j $(nproc)
make install
cd ../ubuntu_kernel
make menuconfig
File system -> ZFS filesystem support (check this)   --> save --> exit
make clean
make -j $(nproc) bindeb-pkg LOCALVERSION=-custom
dpkg -i ../linux*.deb

One thing to mention is, that pve apparmor changes are not included, so without reconfiguring (or have a deeper look @) apparmor is not working, and this results in lxc containers are not starting unless you add

lxc.apparmor.profile = unconfined

to the configuration - for me this is OK, because this is a test environment.

Then add following to your <VMID>.conf file

args: -machine type=q35,kernel_irqchip=on -cpu host,invtsc=on,topoext=on,monitor=off,hv-time,kvm-pv-eoi=on,hv-relaxed,hv-vapic,hv-vpindex,hv-vendor-id=proxmox,hv-crash,kvm=off,kvm-hint-dedicated=on,host-cache-info=on,l3-cache=off

Measured changes by latencymon:
interrupt to process latency (µs): dropped from ~20 to ~5
interrupt to DPC latency (µs): dropped from ~9 to ~2,8

 

[rafalz]

Hi guys. It's doesn't work for me:

HP Proliant d380 g7, + two PCI intel ethernet cards (igb, e1000e)

root@pve-netland:~# uname -a
Linux pve-netland 5.4.34-1-pve-removermrr #1 SMP PVE 5.4.34-2 (Thu, 07 May 2020 10:02:02 +0200) x86_64 GNU/Linux

root@pve-netland:~# dmesg | grep DMAR
[ 0.008754] ACPI: DMAR 0x00000000DF62FE80 00017C (v01 HP ProLiant 00000001 \xd2? 0000162E)
[ 0.125354] DMAR: IOMMU enabled
[ 0.289031] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 1.336971] DMAR: Host address width 39
[ 1.337024] DMAR: DRHD base: 0x000000e7ffe000 flags: 0x1
[ 1.337186] DMAR: dmar0: reg_base_addr e7ffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[ 1.337259] DMAR: RMRR base: 0x000000df7fc000 end: 0x000000df7fdfff
[ 1.337316] DMAR: RMRR base: 0x000000df7f5000 end: 0x000000df7fafff
[ 1.337373] DMAR: RMRR base: 0x000000df63e000 end: 0x000000df63ffff
[ 1.337429] DMAR: ATSR flags: 0x0
[ 1.337705] DMAR: dmar0: Using Queued invalidation
[ 1.345992] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 1.569417] ata_piix 0000:00:1f.2: DMAR: 32bit DMA uses non-identity mapping
[ 1.572810] ehci-pci 0000:00:1d.7: DMAR: Setting identity map [0xdf7fc000 - 0xdf7fdfff]
...

03:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Express Gigabit Server Adapter [103c:7055]
Kernel driver in use: bnx2
Kernel modules: bnx2
03:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Express Gigabit Server Adapter [103c:7055]
Kernel driver in use: bnx2
Kernel modules: bnx2
04:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Express Gigabit Server Adapter [103c:7055]
Kernel driver in use: bnx2
Kernel modules: bnx2
04:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Express Gigabit Server Adapter [103c:7055]
Kernel driver in use: bnx2
Kernel modules: bnx2
0b:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter [8086:115e]
Kernel driver in use: vfio-pci
Kernel modules: e1000e
0b:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter [8086:115e]
Kernel driver in use: e1000e
Kernel modules: e1000e
0e:00.0 Ethernet controller [0200]: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7] (rev 02)
Subsystem: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7]
Kernel driver in use: vfio-pci
Kernel modules: igb
0e:00.1 Ethernet controller [0200]: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7] (rev 02)
Subsystem: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7]
Kernel driver in use: igb
Kernel modules: igb

root@pve-netland:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=8086:105e,8086:115e,8086:10a7
#options e1000e max_vfs=4
options igb max_vfs=4
#options bnx2 max_vfs=4

root@pve-netland:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

but still when i try run vm:

Task viewer: VM 510 - Start
OutputStatus
Stop
kvm: -device vfio-pci,host=0000:0e:00.0,id=hostpci0,bus=pci.0,addr=0x10,rombar=0: vfio 0000:0e:00.0: failed to setup container for group 21: Failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: QEMU exited with code 1

and in dmesg:

...
[ 623.934099] fwbr510i0: port 2(tap510i0) entered blocking state
[ 623.940419] fwbr510i0: port 2(tap510i0) entered forwarding state
[ 624.006923] vfio-pci 0000:0e:00.0: DMAR: Device was ineligible for IOMMU domain attach due to platform RMRR requirement. Patch is in effect.
[ 624.020278] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 624.421630] fwbr510i0: port 2(tap510i0) entered disabled state
[ 624.445985] fwbr510i0: port 1(fwln510i0) entered disabled state
...

Got the same in booth intel cards.
any ideas how to solve that?

 

[rafalz]

Ok. i finally solve my issue.

In booth compile cases all vfio drivers are compiled as built in kernel [*], that is wrong and that's why 'allow_unsafe_interrupts' option does not work... I've just change device drivers -> VFIO Non-Privileged userspace driver framework, to compile as a module <M> inside ubuntu kernel sources dir (by make menuconfig) and CONFIG_VFIO=m, CONFIG_VFIO_IOMMU_TYPE1=m inside oldconfig file which is automatly copied to build directory when make is running. Also removed all stuff from /etc/modprobe.d/vfio.conf and stuff like blacklist e1000e,igb etc from blacklist file- it's not neccesary. Right now everything works on 5.4.34-1 and 5.6.0 kernels ofc with modified intel-iommu.c file in booth cases. Btw the secound integrated Broadcom card - which is not used by pve - works independently as well.

Cheers

 

[jrtz4]

Trying this out on my DL370 G6 but I'm getting this error while trying to build:

make[1]: Entering directory '/usr/src/pve-kernel/build'
debian/rules:263: warning: overriding recipe for target '-removermrr'
debian/rules:248: warning: ignoring old recipe for target '-removermrr'
sed -e 's/@@KVNAME@@/5.4.44-2-pve -removermrr/g' < debian/pve-kernel.prerm.in > debian/pve-kernel-5.4.44-2-pve -removermrr.prerm
sed: -e expression #2, char 1: unknown command: `m'
make[1]: *** [debian/rules:82: debian/control] Error 1
make[1]: Leaving directory '/usr/src/pve-kernel/build'
make: *** [Makefile:78: debian.prepared] Error 2

 

[Whitterquick]

Trying this out on my DL370 G6 but I'm getting this error while trying to build:

make[1]: Entering directory '/usr/src/pve-kernel/build'
debian/rules:263: warning: overriding recipe for target '-removermrr'
debian/rules:248: warning: ignoring old recipe for target '-removermrr'
sed -e 's/@@KVNAME@@/5.4.44-2-pve -removermrr/g' < debian/pve-kernel.prerm.in > debian/pve-kernel-5.4.44-2-pve -removermrr.prerm
sed: -e expression #2, char 1: unknown command: `m'
make[1]: *** [debian/rules:82: debian/control] Error 1
make[1]: Leaving directory '/usr/src/pve-kernel/build'
make: *** [Makefile:78: debian.prepared] Error 2

When you edit the EXTRAVERSION line near the top of the makefile there should not be any spaces:

EXTRAVERSION=-${KREL}-pve-removermrr

 

[Whitterquick]

Has anyone got this working on a HP MicroServer Gen8 with an LSI HBA card?

 

[pr0xnmoz]

I was compiling and hit this error and looking for guidance.
I am using a full newly installed debian system vm with proxmox installed and running, and compiling using option B downloading the ready made attachment.
Vm is fresh installed today just for this purpose.

find-firmware.pl is modified by commenting out and I also tried removing the entire line with no difference

.....
tools/objtool/arch/x86/decode.o
tools/objtool/arch/x86/objtool-in.o
tools/objtool/archwith/x86/lib/with
tools/objtool/arch/x86/lib/inat-tables.c

sent 8,317,516 bytes  received 169,924 bytes  5,658,293.33 bytes/sec
total size is 7,719,108  speedup is 0.91
rm -rf /usr/src/pve-kernel/build/ubuntu-focal_tmp
touch .headers_compile_mark
cp ubuntu-focal/include/generated/compile.h debian/pve-headers-5.4.44-2-pve-removermrr/usr/src/linux-headers-5.4.44-2-pve-removermrr/include/generated/compile.h
install -m 0644 ubuntu-focal/Module.symvers debian/pve-headers-5.4.44-2-pve-removermrr/usr/src/linux-headers-5.4.44-2-pve-removermrr
mkdir -p debian/pve-headers-5.4.44-2-pve-removermrr/lib/modules/5.4.44-2-pve-removermrr
ln -sf /usr/src/linux-headers-5.4.44-2-pve-removermrr debian/pve-headers-5.4.44-2-pve-removermrr/lib/modulewiths/5.4.44-2-pve-removermrr/build
touch .headers_install_mark
dh_installdocs -A debian/copyright debian/SOURCE
dh_installchangelogs
dh_installman
dh_strip_nondeterminism
dh_compress
dh_fixperms
debian/rules fwcheck abicheck
make[2]: Entering
tools/objtool/arch/x86/decode.o
tools/objtool/arch/x86/objtool-in.o
tools/objtool/arch/x86/lib/
tools/objtool/arch/x86/lib/inat-tables.c

sent 8,317,516 bytes  received 169,924 bytes  5,658,293.33 bytes/sec
total size is 7,719,108  speedup is 0.91
rm -rf /usr/src/pve-kernel/build/ubuntu-focal_tmp
touch .headers_compile_mark
cp ubuntu-focal/include/generated/compile.h debian/pve-headers-5.4.44-2-pve-removermrr/usr/src/linux-headers-5.4.44-2-pve-removermrr/include/generated/compile.h
install -m 0644 ubuntu-focal/Module.symvers debian/pve-headers-5.4.44-2-pve-removermrr/usr/src/linux-headers-5.4.44-2-pve-removermrr
mkdir -p debian/pve-headers-5.4.44-2-pve-removermrr/lib/modules/5.4.44-2-pve-removermrr
ln -sf /usr/src/linux-headers-5.4.44-2-pve-removermrr debian/pve-headers-5.4.44-2-pve-removermrr/lib/mwithodules/5.4.44-2-pve-removermrr/build
touch .headers_install_mark
dh_installdocs -A debian/copyright debian/SOURCE
dh_installchangelogs
dh_installman
dh_strip_nondeterminism
dh_compress
dh_fixperms
debian/rules fwcheck abicheck
make[2]: Entering directory '/usr/src/pve-kernel/build'
make[2]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
debian/scripts/find-firmware.pl debian/pve-kernel-5.4.44-2-pve-removermrr/lib/modules/5.4.44-2-pve-removermrr >fwlist.tmp
debian/scripts/find-firmware.pl: 4: debian/scripts/find-firmware.pl: use: not found
debian/scripts/find-firmware.pl: 6: debian/scripts/find-firmware.pl: my: not found
debian/scripts/find-firmware.pl: 8: debian/scripts/find-firmware.pl: die: not found
debian/scripts/find-firmware.pl: 10: debian/scripts/find-firmware.pl: die: not found
debian/scripts/find-firmware.pl: 14: debian/scripts/find-firmware.pl: my: not found
debian/scripts/find-firmware.pl: 16: debian/scripts/find-firmware.pl: Syntax error: word unexpected (expecting ")")
make[2]: *** [debian/rules:248: fwlist-5.4.44-2-pve-removermrr] Error 2
make[2]: Leaving directory '/usr/src/pve-kernel/build'
make[1]: *** [debian/rules:103: binary] Error 2
make[1]: Leaving directory '/usr/src/pve-kernel/build'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make: *** [Makefile:59: pve-kernel-5.4.44-2-pve-removermrr_5.4.44-2_amd64.deb] Error 2
root@debian-src:/usr/src/pve-kernel# 

directory '/usr/src/pve-kernel/build'
make[2]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
debian/scripts/find-firmware.pl debian/pve-kernel-5.4.44-2-pve-removermrr/lib/modules/5.4.44-2-pve-removermrr >fwlist.tmp
debian/scripts/find-firmware.pl: 4: debian/scripts/find-firmware.pl: use: not found
debian/scripts/find-firmware.pl: 6: debian/scripts/find-firmware.pl: my: not found
debian/scripts/find-firmware.pl: 8: debian/scripts/find-firmware.pl: die: not found
debian/scripts/find-firmware.pl: 10: debian/scripts/find-firmware.pl: die: not found
debian/scripts/find-firmwarmoree.pl: 14: debian/scripts/find-firmware.pl: my: not found
debian/scripts/find-firmware.pl: 16: debian/scripts/find-firmware.pl: Syntax error: word unexpected (expecting ")")
make[2]: *** [debian/rules:248: fwlist-5.4.44-2-pve-removermrr] Error 2
make[2]: Leaving directory '/usr/src/pve-kernel/build'
make[1]: *** [debian/rules:103: binary] Error 2
make[1]: Leaving directory '/usr/src/pve-kernel/build'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make: *** [Makefile:59: pve-kernel-5.4.44-2-pve-removermrr_5.4.44-2_amd64.deb] Error 2
root@debian-src:/usr/src/pve-kernel#

 

[Whitterquick]

Ok. i finally solve my issue.

In booth compile cases all vfio drivers are compiled as built in kernel [*], that is wrong and that's why 'allow_unsafe_interrupts' option does not work... I've just change device drivers -> VFIO Non-Privileged userspace driver framework, to compile as a module <M> inside ubuntu kernel sources dir (by make menuconfig) and CONFIG_VFIO=m, CONFIG_VFIO_IOMMU_TYPE1=m inside oldconfig file which is automatly copied to build directory when make is running. Also removed all stuff from /etc/modprobe.d/vfio.conf and stuff like blacklist e1000e,igb etc from blacklist file- it's not neccesary. Right now everything works on 5.4.34-1 and 5.6.0 kernels ofc with modified intel-iommu.c file in booth cases. Btw the secound integrated Broadcom card - which is not used by pve - works independently as well.

Cheers

Can we get this in laymans terms? Step by step? It sounds like what you are saying makes sense but I dont know how or where to start

 

[jrtz4]

When you edit the EXTRAVERSION line near the top of the makefile there should not be any spaces:

EXTRAVERSION=-${KREL}-pve-removermrr

Thanks a lot, that solved it, would be nice to have that in the docs for anyone who hasn't compiled linux and would assume it's an argument.

 

[Whitterquick]

Thanks a lot, that solved it, would be nice to have that in the docs for anyone who hasn't compiled linux and would assume it's an argument.

Yes, I was in the same boat. Iam trying to get this working for MicroServer Gen8 without much luck...

 

[pr0xnmoz]

edit:
Now I see HD space(root) is full...I don't understand the way proxmox handles lvm..help?

Spoiler: screenshot

Today I compiled again, but new install of proxmox on bare metal instead of vm, hba card not plugged in, all disks removed.
Install options:
filesystem [xfs]
hdsize [default]
swapsize [0]
maxroot [85]
maxvz [0]
minfree [default]

This time I followed OPTION A:
git repo: instead of eoan I cloned focal repo.

cd /usr/src/
git clone git://git.proxmox.com/git/mirror_ubuntu-focal-kernel.git
cp -r mirror_ubuntu-focal-kernel ubuntu-focal
rm -rf mirror_ubuntu-focal-kernel

cp ubuntu-focal/drivers/iommu/intel-iommu.c ubuntu-focal/drivers/iommu/intel-iommu_new.c

nano ubuntu-focal/drivers/iommu/intel-iommu_new.c

cd /usr/src/pve-kernel
diff -u /usr/src/ubuntu-focal/drivers/iommu/intel-iommu.c /usr/src/ubuntu-focal/drivers/iommu/intel-iommu_new.c > remove_rmrr_check_v2.patch

nano remove_rmrr_check_v2.patch

--- a/drivers/iommu/intel-iommu.c    2020-08-06 17:43:11.202523978 +0200
+++ b/drivers/iommu/intel-iommu.c    2020-08-06 17:47:03.000000000 +0200

spelling error found in file name "remote_rmrr_check.patch"

cd /usr/src/pve-kernel
mv remote_rmrr_check.patch ./patches/kernel
 
Corrected file name

cd /usr/src/pve-kernel
mv remove_rmrr_check_v2.patch ./patches/kernel

nano /usr/src/pve-kernel/debian/scripts/find-firmware.pl

#!/usr/bin/perl -w

use strict;

my $dir = shift;

die "no directory to scan" if !$dir;

die "no such directory" if ! -d $dir;

#die "strange directory name: $dir" if $dir !~ m|^(.*/)?(5.\d.\d+\-\d+\-pve)(/+)?$|;

my $apiver = $2;

open(TMP, "find '$dir' -name '*.ko'|");
while (defined(my $fn = <TMP>)) {
    chomp $fn;
    my $relfn = $fn;
    $relfn =~ s|^$dir/*||;

    my $cmd = "/sbin/modinfo -F firmware '$fn'";
    open(MOD, "$cmd|");
    while (defined(my $fw = <MOD>)) {
    chomp $fw;
    print "$fw $relfn\n";
    }
    close(MOD);

}
close TMP;

exit 0;

nano /usr/src/pve-kernel/Makefile

# also bump pve-kernel-meta if either of MAJ.MIN, PATCHLEVEL or KREL change
KERNEL_MAJ=5
KERNEL_MIN=4
KERNEL_PATCHLEVEL=44
# increment KREL if the ABI changes (abicheck target in debian/rules)
# rebuild packages with new KREL and run 'make abiupdate'
KREL=2

PKGREL=2

KERNEL_MAJMIN=$(KERNEL_MAJ).$(KERNEL_MIN)
KERNEL_VER=$(KERNEL_MAJMIN).$(KERNEL_PATCHLEVEL)

EXTRAVERSION=-${KREL}-pve-removermrr
KVNAME=${KERNEL_VER}${EXTRAVERSION}
PACKAGE=pve-kernel-${KVNAME}
HDRPACKAGE=pve-headers-${KVNAME}

ARCH=$(shell dpkg-architecture -qDEB_BUILD_ARCH)

# amd64/x86_64/x86 share the arch subdirectory in the kernel, 'x86' so we need
# a mapping
KERNEL_ARCH=x86
ifneq (${ARCH}, amd64)
KERNEL_ARCH=${ARCH}
endif

GITVERSION:=$(shell git rev-parse HEAD)

SKIPABI=0

BUILD_DIR=build

KERNEL_SRC=ubuntu-focal
KERNEL_SRC_SUBMODULE=submodules/$(KERNEL_SRC)
KERNEL_CFG_ORG=config-${KERNEL_VER}.org

ZFSONLINUX_SUBMODULE=submodules/zfsonlinux
ZFSDIR=pkg-zfs

MODULES=modules
MODULE_DIRS=${ZFSDIR}

# exported to debian/rules via debian/rules.d/dirs.mk
DIRS=KERNEL_SRC ZFSDIR MODULES

DST_DEB=${PACKAGE}_${KERNEL_VER}-${PKGREL}_${ARCH}.deb
HDR_DEB=${HDRPACKAGE}_${KERNEL_VER}-${PKGREL}_${ARCH}.deb
LINUX_TOOLS_DEB=linux-tools-$(KERNEL_MAJMIN)_${KERNEL_VER}-${PKGREL}_${ARCH}.deb

DEBS=${DST_DEB} ${HDR_DEB} ${LINUX_TOOLS_DEB}

all: deb
deb: ${DEBS}

${LINUX_TOOLS_DEB} ${HDR_DEB}: ${DST_DEB}
${DST_DEB}: ${BUILD_DIR}.prepared
    cd ${BUILD_DIR}; dpkg-buildpackage --jobs=auto -b -uc -us
    lintian ${DST_DEB}
    #lintian ${HDR_DEB}
    lintian ${LINUX_TOOLS_DEB}

${BUILD_DIR}.prepared: $(addsuffix .prepared,${KERNEL_SRC} ${MODULES} debian)
    cp -a fwlist-previous ${BUILD_DIR}/
    cp -a abi-prev-* ${BUILD_DIR}/
    cp -a abi-blacklist ${BUILD_DIR}/
    touch $@

debian.prepared: debian
    rm -rf ${BUILD_DIR}/debian
    mkdir -p ${BUILD_DIR}
    cp -a debian ${BUILD_DIR}/debian
    echo "git clone git://git.proxmox.com/git/pve-kernel.git\\ngit checkout ${GITVERSION}" > ${BUILD_DIR}/debian/SOURCE
    @$(foreach dir, ${DIRS},echo "${dir}=${${dir}}" >> ${BUILD_DIR}/debian/rules.d/env.mk;)
    echo "KVNAME=${KVNAME}" >> ${BUILD_DIR}/debian/rules.d/env.mk
    echo "KERNEL_MAJMIN=${KERNEL_MAJMIN}" >> ${BUILD_DIR}/debian/rules.d/env.mk
    cd ${BUILD_DIR}; debian/rules debian/control
    touch $@

${KERNEL_SRC}.prepared: ${KERNEL_SRC_SUBMODULE} | submodule
    rm -rf ${BUILD_DIR}/${KERNEL_SRC} $@
    mkdir -p ${BUILD_DIR}
    cp -a ${KERNEL_SRC_SUBMODULE} ${BUILD_DIR}/${KERNEL_SRC}
# TODO: split for archs, track and diff in our repository?
    cat ${BUILD_DIR}/${KERNEL_SRC}/debian.master/config/config.common.ubuntu ${BUILD_DIR}/${KERNEL_SRC}/debian.master/config/${ARCH}/config.common.${ARCH} ${BUILD_DIR}/${KERNEL_SRC}/debian.master/config/${ARCH}/config.flavour.generic > ${KERNEL_CFG_ORG}
    cp ${KERNEL_CFG_ORG} ${BUILD_DIR}/${KERNEL_SRC}/.config
    sed -i ${BUILD_DIR}/${KERNEL_SRC}/Makefile -e 's/^EXTRAVERSION.*$$/EXTRAVERSION=${EXTRAVERSION}/'
    rm -rf ${BUILD_DIR}/${KERNEL_SRC}/debian ${BUILD_DIR}/${KERNEL_SRC}/debian.master
    set -e; cd ${BUILD_DIR}/${KERNEL_SRC}; for patch in ../../patches/kernel/*.patch; do echo "applying patch '$$patch'" && patch -p1 < $${patch}; done
    touch $@

${MODULES}.prepared: $(addsuffix .prepared,${MODULE_DIRS})
    touch $@

${ZFSDIR}.prepared: ${ZFSONLINUX_SUBMODULE}
    rm -rf ${BUILD_DIR}/${MODULES}/${ZFSDIR} ${BUILD_DIR}/${MODULES}/tmp $@
    mkdir -p ${BUILD_DIR}/${MODULES}/tmp
    cp -a ${ZFSONLINUX_SUBMODULE}/* ${BUILD_DIR}/${MODULES}/tmp
    cd ${BUILD_DIR}/${MODULES}/tmp; make kernel
    rm -rf ${BUILD_DIR}/${MODULES}/tmp
    touch ${ZFSDIR}.prepared

.PHONY: upload
upload: ${DEBS}
    tar cf - ${DEBS}|ssh -X repoman@repo.proxmox.com -- upload --product pve,pmg --dist buster --arch ${ARCH}

.PHONY: distclean
distclean: clean
    git submodule deinit --all

# upgrade to current master
.PHONY: update_modules
update_modules: submodule
    git submodule foreach 'git pull --ff-only origin master'
    cd ${ZFSONLINUX_SUBMODULE}; git pull --ff-only origin master

# make sure submodules were initialized
.PHONY: submodule
submodule:
    test -f "${KERNEL_SRC_SUBMODULE}/README" || git submodule update --init ${KERNEL_SRC_SUBMODULE}
    test -f "${ZFSONLINUX_SUBMODULE}/Makefile" || git submodule update --init --recursive ${ZFSONLINUX_SUBMODULE}

# call after ABI bump with header deb in working directory
.PHONY: abiupdate
abiupdate: abi-prev-${KVNAME}
abi-prev-${KVNAME}: abi-tmp-${KVNAME}
ifneq ($(strip $(shell git status --untracked-files=no --porcelain -z)),)
    @echo "working directory unclean, aborting!"
    @false
else
    git rm "abi-prev-*"
    mv $< $@
    git add $@
    git commit -s -m "update ABI file for ${KVNAME}" -m "(generated with debian/scripts/abi-generate)"
    @echo "update abi-prev-${KVNAME} committed!"
endif

abi-tmp-${KVNAME}:
    @ test -e ${HDR_DEB} || (echo "need ${HDR_DEB} to extract ABI data!" && false)
    debian/scripts/abi-generate ${HDR_DEB} $@ ${KVNAME} 1

.PHONY: clean
clean:
    rm -rf *~ build *.prepared ${KERNEL_CFG_ORG}
    rm -f *.deb *.changes *.buildinfo

And this is the result down below

root@pve:/usr/src/pve-kernel# make
............
  LD [M]  drivers/net/wireless/intersil/hostap/hostap.ko
  LD [M]  drivers/net/wireless/intersil/hostap/hostap_cs.ko
  LD [M]  drivers/net/wireless/intersil/hostap/hostap_pci.ko
  LD [M]  drivers/net/wireless/intersil/hostap/hostap_plx.ko
ldld: : final link failed: drivers/net/wireless/intersil/hostap/hostap_pci.ko: final close failed: No space left on deviceNo space left on device

make[4]: *** [scripts/Makefile.modfinal:41: drivers/net/wireless/intersil/hostap/hostap_pci.ko] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: *** [scripts/Makefile.modfinal:41: drivers/net/wireless/intel/iwlwifi/mvm/iwlmvm.ko] Error 1
ld: final link failed: No space left on device
make[4]: *** [scripts/Makefile.modfinal:41: drivers/net/wireless/intel/iwlwifi/dvm/iwldvm.ko] Error 1
/bin/sh: 1: cannot create drivers/net/wireless/intersil/hostap/.hostap_cs.ko.cmd: No space left on device
make[4]: *** [scripts/Makefile.modfinal:41: drivers/net/wireless/intersil/hostap/hostap_cs.ko] Error 2
make[4]: *** Deleting file 'drivers/net/wireless/intersil/hostap/hostap_cs.ko'
ld: final link failed: No space left on device
make[4]: *** [scripts/Makefile.modfinal:41: drivers/net/wireless/intel/iwlwifi/iwlwifi.ko] Error 1
ld: final link failed: No space left on device
make[4]: *** [scripts/Makefile.modfinal:41: drivers/net/wireless/intersil/hostap/hostap.ko] Error 1
make[3]: *** [scripts/Makefile.modpost:95: __modpost] Error 2
make[2]: *** [Makefile:1321: modules] Error 2
make[2]: Leaving directory '/usr/src/pve-kernel/build/ubuntu-focal'
make[1]: *** [debian/rules:118: .compile_mark] Error 2
make[1]: Leaving directory '/usr/src/pve-kernel/build'
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
make: *** [Makefile:58: pve-kernel-5.4.44-2-pve-removermrr_5.4.44-2_amd64.deb] Error 2
root@pve:/usr/src/pve-kernel# pvdisplay
  /dev/sda: open failed: No medium found
  --- Physical volume ---
  PV Name               /dev/sdb3
  VG Name               pve
  PV Size               <111.29 GiB / not usable 4.44 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              28489
  Free PE               21385
  Allocated PE          7104
  PV UUID               [REDACTED]

root@pve:/usr/src/pve-kernel# pvs
  /dev/sda: open failed: No medium found
  PV         VG  Fmt  Attr PSize    PFree
  /dev/sdb3  pve lvm2 a--  <111.29g <83.54g
root@pve:/usr/src/pve-kernel# pvs -o name,free --units g --noheadings
  /dev/sda: open failed: No medium found
  /dev/sdb3  83.54g
root@pve:/usr/src/pve-kernel#

 

[Whitterquick]

Tutorial:

Well it took a bit more time but here it is, a tutorial to disable the RMRR check. There are other tutorials but those aren't complete, don't work or are for v4.4 and earlier. If you've been affected by RMRR on HP Proliant G7 or earlier, other solutions will not work (excluding RMRR with conrep, acs_override, etc. etc.).

You have probably read this already as well, otherwise you wouldn't be here. ;-)
https://forum.proxmox.com/threads/gpu-passthrough-tutorial-reference.34303/

You should only use this tutorial if you have old hardware that cannot disable RMRR (Reserved Memory Region Reporting). Red Hat has a good whitepaper on the subject, please read it before continuing: https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf

Anyway, if you're aware of the security risk of accepting RMRR in a virtualized environment, you accept the inherent risk of compiling a Linux kernel with a self-made patch (typos in the patch may kill your system) and still want to use passthrough: here's a how to for Proxmox VE v5.1 v5.2 v5.3 v5.4 v6.0 v6.2. I'm pretty new to all this stuff (first time compiling ever) so don't expect this to be the prettiest tutorial ever and there's probably a better or quicker way to do things. This however did work for me where all other suggestions failed.

If you're trying to passthrough a GPU but can't get it to work and get a message like this in the GUI:

vfio: failed to set iommu for container: Operation not permitted

RMRR exclusion might be in effect, you can check this like this:

dmesg | grep -e DMAR -e IOMMU

If that returns

vfio-pci 0000:0X:YY.Z: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

RMRR exclusion is indeed in effect and passthrough for that device has been disabled.

Still on board? Okay then, we're going to make a patch to disable the RMRR check in the offending intel-iommu driver. Sidenote: this patch does NOT survive a kernel update. You'll need to patch again using this tutorial (works as long as kernel = Zesty, Artful Aardvark, Bionic Beaver, Eoan Ermine).

Also, the required patch has changed since Eoan Ermine because Intel changed their intel-iommu driver. Make sure to use the v2 patch if you are on Eoan.

First, download and install required packages. I'm not sure if you need all of them, but this is what other users used and I've been adding to the list whenever I got an error stating I was missing a package:

apt-get update
apt-get install git nano screen patch fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev asciidoc xmlto gnupg gnupg2 rsync lintian debhelper libdw-dev libnuma-dev libslang2-dev sphinx-common asciidoc-base automake cpio dh-python file gcc kmod libiberty-dev libpve-common-perl libtool perl-modules python-minimal sed tar zlib1g-dev lz4

Then download the pve-kernel git:

cd /usr/src/
git clone git://git.proxmox.com/git/pve-kernel.git

This makes a map called pve-kernel in /usr/src/ (the preferred folder for compiling stuff) where all the magic is going to happen.

Next you can either create the patch yourself (option A) or download my variant (option B).

OPTION A:
If you want to build the patch yourself, download the Ubuntu kernel, this is needed to make a patch for the driver:

cd /usr/src/
git clone git://git.proxmox.com/git/mirror_ubuntu-eoan-kernel.git
cp -r mirror_ubuntu-eoan-kernel ubuntu-eoan
rm -rf mirror_ubuntu-eoan-kernel

Then, copy the driver file so we can edit the copy and make a patch out of the differences between the copy and original:

cp ubuntu-eoan/drivers/iommu/intel-iommu.c ubuntu-eoan/drivers/iommu/intel-iommu_new.c

And use your preferred editor (mine is Nano) to edit the intel.iommu_new.c driver file:

nano ubuntu-eoan/drivers/iommu/intel-iommu_new.c

Find this section (ctrl-w in Nano):

        if (domain->type == IOMMU_DOMAIN_UNMANAGED &&
            device_is_rmrr_locked(dev)) {
                dev_warn(dev, "Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.\n");
                return -EPERM;

       }

Edit out the offending "return -EPERM" line, modify the text for your convenience and close with ctrl-o and ctrl-x, it should look like this:

        if (domain->type == IOMMU_DOMAIN_UNMANAGED &&
            device_is_rmrr_locked(dev)) {
                dev_warn(dev, "Device was ineligible for IOMMU domain attach due to platform RMRR requirement. Patch is in effect.\n");
        }

Then, make a patch file in the pve-kernel directory out of the original and new driver file:

cd /usr/src/pve-kernel
diff -u /usr/src/ubuntu-eoan/drivers/iommu/intel-iommu.c /usr/src/ubuntu-eoan/drivers/iommu/intel-iommu_new.c > remove_rmrr_check_v2.patch

Open your new patch file, you need to edit the first two lines to co-align with the necessary filenames and structure when compiling.

nano remove_rmrr_check.patch

The first two lines should look like this (don't mind the dates, just edit the directories en filenames):

--- a/drivers/iommu/intel-iommu.c   2019-11-24 17:47:46.031359158 +0200
+++ b/drivers/iommu/intel-iommu.c       2019-11-24 17:50:59.904823799 +0200

Close and save with ctrl-o, ctrl-x (or ctrl-x and y)

Find your newly created (or downloaded) patch file in the pve-kernel directory and place it in the patches directory.

cd /usr/src/pve-kernel
mv remote_rmrr_check.patch ./patches/kernel

OPTION B
If you want to use my patch, download the attachment (remove_rmrr_check for Disco and earliers, remove_rmrr_check_v2 for Eoan and further), unzip it and place it in the following directory with WinSCP, FTP or another program of choice:

/usr/src/pve-kernel/patches/kernel/

PROCESS AFTER BOTH OPTION A AND B
There is a script that checks for strange dir names that needs to be disabled in order for us to have a custom kernel version name so we can update systems that are already up to date:

nano /usr/src/pve-kernel/debian/scripts/find-firmware.pl

Comment out the fourth line from above with a # so that it looks as such:

#die "strange directory name" if..

Save with ctrl-x and y.

To finish up and give your system a nice identifier, edit the Makefile:

nano /usr/src/pve-kernel/Makefile

Edit the EXTRAVERSION line near the top of the Makefile and add this to the end:

-removermrr

Again, save with ctrl-x and y.

Now, you've made a patch to modify the intel-iommu driver and set everything to successfully compile the new kernel. All that's left is to compile! Since the Makefile does all the thinking, you don't need to pass arguments like -j:

make

Now watch the magic, it will take a while and put quite a load on your system. It took 30 minutes on my HP DL380 G6 with twin X5670's. When it's finished, you should find a couple of new .deb files in your /usr/src/pve-kernel directory. Install them:

dpkg -i *.deb

The Makefile also updates GRUB and initramfs, so no need to update those manually. When finished, reboot and check again:

dmesg | grep -e DMAR -e IOMMU

If you still get an error about unsafe interrupts, note that since 6.1 the allow_unsafe_interrupts method has changed: https://forum.proxmox.com/threads/i...nabled-still-error-message.67341/#post-312870

You should see your newly modified line about RMRR patch being in effect and passthrough should now work. If not, something else might be wrong, you made a typo somewhere or I forgot something (reply below) or this tutorial is outdated already because Proxmox uses a different kernel than Zesty (updated already to Artful after writing this tutorial, had to rewrite. Thanks to and3l12 for providing info on the new makefile).Bionic already. ;-). And on to Disco. And it's Eoan now. Focal.

Happy compiling and don't forget to like this post if it helped you.

Hello Feni,
I have not been able to get this working on MicroServer Gen8. I can see you started this tutorial a long time ago but posted as recently as a couple of months back, so I was wondering if you have got this working on a Gen8? Thanks.

 

[raghu_mp]

I tried this on Microsever Gen8 as well and make failed due to space error at the end. I already have proxmox LVM with about 35G left for /usr/src. Any idea if the compile requires more space?

 

[Whitterquick]

I tried this on Microsever Gen8 as well and make failed due to space error at the end. I already have proxmox LVM with about 35G left for /usr/src. Any idea if the compile requires more space?

You can use a script such as this one To clear the root partition so you have enough space to make. I read everywhere that Proxmox requires as little as 10GB so I was also surprised by that happening.

 

[raghu_mp]

You can use a script such as this one To clear the root partition so you have enough space to make. I read everywhere that Proxmox requires as little as 10GB so I was also surprised by that happening.

Thanks! I have been researching this a lot and found the below.
https://forum.level1techs.com/t/loo...-to-troubleshoot-error-vfio-dma-map-22/153539

It appears even if i could compile the patched kernel successfully, will eventually hit the error mentioned at the link above since latest proxmox is on kernel 5.4. Only option appears to be patched kernel version 5.3.

 

[Whitterquick]

Thanks! I have been researching this a lot and found the below.
https://forum.level1techs.com/t/loo...-to-troubleshoot-error-vfio-dma-map-22/153539

It appears even if i could compile the patched kernel successfully, will eventually hit the error mentioned at the link above since latest proxmox is on kernel 5.4. Only option appears to be patched kernel version 5.3.

This is interesting, I have scoured the internet and had not come across that yet. I may try Kernel 5.3 over the weekend to see if this makes a difference...

 

[Whitterquick]

For the following step...

cd /usr/src/
git clone git://git.proxmox.com/git/pve-kernel.git

How do I clone/download 5.3 instead of the latest?

 

[Whitterquick]

If I download and manually FTP it to the correct path I get the following error:

fatal: not a git repository (or any of the parent directories): .git

 

[Whitterquick]

Nevermind, someone helped me in another thread.