[TUTORIAL] Compile Proxmox VE with patched intel-iommu driver to remove RMRR check

geronimobb

Member
Apr 22, 2017
25
0
6
45
Well before this all worked.

When i try it now i get:
Fetched in submodule path 'submodules/ubuntu-disco', but it did not contain fdc10a3711d4cb8ae2c42efb2c89dcca919d1e4d. Direct fetching of that commit failed.
Makefile:120: recipe for target 'submodule' failed
 

WolfpacK

New Member
Nov 11, 2018
9
0
1
38
Rather than cloning the entire repo, you can use: git clone --depth=1 git://git.proxmox.com/git/mirror_ubuntu-disco-kernel.git

to download only the latest branch.

Also rather than copying, use mv

so:

Code:
cd /usr/src/
git clone --depth=1 git://git.proxmox.com/git/mirror_ubuntu-disco-kernel.git
mv mirror_ubuntu-disco-kernel ubuntu-disco
Tutorial:
Code:
cd /usr/src/
git clone git://git.proxmox.com/git/mirror_ubuntu-disco-kernel.git
cp -r mirror_ubuntu-disco-kernel ubuntu-disco
rm -rf mirror_ubuntu-disco-kernel
 

WolfpacK

New Member
Nov 11, 2018
9
0
1
38
I've created a script which will create a package with the current patched kernel. It uses Vagrant to keep things nice and tidy.

It's not finished. So far it will create a vagrant vm and clone the current kernel repo. It also installs the required packages.

It's available on github https://github.com/wolfpackmars2/rm_rmrr

Download rm_rmrr.sh and run it with
Code:
sh rm_rmrr.sh
Once that is complete, run vagrant ssh to log into the VM then sudo su then cd /root/ubuntu_disco. From there, continue after "Then, copy the driver file so we can edit the copy and make a patch out of the differences between the copy and original:" in Feni's tutorial
 

WolfpacK

New Member
Nov 11, 2018
9
0
1
38
Is this the only way this can be accomplished? Rebuilding the entire kernel is time consuming and resource intensive.

Could a driver be created or a kernel module be made and applied to the existing kernel without having to rebuild every time a PVE kernel is released?
 

warnerizo

New Member
Mar 18, 2019
2
1
3
30
Hi, after a time with this kernel patch activate and my VM's running with the pci gpu card on my HP Gen8 microserver, a friend told me that i should look this post:

http://www.jimmdenton.com/proliant-intel-dpdk/

@Feni and everyone maybe this post can help you guys.

I reinstall proxmox and made the steps of the jimmdenton tutorial and to get iommu detecte I using this post https://forum.proxmox.com/threads/gpu-passthrough-tutorial-reference.34303/ and when I boot up proxmox if I run the command I get this output:
Code:
root@pve:~# dmesg | grep -e DMAR -e IOMMU
[    0.009762] ACPI: DMAR 0x00000000E5DE4A80 0003B4 (v01 HP     ProLiant 00000001 \xd2?   0000162E)
[    0.124061] DMAR: IOMMU enabled
[    0.200176] DMAR: Host address width 39
[    0.200177] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.200182] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f010da
[    0.200182] DMAR: RMRR base: 0x000000e5ffd000 end: 0x000000e5ffffff
[    0.200183] DMAR: RMRR base: 0x000000e5ff6000 end: 0x000000e5ffcfff
[    0.200184] DMAR: RMRR base: 0x000000e5f93000 end: 0x000000e5f94fff
[    0.200185] DMAR: RMRR base: 0x000000e5f8f000 end: 0x000000e5f92fff
[    0.200185] DMAR: RMRR base: 0x000000e5f7f000 end: 0x000000e5f8efff
[    0.200186] DMAR: RMRR base: 0x000000e5f7e000 end: 0x000000e5f7efff
[    0.200187] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
[    0.200187] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
[    0.200188] DMAR: RMRR base: 0x000000e5dee000 end: 0x000000e5deefff
[    0.200190] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
[    0.200191] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.200191] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.200192] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.200442] DMAR-IR: Enabled IRQ remapping in xapic mode
[    1.069148] DMAR: No ATSR found
[    1.069339] DMAR: dmar0: Using Queued invalidation
[    1.069348] DMAR: Setting RMRR:
[    1.069580] DMAR: Setting identity map for device 0000:01:00.0 [0xe5dee000 - 0xe5deefff]
[    1.069842] DMAR: Setting identity map for device 0000:01:00.2 [0xe5dee000 - 0xe5deefff]
[    1.070085] DMAR: Setting identity map for device 0000:01:00.4 [0xe5dee000 - 0xe5deefff]
[    1.070352] DMAR: Setting identity map for device 0000:00:1f.2 [0xe8000 - 0xe8fff]
[    1.070604] DMAR: Setting identity map for device 0000:07:00.1 [0xe8000 - 0xe8fff]
[    1.070874] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
[    1.071131] DMAR: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
[    1.071407] DMAR: Setting identity map for device 0000:04:00.0 [0xe8000 - 0xe8fff]
[    1.071416] DMAR: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
[    1.071428] DMAR: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
[    1.071437] DMAR: Setting identity map for device 0000:00:1f.2 [0xf4000 - 0xf4fff]
[    1.071439] DMAR: Setting identity map for device 0000:07:00.1 [0xf4000 - 0xf4fff]
[    1.071440] DMAR: Setting identity map for device 0000:03:00.0 [0xf4000 - 0xf4fff]
[    1.071441] DMAR: Setting identity map for device 0000:03:00.1 [0xf4000 - 0xf4fff]
[    1.071442] DMAR: Setting identity map for device 0000:04:00.0 [0xf4000 - 0xf4fff]
[    1.071443] DMAR: Setting identity map for device 0000:01:00.0 [0xf4000 - 0xf4fff]
[    1.071444] DMAR: Setting identity map for device 0000:01:00.2 [0xf4000 - 0xf4fff]
[    1.071446] DMAR: Setting identity map for device 0000:00:1f.2 [0xe5f7e000 - 0xe5f7efff]
[    1.071454] DMAR: Setting identity map for device 0000:07:00.1 [0xe5f7e000 - 0xe5f7efff]
[    1.071463] DMAR: Setting identity map for device 0000:03:00.0 [0xe5f7e000 - 0xe5f7efff]
[    1.071471] DMAR: Setting identity map for device 0000:03:00.1 [0xe5f7e000 - 0xe5f7efff]
[    1.071481] DMAR: Setting identity map for device 0000:04:00.0 [0xe5f7e000 - 0xe5f7efff]
[    1.071489] DMAR: Setting identity map for device 0000:01:00.0 [0xe5f7e000 - 0xe5f7efff]
[    1.071494] DMAR: Setting identity map for device 0000:01:00.2 [0xe5f7e000 - 0xe5f7efff]
[    1.071499] DMAR: Setting identity map for device 0000:00:1f.2 [0xe5f7f000 - 0xe5f8efff]
[    1.071500] DMAR: Setting identity map for device 0000:07:00.1 [0xe5f7f000 - 0xe5f8efff]
[    1.071502] DMAR: Setting identity map for device 0000:03:00.0 [0xe5f7f000 - 0xe5f8efff]
[    1.071503] DMAR: Setting identity map for device 0000:03:00.1 [0xe5f7f000 - 0xe5f8efff]
[    1.071505] DMAR: Setting identity map for device 0000:04:00.0 [0xe5f7f000 - 0xe5f8efff]
[    1.071506] DMAR: Setting identity map for device 0000:01:00.0 [0xe5f7f000 - 0xe5f8efff]
[    1.071508] DMAR: Setting identity map for device 0000:01:00.2 [0xe5f7f000 - 0xe5f8efff]
[    1.071509] DMAR: Setting identity map for device 0000:00:1f.2 [0xe5f8f000 - 0xe5f92fff]
[    1.071511] DMAR: Setting identity map for device 0000:07:00.1 [0xe5f8f000 - 0xe5f92fff]
[    1.071512] DMAR: Setting identity map for device 0000:03:00.0 [0xe5f8f000 - 0xe5f92fff]
[    1.071513] DMAR: Setting identity map for device 0000:03:00.1 [0xe5f8f000 - 0xe5f92fff]
[    1.071515] DMAR: Setting identity map for device 0000:04:00.0 [0xe5f8f000 - 0xe5f92fff]
[    1.071516] DMAR: Setting identity map for device 0000:01:00.0 [0xe5f8f000 - 0xe5f92fff]
[    1.071517] DMAR: Setting identity map for device 0000:01:00.2 [0xe5f8f000 - 0xe5f92fff]
[    1.071519] DMAR: Setting identity map for device 0000:00:1f.2 [0xe5f93000 - 0xe5f94fff]
[    1.071520] DMAR: Setting identity map for device 0000:07:00.1 [0xe5f93000 - 0xe5f94fff]
[    1.071521] DMAR: Setting identity map for device 0000:03:00.0 [0xe5f93000 - 0xe5f94fff]
[    1.071522] DMAR: Setting identity map for device 0000:03:00.1 [0xe5f93000 - 0xe5f94fff]
[    1.071524] DMAR: Setting identity map for device 0000:04:00.0 [0xe5f93000 - 0xe5f94fff]
[    1.071525] DMAR: Setting identity map for device 0000:01:00.0 [0xe5f93000 - 0xe5f94fff]
[    1.071526] DMAR: Setting identity map for device 0000:01:00.2 [0xe5f93000 - 0xe5f94fff]
[    1.071527] DMAR: Setting identity map for device 0000:01:00.0 [0xe5ff6000 - 0xe5ffcfff]
[    1.071528] DMAR: Setting identity map for device 0000:01:00.2 [0xe5ff6000 - 0xe5ffcfff]
[    1.071530] DMAR: Setting identity map for device 0000:01:00.4 [0xe5ff6000 - 0xe5ffcfff]
[    1.071778] DMAR: Setting identity map for device 0000:00:1a.0 [0xe5ffd000 - 0xe5ffffff]
[    1.072003] DMAR: Setting identity map for device 0000:00:1d.0 [0xe5ffd000 - 0xe5ffffff]
[    1.072014] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.072289] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.072379] DMAR: Intel(R) Virtualization Technology for Directed I/O
It's seems that it's works but when start the VM it's show again the iommu blocked message and if I run the last command the output show it:
Code:
[  203.716252] vfio-pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
I don't know exactly if this can help but I hope to find a easy way to use gpu pci passthrougth to VM's on the HP server.
 
Last edited:

lyter

Member
Oct 15, 2013
16
1
23
I've just tried to follow this great tutorial for Proxmox 6.0, but I get errors when compiling the kernel. Has anyone succeeded with this with Proxmox 6.0 already?
 

tehtafara0

New Member
Oct 9, 2019
1
1
1
18
I have done all the steps with absolute success. Thank you for the amazing tutorial :)

EDIT: I compiled it on an 72-core AWS EC2 instance running ubuntu 18.04
 
Last edited:
  • Like
Reactions: Feni

Feni

Member
Jun 22, 2017
27
10
8
34
I've just tried to follow this great tutorial for Proxmox 6.0, but I get errors when compiling the kernel. Has anyone succeeded with this with Proxmox 6.0 already?
Any luck so far? My system is patched and on 6.0 at the moment using the attached patch. What kind of errors do you get?
 

siratreyu

New Member
Nov 20, 2019
1
0
1
40
Hello. Thanks for this Tutorial.

I get this error:

Code:
BEGIN failed--compilation aborted at (eval 16) line 1.

make[2]: *** [debian/rules:13: clean] Error 2
make[2]: se sale del directorio '/usr/src/pve-kernel/build/modules/tmp/zfs-linux_0.8.2'
dpkg-buildpackage: fallo: debian/rules clean subprocess returned exit status 2
make[1]: *** [Makefile:53: zfs-linux_0.8.2-pve2.dsc] Error 2
make[1]: se sale del directorio '/usr/src/pve-kernel/build/modules/tmp'
Solution???


[SOLVED] -> apt install python-sphinx

Compiling...
 
Last edited:

Feni

Member
Jun 22, 2017
27
10
8
34
Hello. Thanks for this Tutorial.

I get this error:

Code:
BEGIN failed--compilation aborted at (eval 16) line 1.

make[2]: *** [debian/rules:13: clean] Error 2
make[2]: se sale del directorio '/usr/src/pve-kernel/build/modules/tmp/zfs-linux_0.8.2'
dpkg-buildpackage: fallo: debian/rules clean subprocess returned exit status 2
make[1]: *** [Makefile:53: zfs-linux_0.8.2-pve2.dsc] Error 2
make[1]: se sale del directorio '/usr/src/pve-kernel/build/modules/tmp'
Solution???


[SOLVED] -> apt install python-sphinx

Compiling...
Ah yes, dependencies.. :) Happy you found it.
 

xeroiv

New Member
May 19, 2018
4
0
1
36
Thank you for the tutorial, I can get as far as booting my VM with the HBA passed through. However once I boot the VM my proxmox host throws some errors and puts the fans on my DL380e at 100%. My DL380e has a p420 HBA connected to slot 3 that I am trying to pass through to a FreeNAS VM.

Do you have any way to solve the error?

Code:
DMAR: [DMA Read] Request device [0d:00.0] fault addr 7df83000 [fault reason 06] PTE Read access is not set?
 

Feni

Member
Jun 22, 2017
27
10
8
34
Thank you for the tutorial, I can get as far as booting my VM with the HBA passed through. However once I boot the VM my proxmox host throws some errors and puts the fans on my DL380e at 100%. My DL380e has a p420 HBA connected to slot 3 that I am trying to pass through to a FreeNAS VM.

Do you have any way to solve the error?

Code:
DMAR: [DMA Read] Request device [0d:00.0] fault addr 7df83000 [fault reason 06] PTE Read access is not set?
Hi there,

If an HP goes to 100% fan speed, it usually indicates a hardware or firmware issue. A quick search returns this thread with more information:
https://forums.unraid.net/topic/42936-dmarfault-reason-06-pte-read-access-is-not-se/
https://bugzilla.redhat.com/show_bug.cgi?id=1339567
https://bugzilla.kernel.org/show_bug.cgi?id=82761
https://access.redhat.com/solutions/183373

One of the fixes mentioned is:
"This issue was resolved by disabling the "HP Shared Memory features" for the NIC in BIOS." Perhaps you can also do this for the p420 HBA.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!