[TUTORIAL] Compile Proxmox VE with patched intel-iommu driver to remove RMRR check
- Thread starter Feni
- Start date
Oh great!
If I were to run Pop!_OS ! Debian or Ubuntu in a VM, would I be able to use that to compile a Proxmox kernel? (as I type that it sounds like an obvious NO lol); if not, can I run Proxmox in a Proxmox VM for the purpose of this particular task only? I have my VMs on a separate SSD which has more space and less likely to run out of disk space when compiling...
On a 5.4 kernel ?
I don't think there's a link between a passthrough error and zfs
Sorry, not sure I understand your question.
If the question is can you compile a kernel from a VM, then yes absolutely. a Kernel is just a very advanced/complex program. If you wanted to you could compile the same Kernel IBM uses for PowerPC from a raspberry pi etc.
if you're asking can you then USE the patch to passthrough things from within a vm (for nested virtualization) then no. You would apply the patch to the hypervisor (the os on bare metal) then passthrough the hard ware in question to the vm. The vm would then be able to do things as needed. if you wanted to do nested virtualization, you would then wan't to have the host allow the guest to use virtualization instructions. the wiki is your friend: https://pve.proxmox.com/wiki/Nested_Virtualization
Pretty sure he meant to tag the other Alex that popped into this thread within the last week xD
I have tried that already, it doesn’t work. I think the issue is with what has changed/moved around since 5.3 has not been documented by the Proxmox team other than saying some drivers are now built into the kernel. There must be a way of getting it working...
Yes, I meant compiling a patched kernel in a VM to then copy over the .deb files to my main Proxmox host to install/use. Does this RMRR patch need to be created/compiled in Proxmox or can I use any Debian-based distro to do it?
Comments on the ubuntu bug are not very encouraging. The RMRR thing seems to be the right thing to do as a design decision however the HP hardware is not looking to be compatible with it. Until HP changes his mind and modify the BIOS in order to be compatible I think we are out of luck.
It don't think tweaking the kernel by disabling part of it to be a good long term solution.
I think I'll stick with ESXI for the moment as it does not have this issue until I can afford to change my hardware.
It's actually a terrible solution as described in the QEmu bug report you linked (https://bugs.launchpad.net/qemu/+bug/1869006/comments/18). We can only hope that iLO doesn't tinker with these regions and just reads them.... but if it writes them... well, that may lead to a catastrophe if the checks are worked around
This patch works on the Debian iommu driver, it can be created on any debian-based system (although with a different tutorial).
This bugreport is pretty scary for 5.4. So far, I haven't seen instability but I'll monitor for issues.It's actually a terrible solution as described in the QEmu bug report you linked (https://bugs.launchpad.net/qemu/+bug/1869006/comments/18). We can only hope that iLO doesn't tinker with these regions and just reads them.... but if it writes them... well, that may lead to a catastrophe if the checks are worked around
The thing is it was always this way. I looked around the changes in 5.4 and they essentially just added checks to more places than just IOMMU API.
After reading about IOMMU and RMRR in general I realized how dangerous of a game we're playing here. Each x86 system has some untouchable memory regions by default. From what I understand RMRR allows the motherboard to inform OS that there are more regions which under no circumstances should be remapped as there are some opaque-to-the-OS processes touching that memory outside of the OSes knowledge.
Linux already has a workaround where it pretty much ignores GPUs & USBs RMRRs and assumes it's always safe to touch them: https://github.com/torvalds/linux/b...f9e0915a252/drivers/iommu/intel/iommu.c#L2858
The problem is HP clearly abused RMRR. It wasn't meant to mark every device memory mapping in the system as protected just to MAYBE pull some stats from it. What we're doing here is essentially saying "screw you HP" and ignoring their RMRR mapping. However, we don't REALLY know if some of these regions were legitimately marked as non-touchable, HP's software is closed-source and they're not willing to help either. One example will probably be attempting to use shared ethernet for iLO while trying to pass that card to a guest... I can only predict a catastrophe as iLO tries to poke that ethernet's memory.
The only stable long-term solution will be someone knowledgable looking at the bios to disable "HPE Shared Memory features" for all PCIe devices.
EDIT
I read a bit about IOMMU and the purpose of RMRRs (RedHat actually published a nice document about that). Besides me hating Intel now I dig into the kernel in a hope of writing a kernel module which can remove RMRR restriction. Sadly no RMRR structures are exposed to the shared space + I don't know what implications it may have if we just remove RMRR.
However, the fact it worked in <5.4 was actually a bug. QEMU request the device to be remapped to the guest address space using standard ioctl(). After chain of events it lands in vfio_dma_do_map() which performs some checks and does the remap.
The check which makes RMRR patch from this topic unusable was introduced in this commit: https://github.com/torvalds/linux/commit/9b77e5c79840fc334a5b7f770c5ab0c09dc0e028 However, this is a part of a bigger project dealing with IOVA lists state overall, which in my understanding is attempting to remove Intel-only snowflakiness and introduce kernel-wide IOVA status/permission. NULLing this check for now is probably as safe as NULLing the RMRR check in IOMMU API.
Last edited:
I tested the packages throughly and prepared a complete rundown of the issue with all the possible fixes & technical reasons.
https://github.com/kiler129/relax-intel-rmrr
Anyone interested can either download precompiled debs or build them from sources. After installation flipping a kernel switch will activate the patch.
Enjoy. Open source FTW
https://github.com/kiler129/relax-intel-rmrr
Anyone interested can either download precompiled debs or build them from sources. After installation flipping a kernel switch will activate the patch.
Enjoy. Open source FTW
I tested the packages throughly and prepared a complete rundown of the issue with all the possible fixes & technical reasons.
https://github.com/kiler129/relax-intel-rmrr
Anyone interested can either download precompiled debs or build them from sources. After installation flipping a kernel switch will activate the patch.
Enjoy. Open source FTW
So for the non-technical (myself), are you saying you have passthrough working on the latest kernels?
So for the non-technical (myself), are you saying you have passthrough working on the latest kernels?
Yes, it is running Proxmox 6.2-4 with Linux v5.4.65-1. For the ultimate test I was actually stress-testing the solution with the worst case scenario: passthrough of a RAID/HBA soldered in the motherboard of my HPE Microserver Gen8 which is also tied in one IOMMU group with some LPC controller (handles RS232 and such):
Works perfectly after installation of the debs + adding the boot option
For a normal user it's literally a 5 minutes fix.
Last edited:
Yes, it is running Proxmox 6.2-4 with Linux v5.4.65-1. For the ultimate test I was actually stress-testing the solution with the worst case scenario: passthrough of a RAID/HBA soldered in the motherboard of my HPE Microserver Gen8 which is also tied in one IOMMU group with some LPC controller (handles RS232 and such):
View attachment 20733
Works perfectly after installation of the debs + adding the boot option
This is excellent if it works out. I will give it a try for sure! Thanks.