[TUTORIAL] Compile Proxmox VE with patched intel-iommu driver to remove RMRR check

rmcelroy

New Member
May 13, 2022
6
1
1
Your debs work perfectly on my machine! Not sure what was going wrong with my build process. I'll worry about that on the next version. :)

Super appreciate you taking the time to keep the repo current and sharing the built packages with the novices who struggle to build their own.
 
  • Like
Reactions: Whitterquick

Aterfax

New Member
Mar 21, 2021
17
10
3
30
Your debs work perfectly on my machine! Not sure what was going wrong with my build process. I'll worry about that on the next version. :)

Super appreciate you taking the time to keep the repo current and sharing the built packages with the novices who struggle to build their own.

I resolved the weirdness with the Docker build process, I forgot to copy the patches into the docker image! :rolleyes:

As to broken builds: One possibility if people are not cloning the repo itself byte for byte and are copying and pasting certain files, they will almost certainly break the correct tabbing / padding in in the patch file (https://github.com/Aterfax/relax-intel-rmrr/blob/master/patches/relaxable-rmrr-patch-sed.txt)

That might result in it building without the correct patches in place.

tl;dr make sure if you build you clone the repo!
 

Aterfax

New Member
Mar 21, 2021
17
10
3
30
I've now finished tooling about with the README files to match the current setup.
 

dominiaz

Member
Sep 16, 2016
25
1
23
35
I have 5.15.35-1-pve-relaxablermrr on proxmox 7.2 and when I am trying to passthrought X520-DA2 card then dmesg show error like this.

Code:
[ 1134.342356] DMAR: DRHD: handling fault status reg 2
[ 1134.342437] DMAR: [DMA Read NO_PASID] Request device [04:00.0] fault addr 0x176014000 [fault reason 0x0c] non-zero reserved fields in PTE

The card on VM is working fine but that error is annoying. Do you know how to fix it?
 

Spirog

Member
Jan 31, 2022
230
25
18
Chicago, IL
I have 5.15.35-1-pve-relaxablermrr on proxmox 7.2 and when I am trying to passthrought X520-DA2 card then dmesg show error like this.

Code:
[ 1134.342356] DMAR: DRHD: handling fault status reg 2
[ 1134.342437] DMAR: [DMA Read NO_PASID] Request device [04:00.0] fault addr 0x176014000 [fault reason 0x0c] non-zero reserved fields in PTE

The card on VM is working fine but that error is annoying. Do you know how to fix it?
https://access.redhat.com/solutions/2047843

maybe helpful ?
 
  • Like
Reactions: Whitterquick

moretty

New Member
Sep 26, 2022
3
0
1
I resolved the weirdness with the Docker build process, I forgot to copy the patches into the docker image! :rolleyes:

As to broken builds: One possibility if people are not cloning the repo itself byte for byte and are copying and pasting certain files, they will almost certainly break the correct tabbing / padding in in the patch file (https://github.com/Aterfax/relax-intel-rmrr/blob/master/patches/relaxable-rmrr-patch-sed.txt)

That might result in it building without the correct patches in place.

tl;dr make sure if you build you clone the repo!
First of all thank you very much for your work, it is very exciting. I just got to proxmox and I don't have any experience. I just ran into the rmrr issue on my hp g8. I am a bit lost. I am in version 5.15.53-1, I understand that I would have to downgrade it to 5.15.39-3 and use the latest one that you have in your repository, which is 5.15.39-3. No? Thank you
 

Aterfax

New Member
Mar 21, 2021
17
10
3
30
You could use the most recent signed and tested version which would indeed be a downgrade from your current version but should function fine (although you may need to pin the kernel version to be booted (https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_kernel_pin).

Alternatively you can use the 2nd most recent version produced automatically by the Github Actions process here: https://github.com/Aterfax/relax-intel-rmrr/actions/runs/3162013761

You can download the zip and follow the normal instructions from the Github: https://github.com/Aterfax/relax-intel-rmrr/blob/master/README.md#proxmox---premade-packages-easy

I've not personally tested this version yet but accessiblepixel on Github has and gave it a thumbs up here: https://github.com/Aterfax/relax-intel-rmrr/issues/19#issuecomment-1264341426
 
  • Like
Reactions: moretty

moretty

New Member
Sep 26, 2022
3
0
1
You could use the most recent signed and tested version which would indeed be a downgrade from your current version but should function fine (although you may need to pin the kernel version to be booted (https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_kernel_pin).

Alternatively you can use the 2nd most recent version produced automatically by the Github Actions process here: https://github.com/Aterfax/relax-intel-rmrr/actions/runs/3162013761

You can download the zip and follow the normal instructions from the Github: https://github.com/Aterfax/relax-intel-rmrr/blob/master/README.md#proxmox---premade-packages-easy

I've not personally tested this version yet but accessiblepixel on Github has and gave it a thumbs up here: https://github.com/Aterfax/relax-intel-rmrr/issues/19#issuecomment-1264341426
thank you very much. I am going to read and try one of the options that you indicate. You are very kind. Thank you
 

YellowSnowman23

New Member
Oct 6, 2022
1
0
1
I've been told if you have a Gen9+ machine RMRR is a non-issue. For some reason i cannot passthrough a 10GbE NIC (FlexibleLOM). There is no option in bios to disable HPE Shared Memory like the SmartArray controller. I ended up using the 5.15.39-3-pve-relaxablermrr kernel and still no dice.

Any ideas?
Edit: To add, im using 7.2-7 on a DL360 Gen 9

Code:
root@pve:~# dmesg | grep -e DMAR -e IOMMU
[    0.011367] ACPI: DMAR 0x000000007B7E7000 000302 (v01 HP     ProLiant 00000001 HP   00000001)
[    0.011409] ACPI: Reserving DMAR table memory at [mem 0x7b7e7000-0x7b7e7301]
[    0.369506] DMAR: IOMMU enabled
[    0.833714] DMAR: Host address width 46
[    0.833716] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.833721] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
[    0.833724] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    0.833728] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de
[    0.833730] DMAR: RMRR base: 0x00000079174000 end: 0x00000079176fff
[    0.833732] DMAR: RMRR base: 0x000000791f4000 end: 0x000000791f7fff
[    0.833737] DMAR: RMRR base: 0x000000791de000 end: 0x000000791f3fff
[    0.833738] DMAR: RMRR base: 0x000000791cb000 end: 0x000000791dbfff
[    0.833740] DMAR: RMRR base: 0x000000791dc000 end: 0x000000791ddfff
[    0.833741] DMAR: ATSR flags: 0x0
[    0.833742] DMAR: ATSR flags: 0x0
[    0.833745] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    0.833747] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    0.833748] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    0.833750] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    0.833751] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.834589] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.345628] DMAR: No SATC found
[    1.345631] DMAR: dmar0: Using Queued invalidation
[    1.345640] DMAR: dmar1: Using Queued invalidation
[    1.358823] DMAR: Intel(R) Virtualization Technology for Directed I/O
[   99.078811] vfio-pci 0000:04:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[  120.817035] vfio-pci 0000:04:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[  131.897813] vfio-pci 0000:04:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.


Code:
root@pve:~# uname -r
5.15.39-3-pve-relaxablermrr

Code:
root@pve:~# lsmod | grep vfio
vfio_pci               16384  0
vfio_pci_core          73728  1 vfio_pci
vfio_virqfd            16384  1 vfio_pci_core
irqbypass              16384  14 vfio_pci_core,kvm
vfio_iommu_type1       40960  0
vfio                   45056  2 vfio_pci_core,vfio_iommu_type1


Code:
root@pve:~# lspci -vvv
[...]
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
    DeviceName: Embedded FlexibleLOM 1 Port 1
    Subsystem: Hewlett-Packard Company 561FLR-T 2-port 10Gb Ethernet Adapter
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 16
    NUMA node: 0
    IOMMU group: 36
    Region 0: Memory at 95400000 (64-bit, prefetchable) [disabled] [size=2M]
    Region 4: Memory at 95604000 (64-bit, prefetchable) [disabled] [size=16K]
    Expansion ROM at 95680000 [virtual] [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [70] MSI-X: Enable- Count=64 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00002000
    Capabilities: [a0] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
        LnkCap:    Port #2, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s (ok), Width x8 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [e0] Vital Product Data
        Product Name: HP Ethernet 10Gb 2-port 561FLR-T Adapter
        Read-only fields:
            [PN] Part number: 700697-001
            [EC] Engineering changes: C-5338
            [SN] Serial number: MYI4130329
            [V0] Vendor specific: 15W/4W  PCIeG2x8 2p 10G BT Intel X540
            [V2] Vendor specific: 5413
            [V4] Vendor specific: 38EAA7370684
            [V5] Vendor specific: 0C
            [RV] Reserved: checksum good, 0 byte(s) reserved
        Read/write fields:
            [V1] Vendor specific: 3.8.04
            [V3] Vendor specific: 2.8.20
            [V6] Vendor specific: 2.3.06
            [YA] Asset tag: N/A
            [YB] System specific: xxxxxxxxxxxxxxxx
            [YC] System specific: xxxxxxxxxxxxxxxx
        End
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [140 v1] Device Serial Number 38-ea-a7-ff-ff-37-06-84
    Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 1
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
        IOVCap:    Migration-, Interrupt Message Number: 000
        IOVCtl:    Enable- Migration- Interrupt- MSE- ARIHierarchy+
        IOVSta:    Migration-
        Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
        VF offset: 128, stride: 2, Device ID: 1515
        Supported Page Size: 00000553, System Page Size: 00000001
        Region 0: Memory at 0000000095800000 (64-bit, non-prefetchable)
        Region 3: Memory at 0000000095900000 (64-bit, non-prefetchable)
        VF Migration: offset: 00000000, BIR: 0
    Capabilities: [1d0 v1] Access Control Services
        ACSCap:    SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        ACSCtl:    SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
    Kernel driver in use: vfio-pci
    Kernel modules: ixgbe
 
Last edited:

GastonJ

New Member
Aug 22, 2022
5
0
1
I take it passing through onboard devices using this is a no go?
I was planning on using 2 of the internal NIC's on my DL380G7 for a firewall configuration, but all that happens for me is it locks my server. I'll have to find a dual port Intel NIC to try it with.

For info

Linux pve 5.19.7-2-pve-relaxablermrr #1 SMP PREEMPT_DYNAMIC PVE 5.19.7-2 (Tue, 04 Oct 2022 17:18:40 + x86_64 GNU/Linux

dmesg|grep -i DMAR
[ 0.010312] ACPI: DMAR 0x00000000DF62FE80 000154 (v01 HP ProLiant 00000001 \xd2? 0000162E)
[ 0.010362] ACPI: Reserving DMAR table memory at [mem 0xdf62fe80-0xdf62ffd3]
[ 0.670422] DMAR: IOMMU enabled
[ 0.670424] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
[ 1.533794] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 2.034030] DMAR: Host address width 39
[ 2.034033] DMAR: DRHD base: 0x000000e7ffe000 flags: 0x1
[ 2.034044] DMAR: dmar0: reg_base_addr e7ffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[ 2.034048] DMAR: RMRR base: 0x000000df7fc000 end: 0x000000df7fdfff
[ 2.034052] DMAR: RMRR base: 0x000000df7f5000 end: 0x000000df7fafff
[ 2.034054] DMAR: RMRR base: 0x000000df63e000 end: 0x000000df63ffff
[ 2.034056] DMAR: ATSR flags: 0x0
[ 2.034099] DMAR: No SATC found
[ 2.034119] DMAR: dmar0: Using Queued invalidation
[ 2.038306] DMAR: Intel(R) Virtualization Technology for Directed I/O

kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

I'll find an external card at the weekend.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!