PCIE ACS override not working (can not split IOMMU groups)

T

takipeti90

New Member
Feb 20, 2019
5
0
1
34
Hello everyone!

I'm working on my home network so I installed pfSense in a VM and I would like to pass my external NIC and M.2 wifi card to pfSense, but the external PCIE slot and the M.2 slot are in the same IOMMU group than the internal NIC.
I have an Asrock J3455-ITX motherboard, where I already enabled Intel virtualization in the BIOS and the iommu is enabled as well. And also tried pcie_acs_override=downstream,multifunction but IOMMU groups are stll the same.

Code: 
dmesg | grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.15.18-10-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on pcie_acs_override=downstream,multifunction
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.15.18-10-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on pcie_acs_override=downstream,multifunction
[    0.000000] DMAR: IOMMU enabled
[    0.004000] DMAR-IR: IOAPIC id 1 under DRHD base  0xfed65000 IOMMU 1
[    1.329307] iommu: Adding device 0000:00:00.0 to group 0
[    1.329322] iommu: Adding device 0000:00:02.0 to group 1
[    1.329337] iommu: Adding device 0000:00:0e.0 to group 2
[    1.329360] iommu: Adding device 0000:00:0f.0 to group 3
[    1.329375] iommu: Adding device 0000:00:12.0 to group 4
[    1.329429] iommu: Adding device 0000:00:13.0 to group 5
[    1.329450] iommu: Adding device 0000:00:13.1 to group 5
[    1.329469] iommu: Adding device 0000:00:13.2 to group 5
[    1.329488] iommu: Adding device 0000:00:13.3 to group 5
[    1.329511] iommu: Adding device 0000:00:15.0 to group 6
[    1.329533] iommu: Adding device 0000:00:1f.0 to group 7
[    1.329546] iommu: Adding device 0000:00:1f.1 to group 7
[    1.329557] iommu: Adding device 0000:01:00.0 to group 5
[    1.329569] iommu: Adding device 0000:02:00.0 to group 5
[    1.329581] iommu: Adding device 0000:03:00.0 to group 5
[    1.329596] iommu: Adding device 0000:04:00.0 to group 5

I think the problem is that my PCI bridge has ACS capability so the acs_override_patch do nothing in this case:

Code: 
/* Never override ACS for legacy devices or devices with ACS caps */
+    if (!pci_is_pcie(dev) ||
+        pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS))
+        return -ENOTTY;

Here is the lspci -vvnn result of the related devices:

Code: 
00:13.0 PCI bridge [0604]: Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A #1 [8086:5ad8] (rev fb) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 122
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: 91500000-915fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #2, PowerLimit 10.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee00218  Data: 0000
        Capabilities: [90] Subsystem: ASRock Incorporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A [1849:5ad8]
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v0] #00
        Capabilities: [140 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [150 v0] #00
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=40us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport
        Kernel modules: shpchp

00:13.1 PCI bridge [0604]: Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A #2 [8086:5ad9] (rev fb) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin B routed to IRQ 123
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        Memory behind bridge: 91000000-911fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #3, PowerLimit 10.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee00258  Data: 0000
        Capabilities: [90] Subsystem: ASRock Incorporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A [1849:5ad9]
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v0] #00
        Capabilities: [140 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [150 v0] #00
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=40us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
                           T_CommonMode=50us LTR1.2_Threshold=163840ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport
        Kernel modules: shpchp

00:13.2 PCI bridge [0604]: Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A #3 [8086:5ada] (rev fb) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin C routed to IRQ 124
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: 91400000-914fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #4, PowerLimit 10.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee00298  Data: 0000
        Capabilities: [90] Subsystem: ASRock Incorporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A [1849:5ada]
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v0] #00
        Capabilities: [140 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [150 v0] #00
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=40us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport
        Kernel modules: shpchp

00:13.3 PCI bridge [0604]: Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A #4 [8086:5adb] (rev fb) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin D routed to IRQ 125
        Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: 91300000-913fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #5, PowerLimit 10.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee002d8  Data: 0000
        Capabilities: [90] Subsystem: ASRock Incorporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A [1849:5adb]
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v0] #00
        Capabilities: [140 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [150 v0] #00
        Capabilities: [200 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=40us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: pcieport
        Kernel modules: shpchp

lspci
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
02:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

lspci -t
-[0000:00]-+-00.0
           +-02.0
           +-0e.0
           +-0f.0
           +-12.0
           +-13.0-[01]----00.0
           +-13.1-[02]----00.0
           +-13.2-[03]----00.0
           +-13.3-[04]----00.0
           +-15.0
           +-1f.0
           \-1f.1

Back to the original issue, I would like to split iommu group 5 to separate
01:00.0 internal NIC
02:00.0 external wifi adapter
04:00.0 external NIC
and pass through wifi and external NIC to pfSense VM.

Anyone has a working solution?

Thanks
 
did you try without 'multifuncion' ? so
pcie_acs_override=downstream
 
dcsapak said:
did you try without 'multifuncion' ? so
pcie_acs_override=downstream
Click to expand...

yes, that was my first try

Btw, I would like to recompile the kernel (insert debug messeges into the ACS patch) but I have git error:

Code: 
gcc --version|grep "6\.3" || false
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
test -f "submodules/ubuntu-bionic/README" || git submodule update --init submodules/ubuntu-bionic
test -f "submodules/zfsonlinux/Makefile" || git submodule update --init submodules/zfsonlinux
(test -f "submodules/zfsonlinux/zfs/upstream/README.markdown" && test -f "submodules/zfsonlinux/spl/upstream/README.markdown") || (cd submodules/zfsonlinux; git submodule update --init)
Cloning into '/usr/src/pve-kernel/submodules/zfsonlinux/spl/upstream'...
fatal: remote error: access denied or repository not exported: /git/mirror_spl
fatal: clone of 'git://git.proxmox.com/git/mirror_spl' into submodule path '/usr/src/pve-kernel/submodules/zfsonlinux/spl/upstream' failed
Failed to clone 'spl/upstream'. Retry scheduled
Cloning into '/usr/src/pve-kernel/submodules/zfsonlinux/zfs/upstream'...
fatal: remote error: access denied or repository not exported: /git/mirror_zfs
fatal: clone of 'git://git.proxmox.com/git/mirror_zfs' into submodule path '/usr/src/pve-kernel/submodules/zfsonlinux/zfs/upstream' failed
Failed to clone 'zfs/upstream'. Retry scheduled
Cloning into '/usr/src/pve-kernel/submodules/zfsonlinux/spl/upstream'...
fatal: remote error: access denied or repository not exported: /git/mirror_spl
fatal: clone of 'git://git.proxmox.com/git/mirror_spl' into submodule path '/usr/src/pve-kernel/submodules/zfsonlinux/spl/upstream' failed
Failed to clone 'spl/upstream' a second time, aborting
Makefile:158: recipe for target 'submodule' failed
make: *** [submodule] Error 1

I have seen there is a new commit "ZFS/SPL: rework submodule and build" on master (https://git.proxmox.com/?p=pve-kernel.git;a=shortlog;h=refs/heads/master), do you know anything about this issue?

Thanks
 
Hello, I am also j3455-itx, encountered the same problem, re-compiled the kernel, or the same, no change, you solve this problem？ need your help
Machine translation, please forgive me!
 
Got the same problem here, with the pve-test kernel as well (same ASRock J3455-ITX).
Code: 
# dmesg |grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.3.7-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:off,vesafb:off
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.148691] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.3.7-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:off,vesafb:off
[    0.148818] DMAR: IOMMU enabled
[    0.307950] DMAR-IR: IOAPIC id 1 under DRHD base  0xfed65000 IOMMU 1
[    1.842524] pci 0000:00:00.0: Adding to iommu group 0
[    1.878672] pci 0000:00:02.0: Adding to iommu group 1
[    1.878825] pci 0000:00:0e.0: Adding to iommu group 2
[    1.878943] pci 0000:00:0f.0: Adding to iommu group 3
[    1.879066] pci 0000:00:12.0: Adding to iommu group 4
[    1.879228] pci 0000:00:13.0: Adding to iommu group 5
[    1.879258] pci 0000:00:13.1: Adding to iommu group 5
[    1.879285] pci 0000:00:13.2: Adding to iommu group 5
[    1.879319] pci 0000:00:13.3: Adding to iommu group 5
[    1.879446] pci 0000:00:15.0: Adding to iommu group 6
[    1.879581] pci 0000:00:1f.0: Adding to iommu group 7
[    1.879604] pci 0000:00:1f.1: Adding to iommu group 7
[    1.879625] pci 0000:01:00.0: Adding to iommu group 5
[    1.879650] pci 0000:02:00.0: Adding to iommu group 5
[    1.879670] pci 0000:03:00.0: Adding to iommu group 5
[    1.879690] pci 0000:04:00.0: Adding to iommu group 5
[    1.879707] pci 0000:04:00.1: Adding to iommu group 5


Code: 
# lspci -t
-[0000:00]-+-00.0
           +-02.0
           +-0e.0
           +-0f.0
           +-12.0
           +-13.0-[01]----00.0
           +-13.1-[02]----00.0
           +-13.2-[03]----00.0
           +-13.3-[04]--+-00.0
           |            \-00.1
           +-15.0
           +-1f.0
           \-1f.1
 
PVE V6.1.5.
Same issue with J3455 ITX board after change /etc/default/grub file with "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream".
Code: 
root@pve:/# dmesg | grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.3.13-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.081363] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.3.13-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on pcie_acs_override=downstream
[    0.081488] DMAR: IOMMU enabled
[    0.189929] DMAR-IR: IOAPIC id 1 under DRHD base  0xfed65000 IOMMU 1
[    1.735772] pci 0000:00:00.0: Adding to iommu group 0
[    1.772013] pci 0000:00:02.0: Adding to iommu group 1
[    1.772169] pci 0000:00:0e.0: Adding to iommu group 2
[    1.772285] pci 0000:00:0f.0: Adding to iommu group 3
[    1.772411] pci 0000:00:12.0: Adding to iommu group 4
[    1.772568] pci 0000:00:13.0: Adding to iommu group 5
[    1.772598] pci 0000:00:13.1: Adding to iommu group 5
[    1.772625] pci 0000:00:13.2: Adding to iommu group 5
[    1.772653] pci 0000:00:13.3: Adding to iommu group 5
[    1.772787] pci 0000:00:15.0: Adding to iommu group 6
[    1.772926] pci 0000:00:1f.0: Adding to iommu group 7
[    1.772949] pci 0000:00:1f.1: Adding to iommu group 7
[    1.772970] pci 0000:01:00.0: Adding to iommu group 5
[    1.772990] pci 0000:03:00.0: Adding to iommu group 5
[    1.773010] pci 0000:04:00.0: Adding to iommu group 5
[    1.773040] pci 0000:04:00.1: Adding to iommu group 5
root@pve:/# lspci -t
-[0000:00]-+-00.0
           +-02.0
           +-0e.0
           +-0f.0
           +-12.0
           +-13.0-[01]----00.0
           +-13.1-[02]--
           +-13.2-[03]----00.0
           +-13.3-[04]--+-00.0
           |            \-00.1
           +-15.0
           +-1f.0
           \-1f.1



"
 
Same issue on my Intel N4200, tried with both pcie_acs_override=downstream,multifunction and pcie_acs_override=downstream and still getting the same grouped iommu groups. Is there some limitation of supported hardware for the patch to work? Or is the patch just broken in Proxmox? I'm on Kernel 5.3.18-3-pve #1 SMP PVE 5.3.18-3 (Tue, 17 Mar 2020 16:33:19 +0100) x86_64
 
Same to me on my Intel i5-8259U running: 5.4.41-1-pve
I need to get the network controller split from the audio controller, but with downstream they appear in the same group as before and with downstream,multifunction the show up in group 12 (11 before)
demesg shows:
Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back