Multi port nic, passthrough one Nic per vm

[svenvg93]

I have a 4 port Nic installed in my server. Server has running three VM's I want to passthrough one Nic of that card to each VM.
Like;
Port 1 to VM1
Port2 to VM2
Port3 to VM3

I setup the passthrough like descrioted here; https://pve.proxmox.com/wiki/Pci_passthrough.
I can passthrough the interface but only have one vm running at the time. Probably cause it uses the full PCI.
It there a way to passthrough the nic directly to the VM without the need of a VM bridge.

 

[Dunuin]

First you should check if each function of the NIC got its own IOMMU group. If that isn't the case, you can't passthrough them individually.
Then you need to make sure that you don't check the "All functions" checkbox when adding the NIC to the VM. Otherwise it will passthrough all 4 ports to a single VM.

And what NIC model do you got? Not all NICs support passingthough of individual ports to different VMs.

 

[svenvg93]

They seem to be in separate groups. Although im a noob in this.

root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/13/devices/0000:07:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:12.0
/sys/kernel/iommu_groups/11/devices/0000:05:00.0
/sys/kernel/iommu_groups/1/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:04:00.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:03:00.1
/sys/kernel/iommu_groups/1/devices/0000:02:02.0
/sys/kernel/iommu_groups/1/devices/0000:04:00.0
/sys/kernel/iommu_groups/1/devices/0000:02:04.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.3
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/12/devices/0000:06:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.5
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
/sys/kernel/iommu_groups/10/devices/0000:00:1f.6
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1d.0
root@pve:~# lspci -nn | grep Ethernet
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (11) I219-V [8086:0d4d]
03:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
03:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
04:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)

 

[Dunuin]

They seem to be in separate groups.

No, they all share IOMMU group 1, so you can't pass them through individually. And there are some other devices in IOMMU group too. What are the other devices? Maybe you put the NIC in a PCIe slot that is connected to the chipset and then also passing through other important devices of the chipset that the host needs.

 

[Neobin]

You can get a way better overview over your IOMMU-groups with:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

(Thanks again to the forum's PCIe-passthrough-master for this command.)

 

[svenvg93]

Thanks @Dunuin. There is only one slot on the motherboard. Its a mini-ITX buid. I will see if I can find another card that might work, or come up with another solution.

 

[Dunuin]

Check your mainboards manual if there are diagrams showing where the PCIe slots are connected to. I would guess it is connected to the chipset, so passthrough won't work without acs override, no matter what card you test.

 

[rokyo401]

I'm currently trying to do the same thing, but it looks like the two ports on my NIC are also in the same IOMMU group:

lspci -nn | grep Ethernet
41:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
41:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
41:00.2 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
41:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
81:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)  <---
81:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)  <---

I wanted to split the 10-Gigabit card to two VMs, one port each, but they are both in iommu_group 115, along with two other devices:

find /sys/kernel/iommu_groups/ -type l | grep 81
/sys/kernel/iommu_groups/115/devices/0000:81:00.0
/sys/kernel/iommu_groups/115/devices/0000:81:00.1
---
find /sys/kernel/iommu_groups/ -type l | grep 115
/sys/kernel/iommu_groups/115/devices/0000:80:01.0
/sys/kernel/iommu_groups/115/devices/0000:81:00.0  <---
/sys/kernel/iommu_groups/115/devices/0000:80:01.3
/sys/kernel/iommu_groups/115/devices/0000:81:00.1  <---

which are these:

lspci | grep 80
80:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
80:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

If I pass through just one of them to one VM, it shows correctly as one card inside the VM:

00:10.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

So, I definitely can't pass the other one through to a different VM, because they're in the same IOMMU group on the host?
Can I split the group somehow?
Is it also impossible to use one port by a VM and one by the host? Only both by the same entity (VM or host)?

 

[rokyo401]

Okay, I checked via lspci -vvv whether the cards have Access Control Service capabilities, as suggested in the Proxmox docs (https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation):

81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Super Micro Computer Inc AOC-STGN-i2S
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin B routed to IRQ 352
        NUMA node: 1
        IOMMU group: 115
        Region 0: Memory at 50170280000 (64-bit, prefetchable) [disabled] [size=512K]
        Region 2: I/O ports at 3000 [disabled] [size=32]
        Region 4: Memory at 50170504000 (64-bit, prefetchable) [disabled] [size=16K]
        Expansion ROM at b0500000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D3 NoSoftRst- PME-Enable+ DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable- Count=64 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [e0] Vital Product Data
                Product Name: Supermicro Network Adapter
                Read-only fields:
                        [PN] Part number: AOC-STGN-I2S   
                        [V0] Vendor specific: 0210
                        [V1] Vendor specific: 2.11 
                        [SN] Serial number: VA19AS005755       
                        [VA] Vendor specific: 2
                        [V2] Vendor specific: AC1F6BF564CA
                        [V3] Vendor specific: AC1F6BF564CB
                        [V4] Vendor specific:             
                        [V5] Vendor specific:             
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                Read/write fields:
                        [VB] Vendor specific: \x00
                End
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number ac-1f-6b-ff-ff-f5-64-ca
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
                VF offset: 128, stride: 2, Device ID: 10ed
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000050170508000 (64-bit, prefetchable)
                Region 3: Memory at 0000050170608000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Kernel driver in use: vfio-pci
        Kernel modules: ixgb

It looks like it does NOT!

Other devices on this machine show a section "Access Control Services":

Capabilities: [2a0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

but not the 10GB NIC :-(


So, no chance?

 

[Dariusz]

Either something changed, or my system about to explode, but I have the same nic and I was able to pass 1 port to TrueNas Scale, not sure... gonna test it soon.

I left other one in Proxmox... so that maybe work?

 

[wahmed]

@Dariusz
Is there any particular reason you don't want to use bridges? That will be the simplest and cleanest configuration without extra layers. This sort of passthrough setup will always depend on the hardware itself. If the NIC goes, you will be down till you find another suitable replacement.

 

[Dariusz]

Hey
Not against bridges tbh, I'm reading all/learning all I can on networking in proxmox now. I just assumed in mean time that passing 1 port from 10gbe card would be the best approach for TrueNas.

Does it matter? Any1 with experience? I move a lot of data & I'd love to get full speed.

 

[wahmed]

No, it does not matter, if that's the specific setup you are looking for.

I will give you my version to make this work. It comes down to if the hardware supports it or not, really.
- First thing to check if the CPU and Mboard supports IOMMU (Inet VT-d for intel, AMD-VI for AMD).
- Also have to ensure that the NIC itself supports pass-through and is capable of being isolated an individual port.
- Enable IOMMU in the BIOS. Enable VT-d for Intel or AMD-Vi for AMD.
- Edit grub in /etc/default/grub and the following to the 'GRUB_CMDLINE_LINUX_DEFAULT' line:
For Intel: intel_iommu=on
For AMD: amd_iommu=on
- Update grub($ update-grub) and reboot.
- On the Proxmox GUI go to Hardware of the VM and add a PCI device. Find the port using the PCI Address.
- Power on the VM.

 

[Dariusz]

Hey
Yep I have iommu setup, with gpu/sata/etc. So its all good on this side. I just wonder if there are benefits to passing 1/2 slots from my nic to TrueNas directly.
I did pass 1/both and they show up in TrueNas so I suppose it "works", But I'm yet to test it in file transfer/etc. I do wonder if there is a "soft way" of testing it without all the network/cabling etc. As I have to take my server to basement where cables live ;c