SR-IOV Not working for Intel i350-T4

[Sic Alpha]

I have 2x GPUs and an LSI HBA successfully passing through to VMs so I know IOMMU is configured properly and working. SR-IOV is turned on in the MB EUFI BIOS along with the other required settings IOMMU etc.

here is my /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

here is the important line in my /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

I have an Intel i350-T4 that I cannot get to show virtual functions. I purchased it from a reputable retailer.

When I create a file to /etc/modprobe.d/igb.conf with the following line in it igb max_vfs=4 I get no virtual function NICs.

lspci -vv doesn't show SR-IOV as a capability (same for 62:00.1 through 62:00.3 for the total of 4 ports on the i350-T4)

62:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 204
        NUMA node: 3
        Region 0: Memory at e8d00000 (32-bit, non-prefetchable) [size=1M]
        Region 3: Memory at e900c000 (32-bit, non-prefetchable) [size=16K]
        Expansion ROM at e8f80000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140 v1] Device Serial Number b4-96-91-ff-ff-55-65-9c
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
                Device specific mode supported
                Steering table in TPH capability structure
        Capabilities: [1c0 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [1d0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: igb
        Kernel modules: igb

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

[    1.148241] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.148325] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[    1.148370] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[    1.148406] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[    1.153713] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.153714] pci 0000:00:00.2: AMD-Vi: Extended features (0xf77ef22294ada):
[    1.153717] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.153718] pci 0000:20:00.2: AMD-Vi: Extended features (0xf77ef22294ada):
[    1.153720] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.153721] pci 0000:40:00.2: AMD-Vi: Extended features (0xf77ef22294ada):
[    1.153722] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.153723] pci 0000:60:00.2: AMD-Vi: Extended features (0xf77ef22294ada):
[    1.153724] AMD-Vi: Interrupt remapping enabled
[    1.153725] AMD-Vi: Virtual APIC enabled
[    1.154334] AMD-Vi: Lazy IO/TLB flushing enabled
[    1.158951] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.158982] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    1.159010] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[    1.159022] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).

Any ideas as to why I can't get SR-IOV working?

 

[apoc]

Honestly I don't recall that the i350 actually was capable in doing SRIOV. That's matching your lspci-output and the fact that it doesn't work.
Going through my mind I actually can't name any 1Gbit adapter which does sriov. 10 Gbit, yes a lot ... But 1 Gbit?

 

[Sic Alpha]

Honestly I don't recall that the i350 actually was capable in doing SRIOV. That's matching your lspci-output and the fact that it doesn't work.
Going through my mind I actually can't name any 1Gbit adapter which does sriov. 10 Gbit, yes a lot ... But 1 Gbit?

According to the Intel product page for it, it does.
https://ark.intel.com/content/www/u.../intel-ethernet-server-adapter-i350-t4v2.html

PCI-SIG* SR-IOV Capable Yes

Unless they sold me a V1. It was labeled as a V2 when I purchased it though; I'm not sure how to tell the difference.

 

[apoc]

I never paid attention to the V1/V2 part. Likely that's the reason I did not pick up on it.
Intel ARK.is accurate according to my experience. Is the slot the nic is in actually electrical an X4?
Do you have another slot to try?

Maybe you can try verifying v1/v2 via the serial number?

 

[Dunuin]

I never paid attention to the V1/V2 part. Likely that's the reason I did not pick up on it.
Intel ARK.is accurate according to my experience. Is the slot the nic is in actually electrical an X4?
Do you have another slot to try?

Maybe you can try verifying v1/v2 via the serial number?

There should be a holographic sticker with an code you can use online to verify that is not a faked i350-T4. Half of all of the i350-T4s are fakes and nearly 99% if the seller is from china. Not sure why you exactly need SR-IOV. I've got that v1 with SR-IOV disabled in BIOS and I can assign each ethernet port to a different VM using PCI passthrough. If that is what you want to accomplish.

 

[dcsapak]

fyi, we have here some i350-t2 (various manufacturing dates) and only some were able to do sr-iov
i even have some bug on the one installed here were it loses the sr-iov bit in the firmware after having been away
from power too long, though there exists some guide on how to set that bit in the firmware...

 

[Sic Alpha]

fyi, we have here some i350-t2 (various manufacturing dates) and only some were able to do sr-iov
i even have some bug on the one installed here were it loses the sr-iov bit in the firmware after having been away
from power too long, though there exists some guide on how to set that bit in the firmware...

Maybe I need to flash the firmware with SR-IOV enabled? Do you have a link to the guide you mentioned on how to dot that?

 

[dcsapak]

well its more of a random mailing listing post than a guide, and i used it only since i knew that my card supported it until the firmware decided to lose that bit...
so use with caution and i do not take any responsibility if the hardware is bricked afterwards...
also the guide is for the -t2 model (which i also have) and yours is -t4, so it may not work at all

https://www.redhat.com/archives/vfio-users/2017-August/msg00086.html

 

[spetrillo]

It works for T4 cards. I have done it on both of my cards!