Dell X710 and SR-IOV

P

plrpilot

New Member
Sep 22, 2023
3
0
1
I've got a Dell XR-12 with an Intel X710. I need to enable SR-IOV, but it's not allowing it.

Version Info:
root@pve:~# pveversion --verbose
proxmox-ve: 8.3.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.3.1 (running version: 8.3.1/fb48e850ef9dde27)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-5
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.2
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.2
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1
Click to expand...

IOMMU is enabled:
root@pve:~# dmesg | grep -e DMAR -e IOMMU
[ 0.012551] ACPI: DMAR 0x000000006F7FD000 0000E8 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.012572] ACPI: Reserving DMAR table memory at [mem 0x6f7fd000-0x6f7fd0e7]
[ 0.220268] DMAR: IOMMU enabled
[ 0.541696] DMAR: Host address width 46
[ 0.541697] DMAR: DRHD base: 0x000000bb7fc000 flags: 0x0
[ 0.541707] DMAR: dmar0: reg_base_addr bb7fc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[ 0.541709] DMAR: DRHD base: 0x000000d0ffc000 flags: 0x0
[ 0.541712] DMAR: dmar1: reg_base_addr d0ffc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[ 0.541714] DMAR: DRHD base: 0x000000e67fc000 flags: 0x0
[ 0.541717] DMAR: dmar2: reg_base_addr e67fc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[ 0.541719] DMAR: DRHD base: 0x000000fb7fc000 flags: 0x0
[ 0.541722] DMAR: dmar3: reg_base_addr fb7fc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[ 0.541723] DMAR: DRHD base: 0x000000a5ffc000 flags: 0x1
[ 0.541726] DMAR: dmar4: reg_base_addr a5ffc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[ 0.541727] DMAR: RMRR base: 0x000000693d8000 end: 0x000000693dafff
[ 0.541729] DMAR: ATSR flags: 0x0
[ 0.541731] DMAR-IR: IOAPIC id 8 under DRHD base 0xa5ffc000 IOMMU 4
[ 0.541733] DMAR-IR: HPET id 0 under DRHD base 0xa5ffc000
[ 0.541734] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.543513] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.212700] DMAR: No SATC found
[ 1.212702] DMAR: dmar2: Using Queued invalidation
[ 1.212708] DMAR: dmar1: Using Queued invalidation
[ 1.212711] DMAR: dmar0: Using Queued invalidation
[ 1.212712] DMAR: dmar4: Using Queued invalidation
[ 1.215826] DMAR: Intel(R) Virtualization Technology for Directed I/O
Click to expand...

lspci:

root@pve:~# lspci | grep X710
8a:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
8a:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
8a:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
8a:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
Click to expand...

getting the named instance:

root@pve:~# ls -l /sys/bus/pci/devices/0000:8a:00.3/net/
total 0
drwxr-xr-x 5 root root 0 Jan 28 12:40 ens4f3np3
Click to expand...

and looking at the ethtool for ens4f3np3:

root@pve:~# ethtool ens4f3np3
Settings for ens4f3np3:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseX/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseX/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Click to expand...

root@pve:~# ethtool -i ens4f3np3
driver: i40e
version: 6.8.12-5-pve
firmware-version: 9.50 0x8000f25d 23.0.8
expansion-rom-version:
bus-info: 0000:8a:00.3
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
Click to expand...

But, if I try to enable the virtual functions on the interface, I get the following:

root@pve:~# echo 4 > /sys/class/net/ens4f3np3/device/sriov_numvfs
-bash: /sys/class/net/ens4f3np3/device/sriov_numvfs: Permission denied
Click to expand...

There are numerous posts on the X710 and SR-IOV, but I've not been able to replicate the success of any of them in my efforts. Any ideas would be greatly appreciated.
 
can you post the output of:

Code: 
cat /sys/class/net/ens4f3np3/device/sriov_totalvfs
lspci -vvv -s 0000:8a:00.3
 
root@pve:~# cat /sys/class/net/ens4f3np3/device/sriov_totalvfs
cat: /sys/class/net/ens4f3np3/device/sriov_totalvfs: No such file or directory
root@pve:~# lspci -vvv -s 0000:8a:00.3
8a:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 18
NUMA node: 0
IOMMU group: 4
Region 0: Memory at d1000000 (64-bit, prefetchable) [size=16M]
Region 3: Memory at d5000000 (64-bit, prefetchable) [size=32K]
Expansion ROM at d5280000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00001000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 512 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [e0] Vital Product Data
Product Name: X710 10GbE Controller
Read-only fields:
[V0] Vendor specific: FFV23.0.8
[PN] Part number: 0G48TYA00
[MN] Manufacture ID: 1028
[V1] Vendor specific: DSV1028VPDR.VER2.0
[V3] Vendor specific: DTINIC
[V4] Vendor specific: DCM1001FFFFFF2101FFFFFF3201FFFFFF4301FFFFFF1402FFFFFF2502FFFFFF3602FFFFFF4702FFFFFF1803FFFFFF2903FFFFFF3A03FFFFFF4B03FFFFFF1C04FFFFFF2D04FFFFFF3E04FFFFFF4F04FFFFFF
[V5] Vendor specific: NPY4
[V6] Vendor specific: PMT7
[V7] Vendor specific: NMVIntel Corp
[V8] Vendor specific: L1D0
[SN] Serial number: MYFLMIT2BR005U
[V2] Vendor specific: P��
[V9] Vendor specific: Intel(R) 10GbE 4P X710 Adapter
[VA] Vendor specific: L1D0
[RV] Reserved: checksum good, 2 byte(s) reserved
Read/write fields:
[Y1] System specific: CCF1
End
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number 90-cc-41-ff-ff-6f-7c-50
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Transaction Processing Hints
Device specific mode supported
No steering table available
Capabilities: [1b0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: i40e
Kernel modules: i40e
Click to expand...
 
ok it seems your card does not support sr-iov (maybe a custom firmware?) e.g. i have here a i350-t2 which supports it and it reports with lspci the following:

Code: 
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)                       
        IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000                
        IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-        
        IOVSta: Migration-                                                           
        Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00 
        VF offset: 128, stride: 4, Device ID: 1520                                   
        Supported Page Size: 00000553, System Page Size: 00000001                    
        Region 0: Memory at 00000060f2360000 (64-bit, prefetchable)                  
        Region 3: Memory at 00000060f2340000 (64-bit, prefetchable)                  
        VF Migration: offset: 00000000, BIR: 0

edit: did you enable sr-iov in your mainboards bios?
 
I agree that it's not reporting it, but that's what I'm trying to figure out. There are numerous documented cases, including Intel's dedicated X710 web page, that discusses the X710's SR-IOV support.

We've got an i350, as well, and it works with SR-IOV, but the VF's of the i350 don't support promiscuous mode, where the X710 and XL710 cards do.

I've been trying to figure out if Proxmox was just not loading the drivers associated with the virtual functions.
 
You must log in or register to reply here.
Top Bottom