PVE4 kernel issue, sr_iov disabled on PV4 kernel

elurex

elurex

Active Member
Proxmox Subscriber
Oct 28, 2015
204
15
38
Taiwan
The same server hardware SuperServer 5028D-TN4T when install on Ubuntu 14.04.3 with kernel to 4.2.3 SR-IOV is successful with Intel X557 but failed on PVE 4.0 with kernel 4.2.2-1-pve, how to fix this issue?

I can enable SR_IOV on for my 10 GBe X557 nic and with successfully pci passthrough (iommu & vfio)
Code: 
root@pdc:~# uname -r
4.2.3-040203-generic

lspci -vnn | grep Ethernet
03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:00.1 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:10.0 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.1 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.2 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.3 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.4 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.5 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.6 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.7 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.0 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.1 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.2 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.3 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.4 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.5 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.6 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.7 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]

root@pdc:~# cat /etc/modprobe.d/ixgbe.conf
options ixgbe max_vfs=8

The iommu group is seperated and ready to be use for kvm vfio pci passthrough

/sys/kernel/iommu_groups/45/devices/0000:03:10.0
/sys/kernel/iommu_groups/46/devices/0000:03:10.2
/sys/kernel/iommu_groups/47/devices/0000:03:10.4
/sys/kernel/iommu_groups/48/devices/0000:03:10.6
/sys/kernel/iommu_groups/49/devices/0000:03:11.0
/sys/kernel/iommu_groups/50/devices/0000:03:11.2
/sys/kernel/iommu_groups/51/devices/0000:03:11.4
/sys/kernel/iommu_groups/52/devices/0000:03:11.6
/sys/kernel/iommu_groups/53/devices/0000:03:10.1
/sys/kernel/iommu_groups/54/devices/0000:03:10.3
/sys/kernel/iommu_groups/55/devices/0000:03:10.5
/sys/kernel/iommu_groups/56/devices/0000:03:10.7
/sys/kernel/iommu_groups/57/devices/0000:03:11.1
/sys/kernel/iommu_groups/58/devices/0000:03:11.3
/sys/kernel/iommu_groups/59/devices/0000:03:11.5
/sys/kernel/iommu_groups/60/devices/0000:03:11.7

root@pdc:~# modinfo ixgbe
filename:       /lib/modules/4.2.3-040203-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        4.0.1-k
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     E08010C0D72963989E9A05E
alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*
....
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        mdio,ptp,dca,vxlan
intree:         Y
vermagic:       4.2.3-040203-generic SMP mod_unload modversions
signer:         Build time autogenerated kernel key
sig_key:        CE:E6:5B:C1:E3:07:7F:6F:F5:58:77:52:AC:20:3F:9D:65:F7:05:DA
sig_hashalgo:   sha512
parm:           max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)
parm:           debug:Debug level (0=none,...,16=all) (int)

root@pdc:~# dmesg|grep ixgbe
[    0.000000] Command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    7.208439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
[    7.208439] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[    7.208607] ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[    9.240064] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): SR-IOV enabled with 8 VFs
[    9.512691] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[    9.512692] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    9.611689] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[    9.678012] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
[    9.678013] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    9.678014] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[    9.678015] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    9.678015] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
[    9.722230] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
[    9.722231] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[   15.215889] ixgbe 0000:03:00.0 eth1: IOV is enabled with 8 VFs
[   15.238000] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[   15.238201] ixgbe 0000:03:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[   16.787177] ixgbevf 0000:03:10.1: enabling device (0000 -> 0002)
[   16.788293] ixgbevf 0000:03:10.1: PF still in reset state.  Is the PF interface up?
[   16.788293] ixgbevf 0000:03:10.1: Assigning random MAC address
[   18.431419] ixgbevf 0000:03:10.1: 86:c7:2f:a5:60:0c
[   18.431420] ixgbevf 0000:03:10.1: MAC: 4
[   18.431420] ixgbevf 0000:03:10.1: Intel(R) 82599 Virtual Function
[   18.431577] ixgbevf 0000:03:10.3: enabling device (0000 -> 0002)
[   18.432709] ixgbevf 0000:03:10.3: PF still in reset state.  Is the PF interface up?
[   18.432709] ixgbevf 0000:03:10.3: Assigning random MAC address
[   19.363463] ixgbevf 0000:03:10.3: 62:5a:23:07:bd:59
[   19.363464] ixgbevf 0000:03:10.3: MAC: 4
[   19.363465] ixgbevf 0000:03:10.3: Intel(R) 82599 Virtual Function
[   19.363609] ixgbevf 0000:03:10.5: enabling device (0000 -> 0002)
[   19.364718] ixgbevf 0000:03:10.5: PF still in reset state.  Is the PF interface up?
[   19.364719] ixgbevf 0000:03:10.5: Assigning random MAC address
[   19.364849] ixgbevf 0000:03:10.5: 92:65:d9:a2:cd:48
[   19.364849] ixgbevf 0000:03:10.5: MAC: 4
[   19.364850] ixgbevf 0000:03:10.5: Intel(R) 82599 Virtual Function
[   19.364978] ixgbevf 0000:03:10.7: enabling device (0000 -> 0002)
[   19.366081] ixgbevf 0000:03:10.7: PF still in reset state.  Is the PF interface up?
[   19.366082] ixgbevf 0000:03:10.7: Assigning random MAC address
[   19.366202] ixgbevf 0000:03:10.7: de:0c:6b:25:63:08
[   19.366203] ixgbevf 0000:03:10.7: MAC: 4
[   19.366203] ixgbevf 0000:03:10.7: Intel(R) 82599 Virtual Function
[   19.366329] ixgbevf 0000:03:11.1: enabling device (0000 -> 0002)
[   19.367439] ixgbevf 0000:03:11.1: PF still in reset state.  Is the PF interface up?
[   19.367440] ixgbevf 0000:03:11.1: Assigning random MAC address
[   19.367568] ixgbevf 0000:03:11.1: 4a:fe:8f:ee:a9:1b
[   19.367569] ixgbevf 0000:03:11.1: MAC: 4
[   19.367569] ixgbevf 0000:03:11.1: Intel(R) 82599 Virtual Function
[   19.367697] ixgbevf 0000:03:11.3: enabling device (0000 -> 0002)
[   19.368804] ixgbevf 0000:03:11.3: PF still in reset state.  Is the PF interface up?
[   19.368804] ixgbevf 0000:03:11.3: Assigning random MAC address
[   19.368925] ixgbevf 0000:03:11.3: ce:85:10:5b:70:dc
[   19.368925] ixgbevf 0000:03:11.3: MAC: 4
[   19.368926] ixgbevf 0000:03:11.3: Intel(R) 82599 Virtual Function
[   19.369050] ixgbevf 0000:03:11.5: enabling device (0000 -> 0002)
[   19.370171] ixgbevf 0000:03:11.5: PF still in reset state.  Is the PF interface up?
[   19.370172] ixgbevf 0000:03:11.5: Assigning random MAC address
[   19.370308] ixgbevf 0000:03:11.5: a6:60:df:d0:d1:34
[   19.370309] ixgbevf 0000:03:11.5: MAC: 4
[   19.370310] ixgbevf 0000:03:11.5: Intel(R) 82599 Virtual Function
[   19.370433] ixgbevf 0000:03:11.7: enabling device (0000 -> 0002)
[   19.371535] ixgbevf 0000:03:11.7: PF still in reset state.  Is the PF interface up?
[   19.371536] ixgbevf 0000:03:11.7: Assigning random MAC address
[   19.371658] ixgbevf 0000:03:11.7: d2:bc:8a:08:3a:8a
[   19.371659] ixgbevf 0000:03:11.7: MAC: 4
[   19.371660] ixgbevf 0000:03:11.7: Intel(R) 82599 Virtual Function
[   19.371673] ixgbe 0000:03:00.1 0000:03:00.1 (uninitialized): SR-IOV enabled with 8 VFs
[   19.748219] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[   19.814547] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[   19.814549] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[   19.814549] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[   19.814550] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[   19.814551] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
[   19.858764] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000
[   19.858765] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[   21.708071] ixgbe 0000:03:00.1 eth6: IOV is enabled with 8 VFs
[   21.727074] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[   21.727153] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)
[   21.728283] ixgbevf 0000:03:10.0: PF still in reset state.  Is the PF interface up?
[   21.728284] ixgbevf 0000:03:10.0: Assigning random MAC address
[   22.123348] ixgbevf 0000:03:10.0: 8e:0c:94:c5:aa:f0
[   22.123349] ixgbevf 0000:03:10.0: MAC: 4
[   22.123349] ixgbevf 0000:03:10.0: Intel(R) 82599 Virtual Function
[   22.123360] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)
[   22.124467] ixgbevf 0000:03:10.2: PF still in reset state.  Is the PF interface up?
[   22.124468] ixgbevf 0000:03:10.2: Assigning random MAC address
[   22.124597] ixgbevf 0000:03:10.2: 9e:a9:c2:aa:1e:18
[   22.124597] ixgbevf 0000:03:10.2: MAC: 4
[   22.124598] ixgbevf 0000:03:10.2: Intel(R) 82599 Virtual Function
[   22.124605] ixgbevf 0000:03:10.4: enabling device (0000 -> 0002)
[   22.125706] ixgbevf 0000:03:10.4: PF still in reset state.  Is the PF interface up?
[   22.125706] ixgbevf 0000:03:10.4: Assigning random MAC address
[   22.125832] ixgbevf 0000:03:10.4: 6e:c3:69:e7:ff:4e
[   22.125832] ixgbevf 0000:03:10.4: MAC: 4
[   22.125833] ixgbevf 0000:03:10.4: Intel(R) 82599 Virtual Function
[   22.125840] ixgbevf 0000:03:10.6: enabling device (0000 -> 0002)
[   22.126941] ixgbevf 0000:03:10.6: PF still in reset state.  Is the PF interface up?
[   22.126942] ixgbevf 0000:03:10.6: Assigning random MAC address
[   22.943349] ixgbevf 0000:03:10.6: 86:df:7b:b6:81:e4
[   22.943350] ixgbevf 0000:03:10.6: MAC: 4
[   22.943351] ixgbevf 0000:03:10.6: Intel(R) 82599 Virtual Function
[   22.943362] ixgbevf 0000:03:11.0: enabling device (0000 -> 0002)
[   22.944516] ixgbevf 0000:03:11.0: PF still in reset state.  Is the PF interface up?
[   22.944517] ixgbevf 0000:03:11.0: Assigning random MAC address
[   22.944643] ixgbevf 0000:03:11.0: a6:41:27:17:7b:f5
[   22.944643] ixgbevf 0000:03:11.0: MAC: 4
[   22.944644] ixgbevf 0000:03:11.0: Intel(R) 82599 Virtual Function
[   22.944649] ixgbevf 0000:03:11.2: enabling device (0000 -> 0002)
[   22.945749] ixgbevf 0000:03:11.2: PF still in reset state.  Is the PF interface up?
[   22.945750] ixgbevf 0000:03:11.2: Assigning random MAC address
[   22.945863] ixgbevf 0000:03:11.2: d6:c7:c2:9f:a3:97
[   22.945864] ixgbevf 0000:03:11.2: MAC: 4
[   22.945865] ixgbevf 0000:03:11.2: Intel(R) 82599 Virtual Function
[   22.945871] ixgbevf 0000:03:11.4: enabling device (0000 -> 0002)
[   22.946978] ixgbevf 0000:03:11.4: PF still in reset state.  Is the PF interface up?
[   22.946979] ixgbevf 0000:03:11.4: Assigning random MAC address
[   22.947093] ixgbevf 0000:03:11.4: 42:3f:49:4f:8f:98
[   22.947094] ixgbevf 0000:03:11.4: MAC: 4
[   22.947094] ixgbevf 0000:03:11.4: Intel(R) 82599 Virtual Function
[   22.947100] ixgbevf 0000:03:11.6: enabling device (0000 -> 0002)
[   22.948245] ixgbevf 0000:03:11.6: PF still in reset state.  Is the PF interface up?
[   22.948246] ixgbevf 0000:03:11.6: Assigning random MAC address
[   22.948354] ixgbevf 0000:03:11.6: 72:02:ed:13:7f:05
[   22.948355] ixgbevf 0000:03:11.6: MAC: 4
[   22.948355] ixgbevf 0000:03:11.6: Intel(R) 82599 Virtual Function
[   23.162283] ixgbevf 0000:03:11.7 em2_7: renamed from eth10
[   23.979141] ixgbevf 0000:03:11.6 em1_7: renamed from eth17
[   34.531118] ixgbe 0000:03:00.1 rename27: renamed from eth6
[   35.502452] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet

And successfully pass to a PFSense VM
Xshot-0041.png

However, if I use PVE 4.0 on the same hardware, I got

Code: 
root@pve:~# uname -r
4.2.2-1-pve

root@pve:~# dmesg|grep ixgbe
[    4.058315] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.058324] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[    4.058331] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    4.080548] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
[    6.006001] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
[    6.006010] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[    6.006016] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    6.028236] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
[    9.621857] ixgbe 0000:03:00.1 rename9: renamed from eth7
[    9.674634] ixgbe 0000:03:00.0 rename8: renamed from eth6
[    9.810747] ixgbe 0000:03:00.1 eth3: renamed from rename9
[    9.866589] ixgbe 0000:03:00.0 eth2: renamed from rename8
[   11.495314] ixgbe 0000:03:00.0: registered PHC device on eth2
[   12.004502] ixgbe 0000:03:00.1: registered PHC device on eth3

root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
-bash: echo: [COLOR=#ff0000]write error: Operation not supported[/COLOR]

root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
pci:ixgbe

root@pve:~# ls /sys/class/net/eth2/device/driver
[COLOR=#ff0000]0000:03:00.0  0000:03:00.1[/COLOR]  bind  module  new_id  remove_id  uevent  unbind

root@pve:~# lspci -s 03:00.0 -vnn
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
        Subsystem: Super Micro Computer Inc Device [15d9:15ad]
        Physical Slot: 0-1
        Flags: bus master, fast devsel, latency 0, IRQ 25
        Memory at f9a00000 (64-bit, prefetchable) [size=2M]
        Memory at f9c04000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at fb180000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        [COLOR=#ff0000]Capabilities: [160] Single Root I/O Virtualization (SR-IOV)[/COLOR]
        Capabilities: [1b0] Access Control Services
        Capabilities: [1c0] Latency Tolerance Reporting
        Kernel driver in use: ixgbe

The other Intel i350 nic on the same board is capable of SR_IOV on both OS and both kernel and can be successfully passthrough
 
with PVE 4.0 and kernel 4.2.3 failed, complained SR_IOV is not supported

Code: 
root@pve:~# uname -r
4.2.3-1-pve

root@pve:~# dmesg|grep ixgbe
[    4.070688] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.070692] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[    4.070695] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    4.092878] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
[    6.022409] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
[    6.022412] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[    6.022414] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    6.044585] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
[   10.199799] ixgbe 0000:03:00.1 rename9: renamed from eth7
[   10.334923] ixgbe 0000:03:00.0 rename8: renamed from eth6
[   10.443240] ixgbe 0000:03:00.1 eth3: renamed from rename9
[   10.475002] ixgbe 0000:03:00.0 eth2: renamed from rename8
[   12.202005] ixgbe 0000:03:00.0: registered PHC device on eth2
[   12.701014] ixgbe 0000:03:00.1: registered PHC device on eth3

root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
-bash: echo: write error: Operation not supported
root@pve:~# dmesg|grep SRIOV
[ 1875.989285] ixgbe 0000:03:00.0: SRIOV not supported on this device

root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
pci:ixgbe

root@pve:~# ls /sys/class/net/eth2/device/driver
0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind

root@pve:~# lspci -s 03:00.0 -vnn
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
  Subsystem: Super Micro Computer Inc Device [15d9:15ad]
  Physical Slot: 0-1
  Flags: bus master, fast devsel, latency 0, IRQ 25
  Memory at f9a00000 (64-bit, prefetchable) [size=2M]
  Memory at f9c04000 (64-bit, prefetchable) [size=16K]
  Expansion ROM at fb180000 [disabled] [size=512K]
  Capabilities: [40] Power Management version 3
  Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
  Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
  Capabilities: [a0] Express Endpoint, MSI 00
  Capabilities: [e0] Vital Product Data
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
  Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
  Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
  Capabilities: [1b0] Access Control Services
  Capabilities: [1c0] Latency Tolerance Reporting
  Kernel driver in use: ixgbe

Code: 
root@pve:~# modinfo ixgbe
filename:       /lib/modules/4.2.3-1-pve/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        4.1.5
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
...

I think the issue is ixgbe version. I am able to successful to enable SR_IOV is 4.0.1-k using ubuntu 4.2.3, and PVE4 is 4.1.5. Very likely it is Intel purposely disable the driver from access SR_IOV function when using Xeon 1500D processors
 
Last edited:
From the ixgbe docs, there is a parameter to enable SR-IOV

Code: 
max_vfs
-------
Valid Range:    1-63


Default Value:  0


If the value is greater than 0, it will also force the VMDq parameter to 1 or
more.


This parameter adds support for SR-IOV. It causes the driver to spawn up to
max_vfs worth of virtual function.

maybe that helps?
 
From the source code, there is a hint:

Code: 
        e_dev_warn("Enabling SR-IOV VFs using the max_vfs module parameter is deprecated.\n");
        e_dev_warn("Please use the pci sysfs interface instead. Ex:\n");
        e_dev_warn("echo '%d' > /sys/bus/pci/devices/%04x:%02x:%02x.%1x/sriov_numvfs\n",

does that help?
 
dietmar said:
From the source code, there is a hint:

Code: 
        e_dev_warn("Enabling SR-IOV VFs using the max_vfs module parameter is deprecated.\n");
        e_dev_warn("Please use the pci sysfs interface instead. Ex:\n");
        e_dev_warn("echo '%d' > /sys/bus/pci/devices/%04x:%02x:%02x.%1x/sriov_numvfs\n",

does that help?
Click to expand...
I will try from system pci bus instead of system net class

Sent from my Nexus 6 using Tapatalk
 
tried

root@pve-kac:/sys/bus/pci/devices/0000:03:00.0# cat sriov_totalvfs
64
root@pve-kac:/sys/bus/pci/devices/0000:03:00.0# echo 4 > sriov_numvfs
-bash: echo: write error: Operation not supported

still failed.. it must be 4.1.5 driver code issue
 
You must log in or register to reply here.
Top Bottom