PVE4 kernel issue, sr_iov disabled on PV4 kernel

elurex

elurex

Active Member
Proxmox Subscriber
Oct 28, 2015
204
14
38
Taiwan
The same server hardware SuperServer 5028D-TN4T when install on Ubuntu 14.04.3 with kernel to 4.2.3 SR-IOV is successful with Intel X557 but failed on PVE 4.0 with kernel 4.2.2-1-pve, how to fix this issue?

I can enable SR_IOV on for my 10 GBe X557 nic and with successfully pci passthrough (iommu & vfio)
Code: 
root@pdc:~# uname -r
4.2.3-040203-generic

lspci -vnn | grep Ethernet
03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:00.1 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:10.0 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.1 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.2 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.3 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.4 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.5 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.6 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:10.7 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.0 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.1 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.2 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.3 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.4 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.5 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.6 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]
03:11.7 Ethernet controller [0200]: Intel Corporation Device [[COLOR=#ff0000]8086:15a8[/COLOR]]

root@pdc:~# cat /etc/modprobe.d/ixgbe.conf
options ixgbe max_vfs=8

The iommu group is seperated and ready to be use for kvm vfio pci passthrough

/sys/kernel/iommu_groups/45/devices/0000:03:10.0
/sys/kernel/iommu_groups/46/devices/0000:03:10.2
/sys/kernel/iommu_groups/47/devices/0000:03:10.4
/sys/kernel/iommu_groups/48/devices/0000:03:10.6
/sys/kernel/iommu_groups/49/devices/0000:03:11.0
/sys/kernel/iommu_groups/50/devices/0000:03:11.2
/sys/kernel/iommu_groups/51/devices/0000:03:11.4
/sys/kernel/iommu_groups/52/devices/0000:03:11.6
/sys/kernel/iommu_groups/53/devices/0000:03:10.1
/sys/kernel/iommu_groups/54/devices/0000:03:10.3
/sys/kernel/iommu_groups/55/devices/0000:03:10.5
/sys/kernel/iommu_groups/56/devices/0000:03:10.7
/sys/kernel/iommu_groups/57/devices/0000:03:11.1
/sys/kernel/iommu_groups/58/devices/0000:03:11.3
/sys/kernel/iommu_groups/59/devices/0000:03:11.5
/sys/kernel/iommu_groups/60/devices/0000:03:11.7

root@pdc:~# modinfo ixgbe
filename:       /lib/modules/4.2.3-040203-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        4.0.1-k
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     E08010C0D72963989E9A05E
alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*
....
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        mdio,ptp,dca,vxlan
intree:         Y
vermagic:       4.2.3-040203-generic SMP mod_unload modversions
signer:         Build time autogenerated kernel key
sig_key:        CE:E6:5B:C1:E3:07:7F:6F:F5:58:77:52:AC:20:3F:9D:65:F7:05:DA
sig_hashalgo:   sha512
parm:           max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)
parm:           debug:Debug level (0=none,...,16=all) (int)

root@pdc:~# dmesg|grep ixgbe
[    0.000000] Command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    7.208439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
[    7.208439] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[    7.208607] ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[    9.240064] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): SR-IOV enabled with 8 VFs
[    9.512691] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[    9.512692] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    9.611689] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[    9.678012] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
[    9.678013] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    9.678014] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[    9.678015] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    9.678015] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
[    9.722230] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
[    9.722231] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[   15.215889] ixgbe 0000:03:00.0 eth1: IOV is enabled with 8 VFs
[   15.238000] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[   15.238201] ixgbe 0000:03:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[   16.787177] ixgbevf 0000:03:10.1: enabling device (0000 -> 0002)
[   16.788293] ixgbevf 0000:03:10.1: PF still in reset state.  Is the PF interface up?
[   16.788293] ixgbevf 0000:03:10.1: Assigning random MAC address
[   18.431419] ixgbevf 0000:03:10.1: 86:c7:2f:a5:60:0c
[   18.431420] ixgbevf 0000:03:10.1: MAC: 4
[   18.431420] ixgbevf 0000:03:10.1: Intel(R) 82599 Virtual Function
[   18.431577] ixgbevf 0000:03:10.3: enabling device (0000 -> 0002)
[   18.432709] ixgbevf 0000:03:10.3: PF still in reset state.  Is the PF interface up?
[   18.432709] ixgbevf 0000:03:10.3: Assigning random MAC address
[   19.363463] ixgbevf 0000:03:10.3: 62:5a:23:07:bd:59
[   19.363464] ixgbevf 0000:03:10.3: MAC: 4
[   19.363465] ixgbevf 0000:03:10.3: Intel(R) 82599 Virtual Function
[   19.363609] ixgbevf 0000:03:10.5: enabling device (0000 -> 0002)
[   19.364718] ixgbevf 0000:03:10.5: PF still in reset state.  Is the PF interface up?
[   19.364719] ixgbevf 0000:03:10.5: Assigning random MAC address
[   19.364849] ixgbevf 0000:03:10.5: 92:65:d9:a2:cd:48
[   19.364849] ixgbevf 0000:03:10.5: MAC: 4
[   19.364850] ixgbevf 0000:03:10.5: Intel(R) 82599 Virtual Function
[   19.364978] ixgbevf 0000:03:10.7: enabling device (0000 -> 0002)
[   19.366081] ixgbevf 0000:03:10.7: PF still in reset state.  Is the PF interface up?
[   19.366082] ixgbevf 0000:03:10.7: Assigning random MAC address
[   19.366202] ixgbevf 0000:03:10.7: de:0c:6b:25:63:08
[   19.366203] ixgbevf 0000:03:10.7: MAC: 4
[   19.366203] ixgbevf 0000:03:10.7: Intel(R) 82599 Virtual Function
[   19.366329] ixgbevf 0000:03:11.1: enabling device (0000 -> 0002)
[   19.367439] ixgbevf 0000:03:11.1: PF still in reset state.  Is the PF interface up?
[   19.367440] ixgbevf 0000:03:11.1: Assigning random MAC address
[   19.367568] ixgbevf 0000:03:11.1: 4a:fe:8f:ee:a9:1b
[   19.367569] ixgbevf 0000:03:11.1: MAC: 4
[   19.367569] ixgbevf 0000:03:11.1: Intel(R) 82599 Virtual Function
[   19.367697] ixgbevf 0000:03:11.3: enabling device (0000 -> 0002)
[   19.368804] ixgbevf 0000:03:11.3: PF still in reset state.  Is the PF interface up?
[   19.368804] ixgbevf 0000:03:11.3: Assigning random MAC address
[   19.368925] ixgbevf 0000:03:11.3: ce:85:10:5b:70:dc
[   19.368925] ixgbevf 0000:03:11.3: MAC: 4
[   19.368926] ixgbevf 0000:03:11.3: Intel(R) 82599 Virtual Function
[   19.369050] ixgbevf 0000:03:11.5: enabling device (0000 -> 0002)
[   19.370171] ixgbevf 0000:03:11.5: PF still in reset state.  Is the PF interface up?
[   19.370172] ixgbevf 0000:03:11.5: Assigning random MAC address
[   19.370308] ixgbevf 0000:03:11.5: a6:60:df:d0:d1:34
[   19.370309] ixgbevf 0000:03:11.5: MAC: 4
[   19.370310] ixgbevf 0000:03:11.5: Intel(R) 82599 Virtual Function
[   19.370433] ixgbevf 0000:03:11.7: enabling device (0000 -> 0002)
[   19.371535] ixgbevf 0000:03:11.7: PF still in reset state.  Is the PF interface up?
[   19.371536] ixgbevf 0000:03:11.7: Assigning random MAC address
[   19.371658] ixgbevf 0000:03:11.7: d2:bc:8a:08:3a:8a
[   19.371659] ixgbevf 0000:03:11.7: MAC: 4
[   19.371660] ixgbevf 0000:03:11.7: Intel(R) 82599 Virtual Function
[   19.371673] ixgbe 0000:03:00.1 0000:03:00.1 (uninitialized): SR-IOV enabled with 8 VFs
[   19.748219] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[   19.814547] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[   19.814549] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[   19.814549] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[   19.814550] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[   19.814551] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
[   19.858764] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000
[   19.858765] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[   21.708071] ixgbe 0000:03:00.1 eth6: IOV is enabled with 8 VFs
[   21.727074] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[   21.727153] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)
[   21.728283] ixgbevf 0000:03:10.0: PF still in reset state.  Is the PF interface up?
[   21.728284] ixgbevf 0000:03:10.0: Assigning random MAC address
[   22.123348] ixgbevf 0000:03:10.0: 8e:0c:94:c5:aa:f0
[   22.123349] ixgbevf 0000:03:10.0: MAC: 4
[   22.123349] ixgbevf 0000:03:10.0: Intel(R) 82599 Virtual Function
[   22.123360] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)
[   22.124467] ixgbevf 0000:03:10.2: PF still in reset state.  Is the PF interface up?
[   22.124468] ixgbevf 0000:03:10.2: Assigning random MAC address
[   22.124597] ixgbevf 0000:03:10.2: 9e:a9:c2:aa:1e:18
[   22.124597] ixgbevf 0000:03:10.2: MAC: 4
[   22.124598] ixgbevf 0000:03:10.2: Intel(R) 82599 Virtual Function
[   22.124605] ixgbevf 0000:03:10.4: enabling device (0000 -> 0002)
[   22.125706] ixgbevf 0000:03:10.4: PF still in reset state.  Is the PF interface up?
[   22.125706] ixgbevf 0000:03:10.4: Assigning random MAC address
[   22.125832] ixgbevf 0000:03:10.4: 6e:c3:69:e7:ff:4e
[   22.125832] ixgbevf 0000:03:10.4: MAC: 4
[   22.125833] ixgbevf 0000:03:10.4: Intel(R) 82599 Virtual Function
[   22.125840] ixgbevf 0000:03:10.6: enabling device (0000 -> 0002)
[   22.126941] ixgbevf 0000:03:10.6: PF still in reset state.  Is the PF interface up?
[   22.126942] ixgbevf 0000:03:10.6: Assigning random MAC address
[   22.943349] ixgbevf 0000:03:10.6: 86:df:7b:b6:81:e4
[   22.943350] ixgbevf 0000:03:10.6: MAC: 4
[   22.943351] ixgbevf 0000:03:10.6: Intel(R) 82599 Virtual Function
[   22.943362] ixgbevf 0000:03:11.0: enabling device (0000 -> 0002)
[   22.944516] ixgbevf 0000:03:11.0: PF still in reset state.  Is the PF interface up?
[   22.944517] ixgbevf 0000:03:11.0: Assigning random MAC address
[   22.944643] ixgbevf 0000:03:11.0: a6:41:27:17:7b:f5
[   22.944643] ixgbevf 0000:03:11.0: MAC: 4
[   22.944644] ixgbevf 0000:03:11.0: Intel(R) 82599 Virtual Function
[   22.944649] ixgbevf 0000:03:11.2: enabling device (0000 -> 0002)
[   22.945749] ixgbevf 0000:03:11.2: PF still in reset state.  Is the PF interface up?
[   22.945750] ixgbevf 0000:03:11.2: Assigning random MAC address
[   22.945863] ixgbevf 0000:03:11.2: d6:c7:c2:9f:a3:97
[   22.945864] ixgbevf 0000:03:11.2: MAC: 4
[   22.945865] ixgbevf 0000:03:11.2: Intel(R) 82599 Virtual Function
[   22.945871] ixgbevf 0000:03:11.4: enabling device (0000 -> 0002)
[   22.946978] ixgbevf 0000:03:11.4: PF still in reset state.  Is the PF interface up?
[   22.946979] ixgbevf 0000:03:11.4: Assigning random MAC address
[   22.947093] ixgbevf 0000:03:11.4: 42:3f:49:4f:8f:98
[   22.947094] ixgbevf 0000:03:11.4: MAC: 4
[   22.947094] ixgbevf 0000:03:11.4: Intel(R) 82599 Virtual Function
[   22.947100] ixgbevf 0000:03:11.6: enabling device (0000 -> 0002)
[   22.948245] ixgbevf 0000:03:11.6: PF still in reset state.  Is the PF interface up?
[   22.948246] ixgbevf 0000:03:11.6: Assigning random MAC address
[   22.948354] ixgbevf 0000:03:11.6: 72:02:ed:13:7f:05
[   22.948355] ixgbevf 0000:03:11.6: MAC: 4
[   22.948355] ixgbevf 0000:03:11.6: Intel(R) 82599 Virtual Function
[   23.162283] ixgbevf 0000:03:11.7 em2_7: renamed from eth10
[   23.979141] ixgbevf 0000:03:11.6 em1_7: renamed from eth17
[   34.531118] ixgbe 0000:03:00.1 rename27: renamed from eth6
[   35.502452] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet

And successfully pass to a PFSense VM
Xshot-0041.png

However, if I use PVE 4.0 on the same hardware, I got

Code: 
root@pve:~# uname -r
4.2.2-1-pve

root@pve:~# dmesg|grep ixgbe
[    4.058315] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.058324] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[    4.058331] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    4.080548] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
[    6.006001] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
[    6.006010] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[    6.006016] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    6.028236] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
[    9.621857] ixgbe 0000:03:00.1 rename9: renamed from eth7
[    9.674634] ixgbe 0000:03:00.0 rename8: renamed from eth6
[    9.810747] ixgbe 0000:03:00.1 eth3: renamed from rename9
[    9.866589] ixgbe 0000:03:00.0 eth2: renamed from rename8
[   11.495314] ixgbe 0000:03:00.0: registered PHC device on eth2
[   12.004502] ixgbe 0000:03:00.1: registered PHC device on eth3

root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
-bash: echo: [COLOR=#ff0000]write error: Operation not supported[/COLOR]

root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
pci:ixgbe

root@pve:~# ls /sys/class/net/eth2/device/driver
[COLOR=#ff0000]0000:03:00.0  0000:03:00.1[/COLOR]  bind  module  new_id  remove_id  uevent  unbind

root@pve:~# lspci -s 03:00.0 -vnn
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
        Subsystem: Super Micro Computer Inc Device [15d9:15ad]
        Physical Slot: 0-1
        Flags: bus master, fast devsel, latency 0, IRQ 25
        Memory at f9a00000 (64-bit, prefetchable) [size=2M]
        Memory at f9c04000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at fb180000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        [COLOR=#ff0000]Capabilities: [160] Single Root I/O Virtualization (SR-IOV)[/COLOR]
        Capabilities: [1b0] Access Control Services
        Capabilities: [1c0] Latency Tolerance Reporting
        Kernel driver in use: ixgbe

The other Intel i350 nic on the same board is capable of SR_IOV on both OS and both kernel and can be successfully passthrough
 
with PVE 4.0 and kernel 4.2.3 failed, complained SR_IOV is not supported

Code: 
root@pve:~# uname -r
4.2.3-1-pve

root@pve:~# dmesg|grep ixgbe
[    4.070688] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.070692] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[    4.070695] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    4.092878] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
[    6.022409] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
[    6.022412] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[    6.022414] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    6.044585] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
[   10.199799] ixgbe 0000:03:00.1 rename9: renamed from eth7
[   10.334923] ixgbe 0000:03:00.0 rename8: renamed from eth6
[   10.443240] ixgbe 0000:03:00.1 eth3: renamed from rename9
[   10.475002] ixgbe 0000:03:00.0 eth2: renamed from rename8
[   12.202005] ixgbe 0000:03:00.0: registered PHC device on eth2
[   12.701014] ixgbe 0000:03:00.1: registered PHC device on eth3

root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
-bash: echo: write error: Operation not supported
root@pve:~# dmesg|grep SRIOV
[ 1875.989285] ixgbe 0000:03:00.0: SRIOV not supported on this device

root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
pci:ixgbe

root@pve:~# ls /sys/class/net/eth2/device/driver
0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind

root@pve:~# lspci -s 03:00.0 -vnn
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
  Subsystem: Super Micro Computer Inc Device [15d9:15ad]
  Physical Slot: 0-1
  Flags: bus master, fast devsel, latency 0, IRQ 25
  Memory at f9a00000 (64-bit, prefetchable) [size=2M]
  Memory at f9c04000 (64-bit, prefetchable) [size=16K]
  Expansion ROM at fb180000 [disabled] [size=512K]
  Capabilities: [40] Power Management version 3
  Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
  Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
  Capabilities: [a0] Express Endpoint, MSI 00
  Capabilities: [e0] Vital Product Data
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
  Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
  Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
  Capabilities: [1b0] Access Control Services
  Capabilities: [1c0] Latency Tolerance Reporting
  Kernel driver in use: ixgbe

Code: 
root@pve:~# modinfo ixgbe
filename:       /lib/modules/4.2.3-1-pve/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        4.1.5
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
...

I think the issue is ixgbe version. I am able to successful to enable SR_IOV is 4.0.1-k using ubuntu 4.2.3, and PVE4 is 4.1.5. Very likely it is Intel purposely disable the driver from access SR_IOV function when using Xeon 1500D processors
 
Last edited:
From the ixgbe docs, there is a parameter to enable SR-IOV

Code: 
max_vfs
-------
Valid Range:    1-63


Default Value:  0


If the value is greater than 0, it will also force the VMDq parameter to 1 or
more.


This parameter adds support for SR-IOV. It causes the driver to spawn up to
max_vfs worth of virtual function.

maybe that helps?
 
From the source code, there is a hint:

Code: 
        e_dev_warn("Enabling SR-IOV VFs using the max_vfs module parameter is deprecated.\n");
        e_dev_warn("Please use the pci sysfs interface instead. Ex:\n");
        e_dev_warn("echo '%d' > /sys/bus/pci/devices/%04x:%02x:%02x.%1x/sriov_numvfs\n",

does that help?
 
dietmar said:
From the source code, there is a hint:

Code: 
        e_dev_warn("Enabling SR-IOV VFs using the max_vfs module parameter is deprecated.\n");
        e_dev_warn("Please use the pci sysfs interface instead. Ex:\n");
        e_dev_warn("echo '%d' > /sys/bus/pci/devices/%04x:%02x:%02x.%1x/sriov_numvfs\n",

does that help?
Click to expand...
I will try from system pci bus instead of system net class

Sent from my Nexus 6 using Tapatalk
 
tried

root@pve-kac:/sys/bus/pci/devices/0000:03:00.0# cat sriov_totalvfs
64
root@pve-kac:/sys/bus/pci/devices/0000:03:00.0# echo 4 > sriov_numvfs
-bash: echo: write error: Operation not supported

still failed.. it must be 4.1.5 driver code issue
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back