Cannot use SR-IOV on proxmox

Discussion in 'Proxmox VE: Networking and Firewall' started by sili, Apr 22, 2016.

  1. sili

    sili New Member

    Joined:
    Dec 21, 2015
    Messages:
    28
    Likes Received:
    2
    Hi,
    I have upgrade my proxmox
    but before upgrade, SR-IOV can't use know

    Here is my pveversion
    Code:
    root@pve:~#  pveversion --verbose
    proxmox-ve: 4.1-48 (running kernel: 4.4.6-1-pve)
    pve-manager: 4.1-33 (running version: 4.1-33/de386c1a)
    pve-kernel-4.4.6-1-pve: 4.4.6-48
    pve-kernel-4.2.8-1-pve: 4.2.8-41
    pve-kernel-4.2.2-1-pve: 4.2.2-16
    lvm2: 2.02.116-pve2
    corosync-pve: 2.3.5-2
    libqb0: 1.0-1
    pve-cluster: 4.0-39
    qemu-server: 4.0-71
    pve-firmware: 1.1-8
    libpve-common-perl: 4.0-59
    libpve-access-control: 4.0-16
    libpve-storage-perl: 4.0-50
    pve-libspice-server1: 0.12.5-2
    vncterm: 1.2-1
    pve-qemu-kvm: 2.5-13
    pve-container: 1.0-61
    pve-firewall: 2.0-25
    pve-ha-manager: 1.0-28
    ksm-control-daemon: 1.2-1
    glusterfs-client: 3.5.2-2+deb8u1
    lxc-pve: 1.1.5-7
    lxcfs: 2.0.0-pve2
    cgmanager: 0.39-pve1
    criu: 1.6.0-1
    zfsutils: 0.6.5-pve9~jessie
    openvswitch-switch: 2.3.2-3
    and I use i350 to do SR-IOV
    Code:
    root@pve:~# lspci |grep Eth
    03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
    03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
    05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    
    Code:
    root@pve:~# lspci -vnnk -s 07:00.1
    07:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
      Subsystem: Super Micro Computer Inc Device [15d9:1521]
      Flags: bus master, fast devsel, latency 0
      Memory at fba00000 (32-bit, non-prefetchable) [size=128K]
      I/O ports at d000 [size=32]
      Memory at fba40000 (32-bit, non-prefetchable) [size=16K]
      Capabilities: [40] Power Management version 3
      Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
      Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
      Capabilities: [a0] Express Endpoint, MSI 00
      Capabilities: [100] Advanced Error Reporting
      Capabilities: [140] Device Serial Number 0c-c4-7a-ff-ff-c3-9f-06
      Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
      Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
      Capabilities: [1a0] Transaction Processing Hints
      Capabilities: [1d0] Access Control Services
      Kernel driver in use: igb
    
    When I do SR-IOV, I'll get this error
    Code:
    root@pve:~# echo 2 > /sys/class/net/eth0/device/sriov_numvfs
    -bash: echo: write error: Function not implemented
    
     
  2. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,446
    Likes Received:
    304
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. sili

    sili New Member

    Joined:
    Dec 21, 2015
    Messages:
    28
    Likes Received:
    2
    hi, I have report bug on sourceforge,
    They was replay me this message
    "Please contact proxmox as this appears to be a kernel error."
     
  4. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,446
    Likes Received:
    304
    I cannot find your bug report there - please can you send me a link?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. sili

    sili New Member

    Joined:
    Dec 21, 2015
    Messages:
    28
    Likes Received:
    2
  6. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,446
    Likes Received:
    304
    That, I will try to clarify the issue there. Please can you test if setting the suggested module flag work?
    In /etc/modules, set

    igb max_vfs=2,2
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,446
    Likes Received:
    304
    I assume the file /sys/class/net/eth0/device/sriov_numvfs does not exist?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. sili

    sili New Member

    Joined:
    Dec 21, 2015
    Messages:
    28
    Likes Received:
    2
    Hi, igb sriov is very Important for me.
    If I am try igb after, I'll post reply on this


    but I'm try my X552, It's different is X552 use ixgbe driver
    I think is maybe the same problem

    this is my friend help me to post this
    https://forum.proxmox.com/threads/p...541s-soc-x552-x557-sr-iov-issue-failed.26666/


    Code:
    
    root@pve:~# pveversion --verbose
    proxmox-ve: 4.1-41 (running kernel: 4.2.8-1-pve)
    pve-manager: 4.1-22 (running version: 4.1-22/aca130cf)
    pve-kernel-4.2.8-1-pve: 4.2.8-41
    pve-kernel-4.2.2-1-pve: 4.2.2-16
    lvm2: 2.02.116-pve2
    corosync-pve: 2.3.5-2
    libqb0: 1.0-1
    pve-cluster: 4.0-36
    qemu-server: 4.0-64
    pve-firmware: 1.1-7
    libpve-common-perl: 4.0-54
    libpve-access-control: 4.0-13
    libpve-storage-perl: 4.0-45
    pve-libspice-server1: 0.12.5-2
    vncterm: 1.2-1
    pve-qemu-kvm: 2.5-9
    pve-container: 1.0-52
    pve-firewall: 2.0-22
    pve-ha-manager: 1.0-25
    ksm-control-daemon: 1.2-1
    glusterfs-client: 3.5.2-2+deb8u1
    lxc-pve: 1.1.5-7
    lxcfs: 2.0.0-pve2
    cgmanager: 0.39-pve1
    criu: 1.6.0-1
    zfsutils: 0.6.5-pve7~jessie
    openvswitch-switch: 2.3.2-2
    
    on proxmox-ve: 4.1-41 (running kernel: 4.2.8-1-pve), I can do SR-IOV in my intel i-350(use igb)

    Code:
    root@pve:~# modinfo ixgbe |grep vf
    parm:           max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
    root@pve:~# modinfo igb |grep vf
    parm:           max_vfs:Maximum number of virtual functions to allocate per physical function (uint)
    
    I try to compiler new ixgbe driver, but SR-IOV It's not work
    after sourceforge reply, I'm try to do ixgbe SR-IOV

    command is this, then I can use SR-IOV in X552
    Code:
    rmmod ixgbe
    modprobe ixgbe max_vfs=4,4
    
    [​IMG]
     
  9. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,446
    Likes Received:
    304
    It gets complicated and confusing if you start mixing different things in a single thread...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. sili

    sili New Member

    Joined:
    Dec 21, 2015
    Messages:
    28
    Likes Received:
    2
    Yes, It's not exist.
    Code:
    root@pve:~# lspci |grep Eth
    03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
    03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
    05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    
    Code:
    root@pve:/sys/class/net# ls -al
    total 0
    drwxr-xr-x  2 root root 0 Apr 26 13:46 .
    drwxr-xr-x 71 root root 0 Apr 26 13:46 ..
    lrwxrwxrwx  1 root root 0 Apr 26 14:01 bond0 -> ../../devices/virtual/net/bond0
    lrwxrwxrwx  1 root root 0 Apr 26 13:46 eth0 -> ../../devices/pci0000:00/0000:00:02.2/0000:03:00.1/net/eth0
    lrwxrwxrwx  1 root root 0 Apr 26 14:01 eth2 -> ../../devices/pci0000:00/0000:00:1c.0/0000:07:00.0/net/eth2
    lrwxrwxrwx  1 root root 0 Apr 26 13:46 eth3 -> ../../devices/pci0000:00/0000:00:1c.0/0000:07:00.1/net/eth3
    lrwxrwxrwx  1 root root 0 Apr 26 13:46 eth6 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.1/net/eth6
    lrwxrwxrwx  1 root root 0 Apr 26 13:46 eth7 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.0/net/eth7
    
    Code:
    root@pve:/sys/class/net/eth3# ls
    addr_assign_type  broadcast  device  dormant  gro_flush_timeout  iflink  name_assign_type  phys_port_id  power  speed  tx_queue_len
    address  carrier  dev_id  duplex  ifalias  link_mode  netdev_group  phys_port_name  proto_down  statistics  type
    addr_len  carrier_changes  dev_port  flags  ifindex  mtu  operstate  phys_switch_id  queues  subsystem  uevent
    root@pve:/sys/class/net/eth2# ls
    addr_assign_type  carrier  dev_port  gro_flush_timeout  link_mode  netdev_group  phys_switch_id  speed  type
    address  carrier_changes  dormant  ifalias  master  operstate  power  statistics  uevent
    addr_len  device  duplex  ifindex  mtu  phys_port_id  proto_down  subsystem  upper_ovs-system
    broadcast  dev_id  flags  iflink  name_assign_type  phys_port_name  queues  tx_queue_len
    
     
  11. sili

    sili New Member

    Joined:
    Dec 21, 2015
    Messages:
    28
    Likes Received:
    2
    Hello, I know how to do SR-IOV...
    I tried/etc/modules, set igb max_vfs=2,2 but didn't work, but it hints me towards another direction make it worked.

    I used following setting and made intel i350 sr-iov enabled.
    Code:
    echo "options igb max_vfs=2,2" > /etc/modprobe.d/igb.conf
    depmod -a
    update-initramfs -u
    
    Then Reboot

    Code:
    root@pve:~# lspci |grep Eth
    03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
    03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
    05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
    08:10.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
    08:10.1 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
    08:10.4 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
    08:10.5 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
    
    thanks for all the helps
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice