Jumbo Frames not work after upgrade pve-kernel to 4.13.8-2

Discussion in 'Proxmox VE: Installation and configuration' started by xcdr, Dec 5, 2017.

  1. xcdr

    xcdr New Member
    Proxmox Subscriber

    Joined:
    May 1, 2015
    Messages:
    11
    Likes Received:
    0
    Hi.

    On my hardware CEPH fail after upgrade pve-kernel from 4.13.4-1 to 4.13.8-2 because MTU 9000 stop working.

    "ip link" shows proper MTU but "ping -M do" with packet larger than 1500 doesn't work, after rollback kernel version all forks fine!

    I have bonded interfaces on Intel 82580 cards.

    Code:
    proxmox-ve: 5.1-28 (running kernel: 4.13.4-1-pve)
    pve-manager: 5.1-36 (running version: 5.1-36/131401db)
    pve-kernel-4.13.4-1-pve: 4.13.4-26
    pve-kernel-4.13.8-2-pve: 4.13.8-28
    libpve-http-server-perl: 2.0-6
    lvm2: 2.02.168-pve6
    corosync: 2.4.2-pve3
    libqb0: 1.0.1-1
    pve-cluster: 5.0-15
    qemu-server: 5.0-17
    pve-firmware: 2.0-3
    libpve-common-perl: 5.0-20
    libpve-guest-common-perl: 2.0-13
    libpve-access-control: 5.0-7
    libpve-storage-perl: 5.0-16
    pve-libspice-server1: 0.12.8-3
    vncterm: 1.5-2
    pve-docs: 5.1-12
    pve-qemu-kvm: 2.9.1-2
    pve-container: 2.0-17
    pve-firewall: 3.0-3
    pve-ha-manager: 2.0-3
    ksm-control-daemon: 1.2-2
    glusterfs-client: 3.8.8-1
    lxc-pve: 2.1.0-2
    lxcfs: 2.0.7-pve4
    criu: 2.11.1-1~bpo90
    novnc-pve: 0.6-4
    smartmontools: 6.5+svn4324-1
    zfsutils-linux: 0.7.3-pve1~bpo9
    ceph: 12.2.1-pve3
     
  2. elurex

    elurex Member
    Proxmox Subscriber

    Joined:
    Oct 28, 2015
    Messages:
    149
    Likes Received:
    4
  3. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    that's strange, as the actual maximum MTU was increased with the recent driver update.. I'll check whether we can reproduce this in our test lab..
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. xcdr

    xcdr New Member
    Proxmox Subscriber

    Joined:
    May 1, 2015
    Messages:
    11
    Likes Received:
    0
    This might be bug in igb driver:
    Code:
    version:        5.3.5.12
    srcversion:     D735972E4CD19103A136184
    vermagic:       4.13.8-2-pve SMP mod_unload modversions
    
    Working one:
    Code:
    version:        5.3.5.10
    srcversion:     B708B1E560AB2E8BECA698C
    vermagic:       4.13.4-1-pve SMP mod_unload modversions
    
     
  6. David Herselman

    David Herselman Active Member
    Proxmox Subscriber

    Joined:
    Jun 8, 2016
    Messages:
    188
    Likes Received:
    39
    We are running legacy non VLAN aware Linux bridge (aka not OVS) with two active/backup bond interfaces:
    • bond0 comprising of 2 x 10GbE Intel X540-AT2 (VM traffic)
    • bond1 comprising of 2 x 10GbSFP+ Intel 82599ES (Ceph traffic)

    All 4 interfaces utilise Intel ixgbe driver v5.3.3, eg:
    Code:
    [root@kvm5a ~]# ethtool -i eth2
    driver: ixgbe
    version: 5.3.3
    firmware-version: 0x8000047d
    expansion-rom-version:
    bus-info: 0000:02:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes

    Pings work as expected:
    Code:
    [root@kvm5a ~]# ping -M do 10.254.1.3 -s 9188
    PING 10.254.1.3 (10.254.1.3) 9188(9216) bytes of data.
    9196 bytes from 10.254.1.3: icmp_seq=1 ttl=64 time=0.135 ms
    ^C
    --- 10.254.1.3 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 0.135/0.135/0.135/0.000 ms
    
    [root@kvm5a ~]# ping -M do 198.19.17.67 -s 9188
    PING 198.19.17.67 (198.19.17.67) 9188(9216) bytes of data.
    9196 bytes from 198.19.17.67: icmp_seq=1 ttl=64 time=0.203 ms
    9196 bytes from 198.19.17.67: icmp_seq=2 ttl=64 time=0.175 ms
    ^C
    --- 198.19.17.67 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 0.175/0.189/0.203/0.014 ms
    
    [root@kvm5a ~]# ping -M do 10.254.1.3 -s 9190
    PING 10.254.1.3 (10.254.1.3) 9190(9218) bytes of data.
    ping: local error: Message too long, mtu=9216
    ^C
    --- 10.254.1.3 ping statistics ---
    1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

    Our network interface configuration file:
    Code:
    /etc/network/interfaces
    auto lo
    iface lo inet loopback
    
    auto bond0
    iface bond0 inet manual
            slaves eth0,eth1
            bond_miimon 100
            bond_mode active-backup
            mtu 9216
    
    auto bond1
    iface bond1 inet static
            address 10.254.1.2
            netmask  255.255.255.0
            slaves eth2,eth3
            bond_miimon 100
            bond_mode active-backup
            mtu 9216
    
    auto eth0
    iface eth0 inet manual
            bond-master bond0
            bond-primary eth0
            mtu 9216
    
    auto eth1
    iface eth1 inet manual
            bond-master bond0
            mtu 9216
    
    auto eth2
    iface eth2 inet manual
            bond-master bond1
            mtu 9216
    
    auto eth3
    iface eth3 inet manual
            bond-master bond1
            bond-primary eth3
            mtu 9216
    
    auto vmbr0
    iface vmbr0 inet static
            address 198.19.17.66
            netmask 255.255.255.224
            gateway 198.19.17.65
            bridge_ports bond0
            bridge_stp off
            bridge_fd 0
            mtu 9216
     
  7. xcdr

    xcdr New Member
    Proxmox Subscriber

    Joined:
    May 1, 2015
    Messages:
    11
    Likes Received:
    0
    ixgbe (2 x 10Gbit) works for me too (without bonding), I have only problem on igb (4 x 1Gbit and 2 x 1Gbit) NICs

    ethtool info on both NICs shows:
    Code:
    firmware-version: 3.19, 0x00013bdb
    expansion-rom-version:
    bus-info: 0000:05:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: no
    
     
    #7 xcdr, Dec 5, 2017
    Last edited: Dec 5, 2017
  8. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    we can reproduce the issue with igb in our testlab! thanks for reporting it - we'll either fix it in .12 or revert to .10 with our own MTU patch.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. xcdr

    xcdr New Member
    Proxmox Subscriber

    Joined:
    May 1, 2015
    Messages:
    11
    Likes Received:
    0
    OK, this is important for me on production environment.
     
  10. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    pve-kernel-4.13.8-3-pve in version 4.13.8-30 (available in pvetest and pve-no-subscription) fixes the MTU issue by reverting to the previous driver version with our in-tree based MTU patch, pending further investigation.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. un1x0d

    un1x0d New Member

    Joined:
    Mar 14, 2012
    Messages:
    25
    Likes Received:
    4
    And stop work queues on Intel I350 :(
     
    robertb likes this.
  12. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    could you be a bit more specific? which kernel version breaks which feature? the only difference between igb 5.3.5.10 and .12 is how the MTU limits are setup and checked..
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. un1x0d

    un1x0d New Member

    Joined:
    Mar 14, 2012
    Messages:
    25
    Likes Received:
    4
    https://bugzilla.proxmox.com/show_bug.cgi?id=1554
     
  14. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    robertb likes this.
  15. robertb

    robertb New Member

    Joined:
    Apr 4, 2017
    Messages:
    16
    Likes Received:
    0
    Hello,
    sorry for bringing up this old topic.
    I just ran across the same Intel i350 issue: MSI-X stopped working and falls back to MSI, which also doesn't allow to configure queues larger than 1.

    PVE 4.13.13-36
    driver: igb
    version: 5.3.5.10
     
  16. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    still on the TODO list unfortunately, the whole KPTI/Spectre/Meltdown mess put this onto the back burner. let's hope there's some time now that things have slowed down a bit again!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    robertb likes this.
  17. robertb

    robertb New Member

    Joined:
    Apr 4, 2017
    Messages:
    16
    Likes Received:
    0
    Good morning,

    I just confirmed that by manually compiling and inserting igb driver 5.3.5.15, the queues will start to be functional again.
    Incase someone relies on this: igb driver modules can be replaced live with minimal impact (short packet loss), no reboot required.
     
  18. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,270
    Likes Received:
    505
    the 4.15 preview kernel from pvetest (which uses the in-tree drivers) supports both MTU setting and multiple queues as well.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice