TSO offloading problem with igb whilst, ixgbe is fine

Discussion in 'Proxmox VE: Networking and Firewall' started by David Herselman, Aug 12, 2019.

  1. David Herselman

    David Herselman Active Member
    Proxmox Subscriber

    Joined:
    Jun 8, 2016
    Messages:
    198
    Likes Received:
    39
    Most of our Proxmox clusters utilise Intel 82599ES 10GbE SFP+ NICs where TSO (TCP/IP Segmentation Offloading) works as expected for VMs where their VirtIO NIC then also has TSO enabled.

    We however have a cluster made up of 3 x Lenovo RD350 servers where Intel 82599ES 10GbE NICs are used for Ceph replication traffic and VMs are bridged using LACP bonds made up of Intel i210 1GbE NICs.

    Is this something I should raise with Lenovo or is this a kernel / driver issue?

    From the host:
    [root@kvm6a ~]# ethtool -k eth0 | grep segmentation
    tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp-mangleid-segmentation: off
    tx-tcp6-segmentation: on
    generic-segmentation-offload: on
    tx-fcoe-segmentation: off [fixed]
    tx-gre-segmentation: on
    tx-gre-csum-segmentation: on
    tx-ipxip4-segmentation: on
    tx-ipxip6-segmentation: on
    tx-udp_tnl-segmentation: on
    tx-udp_tnl-csum-segmentation: on
    tx-sctp-segmentation: off [fixed]
    tx-esp-segmentation: off [fixed]
    tx-udp-segmentation: off [fixed]

    [root@kvm6a ~]# ethtool -i eth0
    driver: igb
    version: 5.4.0-k
    firmware-version: 3.31, 0x800005cc
    expansion-rom-version:
    bus-info: 0000:06:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes

    From a Debian 10 VM:
    [admin@debian10vm ~]# ethtool -k eth0 | grep segmentation
    tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: on
    tx-tcp-mangleid-segmentation: off
    tx-tcp6-segmentation: on

    generic-segmentation-offload: on
    tx-fcoe-segmentation: off [fixed]
    tx-gre-segmentation: off [fixed]
    tx-gre-csum-segmentation: off [fixed]
    tx-ipxip4-segmentation: off [fixed]
    tx-ipxip6-segmentation: off [fixed]
    tx-udp_tnl-segmentation: off [fixed]
    tx-udp_tnl-csum-segmentation: off [fixed]
    tx-sctp-segmentation: off [fixed]
    tx-esp-segmentation: off [fixed]
    tx-udp-segmentation: off [fixed]

    Speedtest.net result:
    [admin@debian10vm ~]# ./speedtest_cli.py --server 23339
    Retrieving speedtest.net configuration...
    Retrieving speedtest.net server list...
    Testing from Syrex (Pty) Ltd (41.79.21.90)...
    Hosted by Syrex (Johannesburg) [3.18 km]: 1.932 ms
    Testing download speed........................................
    Download: 193.19 Mbit/s
    Testing upload speed..................................................
    Upload: 17.55 Mbit/s

    If we disable TSO on the guest:
    [admin@debian10vm ~]# ethtool -K eth0 tso off
    [admin@debian10vm ~]# ethtool -k eth0 | grep segmentation
    tcp-segmentation-offload: off
    tx-tcp-segmentation: off
    tx-tcp-ecn-segmentation: off
    tx-tcp-mangleid-segmentation: off
    tx-tcp6-segmentation: off

    generic-segmentation-offload: on
    tx-fcoe-segmentation: off [fixed]
    tx-gre-segmentation: off [fixed]
    tx-gre-csum-segmentation: off [fixed]
    tx-ipxip4-segmentation: off [fixed]
    tx-ipxip6-segmentation: off [fixed]
    tx-udp_tnl-segmentation: off [fixed]
    tx-udp_tnl-csum-segmentation: off [fixed]
    tx-sctp-segmentation: off [fixed]
    tx-esp-segmentation: off [fixed]
    tx-udp-segmentation: off [fixed]

    [admin@debian10vm ~]# ./speedtest_cli.py --server 23339
    Retrieving speedtest.net configuration...
    Retrieving speedtest.net server list...
    Testing from Syrex (Pty) Ltd (41.79.21.90)...
    Hosted by Syrex (Johannesburg) [3.18 km]: 1.844 ms
    Testing download speed........................................
    Download: 191.94 Mbit/s
    Testing upload speed..................................................
    Upload: 523.06 Mbit/s


    We are able to utilise all hardware acceleration offloading capabilities when using ixgbe. We are using Open vSwitch (OvS) with the following network configuration:
    auto lo
    iface lo inet loopback

    allow-vmbr0 bond0
    iface bond0 inet manual
    ovs_bridge vmbr0
    ovs_type OVSBond
    ovs_bonds eth0 eth1
    pre-up ( ifconfig eth0 mtu 9216 && ifconfig eth1 mtu 9216 )
    ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast tag=1 vlan_mode=native-untagged
    mtu 9216

    auto vmbr0
    allow-ovs vmbr0
    iface vmbr0 inet manual
    ovs_type OVSBridge
    ovs_ports bond0 vlan1
    mtu 9216

    allow-vmbr0 vlan1
    iface vlan1 inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr0
    ovs_options tag=1
    ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
    address 10.19.14.34
    netmask 255.255.255.224
    gateway 10.19.14.33
    mtu 9216

    allow-vmbr1 bond1
    iface bond1 inet manual
    ovs_bridge vmbr1
    ovs_type OVSBond
    ovs_bonds eth2 eth3
    pre-up ( ifconfig eth2 mtu 9216 && ifconfig eth3 mtu 9216 )
    ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast tag=1 vlan_mode=native-untagged
    mtu 9216

    auto vmbr1
    allow-ovs vmbr1
    iface vmbr1 inet manual
    ovs_type OVSBridge
    ovs_ports bond1 vlan18
    mtu 9216

    allow-vmbr1 vlan18
    iface vlan18 inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr1
    ovs_options tag=18
    ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
    address 10.254.1.2
    netmask 255.255.255.0
    mtu 9212

    Interface vlan1 is attached to vmbr0, internet breakout and management of PVE cluster. Interface vlan18 is attached to vmbr1 and used for Ceph replication traffic. We typically attach guests to vmbr0 and experience the problem above, moving the guest to vmbr1 allows us to attain full speed with TSO enabled.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice