Jumbo Frames not work after upgrade pve-kernel to 4.13.8-2

May 1, 2015
27
0
21
45
Poland
Hi.

On my hardware CEPH fail after upgrade pve-kernel from 4.13.4-1 to 4.13.8-2 because MTU 9000 stop working.

"ip link" shows proper MTU but "ping -M do" with packet larger than 1500 doesn't work, after rollback kernel version all forks fine!

I have bonded interfaces on Intel 82580 cards.

Code:
proxmox-ve: 5.1-28 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.13.8-2-pve: 4.13.8-28
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9
ceph: 12.2.1-pve3
 
that's strange, as the actual maximum MTU was increased with the recent driver update.. I'll check whether we can reproduce this in our test lab..
 
This might be bug in igb driver:
Code:
version:        5.3.5.12
srcversion:     D735972E4CD19103A136184
vermagic:       4.13.8-2-pve SMP mod_unload modversions

Working one:
Code:
version:        5.3.5.10
srcversion:     B708B1E560AB2E8BECA698C
vermagic:       4.13.4-1-pve SMP mod_unload modversions
 
We are running legacy non VLAN aware Linux bridge (aka not OVS) with two active/backup bond interfaces:
  • bond0 comprising of 2 x 10GbE Intel X540-AT2 (VM traffic)
  • bond1 comprising of 2 x 10GbSFP+ Intel 82599ES (Ceph traffic)

All 4 interfaces utilise Intel ixgbe driver v5.3.3, eg:
Code:
[root@kvm5a ~]# ethtool -i eth2
driver: ixgbe
version: 5.3.3
firmware-version: 0x8000047d
expansion-rom-version:
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Pings work as expected:
Code:
[root@kvm5a ~]# ping -M do 10.254.1.3 -s 9188
PING 10.254.1.3 (10.254.1.3) 9188(9216) bytes of data.
9196 bytes from 10.254.1.3: icmp_seq=1 ttl=64 time=0.135 ms
^C
--- 10.254.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.135/0.135/0.135/0.000 ms

[root@kvm5a ~]# ping -M do 198.19.17.67 -s 9188
PING 198.19.17.67 (198.19.17.67) 9188(9216) bytes of data.
9196 bytes from 198.19.17.67: icmp_seq=1 ttl=64 time=0.203 ms
9196 bytes from 198.19.17.67: icmp_seq=2 ttl=64 time=0.175 ms
^C
--- 198.19.17.67 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.175/0.189/0.203/0.014 ms

[root@kvm5a ~]# ping -M do 10.254.1.3 -s 9190
PING 10.254.1.3 (10.254.1.3) 9190(9218) bytes of data.
ping: local error: Message too long, mtu=9216
^C
--- 10.254.1.3 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms


Our network interface configuration file:
Code:
/etc/network/interfaces
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
        slaves eth0,eth1
        bond_miimon 100
        bond_mode active-backup
        mtu 9216

auto bond1
iface bond1 inet static
        address 10.254.1.2
        netmask  255.255.255.0
        slaves eth2,eth3
        bond_miimon 100
        bond_mode active-backup
        mtu 9216

auto eth0
iface eth0 inet manual
        bond-master bond0
        bond-primary eth0
        mtu 9216

auto eth1
iface eth1 inet manual
        bond-master bond0
        mtu 9216

auto eth2
iface eth2 inet manual
        bond-master bond1
        mtu 9216

auto eth3
iface eth3 inet manual
        bond-master bond1
        bond-primary eth3
        mtu 9216

auto vmbr0
iface vmbr0 inet static
        address 198.19.17.66
        netmask 255.255.255.224
        gateway 198.19.17.65
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        mtu 9216
 
ixgbe (2 x 10Gbit) works for me too (without bonding), I have only problem on igb (4 x 1Gbit and 2 x 1Gbit) NICs

ethtool info on both NICs shows:
Code:
firmware-version: 3.19, 0x00013bdb
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
 
Last edited:
we can reproduce the issue with igb in our testlab! thanks for reporting it - we'll either fix it in .12 or revert to .10 with our own MTU patch.
 
pve-kernel-4.13.8-3-pve in version 4.13.8-30 (available in pvetest and pve-no-subscription) fixes the MTU issue by reverting to the previous driver version with our in-tree based MTU patch, pending further investigation.
 
And stop work queues on Intel I350 :(

could you be a bit more specific? which kernel version breaks which feature? the only difference between igb 5.3.5.10 and .12 is how the MTU limits are setup and checked..
 
Hello,
sorry for bringing up this old topic.
I just ran across the same Intel i350 issue: MSI-X stopped working and falls back to MSI, which also doesn't allow to configure queues larger than 1.

PVE 4.13.13-36
driver: igb
version: 5.3.5.10
 
still on the TODO list unfortunately, the whole KPTI/Spectre/Meltdown mess put this onto the back burner. let's hope there's some time now that things have slowed down a bit again!
 
  • Like
Reactions: robertb
Good morning,

I just confirmed that by manually compiling and inserting igb driver 5.3.5.15, the queues will start to be functional again.
Incase someone relies on this: igb driver modules can be replaced live with minimal impact (short packet loss), no reboot required.
 
Good morning,

I just confirmed that by manually compiling and inserting igb driver 5.3.5.15, the queues will start to be functional again.
Incase someone relies on this: igb driver modules can be replaced live with minimal impact (short packet loss), no reboot required.

the 4.15 preview kernel from pvetest (which uses the in-tree drivers) supports both MTU setting and multiple queues as well.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!