Jumbo Frames not work after upgrade pve-kernel to 4.13.8-2

xcdr · Dec 5, 2017

Hi.

On my hardware CEPH fail after upgrade pve-kernel from 4.13.4-1 to 4.13.8-2 because MTU 9000 stop working.

"ip link" shows proper MTU but "ping -M do" with packet larger than 1500 doesn't work, after rollback kernel version all forks fine!

I have bonded interfaces on Intel 82580 cards.

Code:

proxmox-ve: 5.1-28 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.13.8-2-pve: 4.13.8-28
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9
ceph: 12.2.1-pve3

elurex · Dec 5, 2017

I think it just igb driver issue... you can compile your own igb driver from intel as a work around
.
https://downloadcenter.intel.com/do...-Based-Gigabit-Network-Connections-for-Linux-

e1000e-x.x.x.x.tar.gz driver should be used for your nic

fabian · Dec 5, 2017

that's strange, as the actual maximum MTU was increased with the recent driver update.. I'll check whether we can reproduce this in our test lab..

fabian · Dec 5, 2017

elurex said:
I think it just igb driver issue... you can compile your own igb driver from intel as a work around
.
https://downloadcenter.intel.com/do...-Based-Gigabit-Network-Connections-for-Linux-

e1000e-x.x.x.x.tar.gz driver should be used for your nic

first off, igb is the correct one for this card, not e1000e. second, this is the exact same driver version that is apparently causing the issue..

xcdr · Dec 5, 2017

This might be bug in igb driver:

Code:

version:        5.3.5.12
srcversion:     D735972E4CD19103A136184
vermagic:       4.13.8-2-pve SMP mod_unload modversions

Working one:

Code:

version:        5.3.5.10
srcversion:     B708B1E560AB2E8BECA698C
vermagic:       4.13.4-1-pve SMP mod_unload modversions

David Herselman · Dec 5, 2017

We are running legacy non VLAN aware Linux bridge (aka not OVS) with two active/backup bond interfaces:

bond0 comprising of 2 x 10GbE Intel X540-AT2 (VM traffic)
bond1 comprising of 2 x 10GbSFP+ Intel 82599ES (Ceph traffic)

All 4 interfaces utilise Intel ixgbe driver v5.3.3, eg:

Code:

[root@kvm5a ~]# ethtool -i eth2
driver: ixgbe
version: 5.3.3
firmware-version: 0x8000047d
expansion-rom-version:
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Pings work as expected:

Code:

[root@kvm5a ~]# ping -M do 10.254.1.3 -s 9188
PING 10.254.1.3 (10.254.1.3) 9188(9216) bytes of data.
9196 bytes from 10.254.1.3: icmp_seq=1 ttl=64 time=0.135 ms
^C
--- 10.254.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.135/0.135/0.135/0.000 ms

[root@kvm5a ~]# ping -M do 198.19.17.67 -s 9188
PING 198.19.17.67 (198.19.17.67) 9188(9216) bytes of data.
9196 bytes from 198.19.17.67: icmp_seq=1 ttl=64 time=0.203 ms
9196 bytes from 198.19.17.67: icmp_seq=2 ttl=64 time=0.175 ms
^C
--- 198.19.17.67 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.175/0.189/0.203/0.014 ms

[root@kvm5a ~]# ping -M do 10.254.1.3 -s 9190
PING 10.254.1.3 (10.254.1.3) 9190(9218) bytes of data.
ping: local error: Message too long, mtu=9216
^C
--- 10.254.1.3 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Our network interface configuration file:

Code:

/etc/network/interfaces
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
        slaves eth0,eth1
        bond_miimon 100
        bond_mode active-backup
        mtu 9216

auto bond1
iface bond1 inet static
        address 10.254.1.2
        netmask  255.255.255.0
        slaves eth2,eth3
        bond_miimon 100
        bond_mode active-backup
        mtu 9216

auto eth0
iface eth0 inet manual
        bond-master bond0
        bond-primary eth0
        mtu 9216

auto eth1
iface eth1 inet manual
        bond-master bond0
        mtu 9216

auto eth2
iface eth2 inet manual
        bond-master bond1
        mtu 9216

auto eth3
iface eth3 inet manual
        bond-master bond1
        bond-primary eth3
        mtu 9216

auto vmbr0
iface vmbr0 inet static
        address 198.19.17.66
        netmask 255.255.255.224
        gateway 198.19.17.65
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        mtu 9216

xcdr · Dec 5, 2017

ixgbe (2 x 10Gbit) works for me too (without bonding), I have only problem on igb (4 x 1Gbit and 2 x 1Gbit) NICs

ethtool info on both NICs shows:

Code:

firmware-version: 3.19, 0x00013bdb
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

fabian · Dec 5, 2017

we can reproduce the issue with igb in our testlab! thanks for reporting it - we'll either fix it in .12 or revert to .10 with our own MTU patch.

xcdr · Dec 5, 2017

OK, this is important for me on production environment.

fabian · Dec 5, 2017

pve-kernel-4.13.8-3-pve in version 4.13.8-30 (available in pvetest and pve-no-subscription) fixes the MTU issue by reverting to the previous driver version with our in-tree based MTU patch, pending further investigation.

un1x0d · Dec 5, 2017

And stop work queues on Intel I350

fabian · Dec 6, 2017

un1x0d said:
And stop work queues on Intel I350

could you be a bit more specific? which kernel version breaks which feature? the only difference between igb 5.3.5.10 and .12 is how the MTU limits are setup and checked..

un1x0d · Dec 6, 2017

fabian said:
could you be a bit more specific? which kernel version breaks which feature? the only difference between igb 5.3.5.10 and .12 is how the MTU limits are setup and checked..

https://bugzilla.proxmox.com/show_bug.cgi?id=1554

fabian · Dec 6, 2017

un1x0d said:
https://bugzilla.proxmox.com/show_bug.cgi?id=1554

thanks, I can reproduce the issue and will investigate.

robertb · Feb 26, 2018

Hello,
sorry for bringing up this old topic.
I just ran across the same Intel i350 issue: MSI-X stopped working and falls back to MSI, which also doesn't allow to configure queues larger than 1.

PVE 4.13.13-36
driver: igb
version: 5.3.5.10

fabian · Feb 26, 2018

still on the TODO list unfortunately, the whole KPTI/Spectre/Meltdown mess put this onto the back burner. let's hope there's some time now that things have slowed down a bit again!

robertb · Mar 13, 2018

Good morning,

I just confirmed that by manually compiling and inserting igb driver 5.3.5.15, the queues will start to be functional again.
Incase someone relies on this: igb driver modules can be replaced live with minimal impact (short packet loss), no reboot required.

fabian · Mar 13, 2018

robertb said:
Good morning,

I just confirmed that by manually compiling and inserting igb driver 5.3.5.15, the queues will start to be functional again.
Incase someone relies on this: igb driver modules can be replaced live with minimal impact (short packet loss), no reboot required.

the 4.15 preview kernel from pvetest (which uses the in-tree drivers) supports both MTU setting and multiple queues as well.

Search

Search

Jumbo Frames not work after upgrade pve-kernel to 4.13.8-2

xcdr

Member

elurex

Active Member

fabian

Proxmox Staff Member

fabian

Proxmox Staff Member

xcdr

Member

David Herselman

Renowned Member

xcdr

Member

fabian

Proxmox Staff Member

xcdr

Member

fabian

Proxmox Staff Member

un1x0d

Renowned Member

fabian

Proxmox Staff Member

un1x0d

Renowned Member

fabian

Proxmox Staff Member

robertb

Active Member

fabian

Proxmox Staff Member

robertb

Active Member

fabian

Proxmox Staff Member