Network very slow after upgrading to 5.1

[decibel83]

Hi.

I upgraded my PVE 4.4-20 system (pve-manager/4.4-20/2650b7b5 (running kernel: 4.4.95-1-pve) to the last PVE 5.1 (pve-manager/5.1-41/0b958203 (running kernel: 4.13.13-2-pve).

The upgrade ended without any error but after the reboot the networking on the server is really slow!

I can connect using SSH but every type takes about 1 second to be taken from the server, and I am not able to connect to the web interface because I receive a blank page after the HTTPS handshake.

I tried to reboot using the old 4.4.95-1-pve kernel but I have the same problem.

If I ping the server from inside my LAN I have many packets dropped:

m@MacBook-Pro-4.local:~$ ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
64 bytes from 192.168.2.1: icmp_seq=4 ttl=64 time=0.225 ms
Request timeout for icmp_seq 5

On the console I see these errors, but I don't know if they are related to my problem:

[  350.678252]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294977391, last ping 4294978688, now 4294979968
[  754.640854]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295078400, last ping 4295079680, now 4295080960
[ 1153.485266]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295178126, last ping 4295179392, now 4295180672
[ 1388.746729]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295236928, last ping 4295238208, now 4295239488

The server is an IBM X3650 M3 with this network interface:

root@servir02:~# lspci|grep Ethernet
0b:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
0b:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
10:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
10:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

The bnx2 driver seems to be correctly loaded:

root@servir01:~# lsmod|grep bnx
bnx2                   81920  0

and its version seems to be the right one:

root@servir01:~# modinfo bnx2
filename:       /lib/modules/4.13.13-2-pve/kernel/drivers/net/ethernet/broadcom/bnx2.ko
firmware:       bnx2/bnx2-rv2p-09ax-6.0.17.fw
firmware:       bnx2/bnx2-rv2p-09-6.0.17.fw
firmware:       bnx2/bnx2-mips-09-6.2.1b.fw
firmware:       bnx2/bnx2-rv2p-06-6.0.15.fw
firmware:       bnx2/bnx2-mips-06-6.2.3.fw
version:        2.2.6
license:        GPL
description:    QLogic BCM5706/5708/5709/5716 Driver
author:         Michael Chan <mchan@broadcom.com>
srcversion:     DAEFDB682746C4E3AE27475
alias:          pci:v000014E4d0000163Csv*sd*bc*sc*i*
alias:          pci:v000014E4d0000163Bsv*sd*bc*sc*i*
alias:          pci:v000014E4d0000163Asv*sd*bc*sc*i*
alias:          pci:v000014E4d00001639sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016ACsv*sd*bc*sc*i*
alias:          pci:v000014E4d000016AAsv*sd*bc*sc*i*
alias:          pci:v000014E4d000016AAsv0000103Csd00003102bc*sc*i*
alias:          pci:v000014E4d0000164Csv*sd*bc*sc*i*
alias:          pci:v000014E4d0000164Asv*sd*bc*sc*i*
alias:          pci:v000014E4d0000164Asv0000103Csd00003106bc*sc*i*
alias:          pci:v000014E4d0000164Asv0000103Csd00003101bc*sc*i*
depends:        
intree:         Y
name:           bnx2
vermagic:       4.13.13-2-pve SMP mod_unload modversions 
parm:           disable_msi:Disable Message Signaled Interrupt (MSI) (int)

This is my network configuration:

root@servir01:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

auto bond0
iface bond0 inet manual
    slaves eth0 eth1
    bond_miimon 100
    bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
    address  192.168.2.1
    netmask  255.255.255.0
    gateway  192.168.2.254
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0
    dns-nameservers 192.168.2.254 8.8.8.8 8.8.4.4
    dns-search my.domain.name

I didn't change any network configuration before the upgrade and I made an in-place upgrade following the guide at https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0#In-place_upgrade.

Could you help me please?
The system is completely unusable now.

Thank you very much!
Bye

 

[udo]

Hi,
what happens if you don't use the bond and select eth0 as bridgeport for vmbr0?

Try iperf to see if you have good performance (in both directions).

Udo

 

[un1x0d]

Show "cat /proc/net/bonding/bond0"

 

[decibel83]

Show "cat /proc/net/bonding/bond0"

root@servir01:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: e4:1f:13:65:f0:c4
Slave queue ID: 0

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: e4:1f:13:36:a3:ac
Slave queue ID: 0

I checked into the switch configuration and I realized that one member of the LAG is offline even if it seems to be up on the server.
I disabled LACP and all works good now, I will check with Cisco what's going wrong.

Sorry!
And thank for your help.