Sudden drop in network performance

zoid

New Member
Mar 23, 2012
10
0
1
Hi, everybody, let me first describe my setup: I have a cluster with 4 nodes,all of them booting from iSCSI, the VMs storage is in iSCSI backend as well.
Every node has 4 NICs, configured as follows:
eth0: iSCSI net1
eth1: iSCSI net2
eth2 eth3 forming bond0, but only one of them is connected.

# pveversion -v
pve-manager: 2.2-31 (pve-manager/2.2/e94e95e9)
running kernel: 2.6.35-2-pve
proxmox-ve-2.6.32: 2.2-82
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.35-2-pve: 2.6.35-13
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-33
qemu-server: 2.0-69
pve-firmware: 1.0-21
libpve-common-perl: 1.0-39
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: not correctly installed
vzquota: 3.1-1
pve-qemu-kvm: 1.2-7
ksm-control-daemon: not correctly installed

Every node has 2 or 3 KVM guests, and everything was working great, but I started to experience problems with one guest in particular. The issue arrises every couple of days, the network of this vm starts to misbehave, and the only way I could find to restore the vm to it's normal state is by rebooting.

this is the result of pingig the gateway:
Before reboot:
root@vm9:~# ping 192.168.1.100
--- 192.168.1.100 ping statistics --- 21 packets transmitted, 20 received, 4% packet loss, time 22098ms
rtt min/avg/max/mdev = 0.358/576.220/2068.210/649.715 ms, pipe 2

After reboot:

root@vm9:~# ping 192.168.1.100
ping statistics --- 22 packets transmitted, 22 received, 0% packet loss, time 20997ms
rtt min/avg/max/mdev = 0.284/0.393/0.645/0.083 ms

This happens both with virtio and e1000 virtual NICs.
The guests are running Ubuntu server 12.04 64 bits, kernel 3.2.0, the only difference I can think of between this guest and the ones that are not affected by this issue is the cpu load. This one's a little higher.

Let me know if you need any more information. Any tips or suggestions will be greatly appreciated.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!