Irregularly but (now) frequent network loss in linux 2.6 guests

kth5

New Member
Nov 18, 2008
11
0
1
We are running two separated clusters using Proxmox VE, one is running on HP Proliant DL380G5 (bnx2) servers and one on Dell PowerEdge R200 (tg3). Both setups have been running - mostly without issues - for about 2 years now. Just today though one linux guest stopped responding via network. Looking at it using the VNC console didn't reveal anything other than that it wasn't able to send any packets to the outside anymore - not even to the host - but after issuing a /etc/init.d/network restart in the guest the problem magically went away. On the host nothing was visible as well, the guest's vmtab was still attached to the bridge and no dmesg hints whatsoever.

If it weren't for the fact exactly the same problem occurred again after a few minutes repeatedly and a reboot of the host also didn't do anything, I'd overlook this. Here is why:

We are sort of used to this problem (on both setups) already as it occurred a couple of times within the 2 years but right now I'm forced to run a network restart via cron every 15 minutes to keep the network up. Obviously I'd like to avoid this and would like to find out what the real cause is. I suspect it's somewhere in qemu as I cannot find any hints inside the guest nor on the host as mentioned.


We are running the following versions which are the latest taken from:
pve-manager: 1.5-10 (pve-manager/1.5/4822)
running kernel: 2.6.24-12-pve
pve-kernel-2.6.24-7-pve: 2.6.24-11
pve-kernel-2.6.24-12-pve: 2.6.24-24
pve-kernel-2.6.24-5-pve: 2.6.24-6
qemu-server: 1.1-16
pve-firmware: 1.0-5
libpve-storage-perl: 1.0-13
vncterm: 0.9-2
vzctl: 3.0.24-1pve2
vzdump: 1.2-7
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
The host's /etc/network/interfaces

# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

auto vmbr0
iface vmbr0 inet static
address 172.24.0.30
netmask 255.255.00.0
gateway 172.24.0.1
bridge_ports eth0
bridge_stp off
bridge_fd 0

auto vmbr1
iface vmbr1 inet static
address 172.22.1.101
netmask 255.255.255.0
bridge_ports eth2
bridge_stp off
bridge_fd 0
And that of the guest's which is failing:

auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 172.24.0.201
netmask 255.255.0.0
gateway 172.24.0.1
brctl show from the host:

bridge name bridge id STP enabled interfaces
vmbr0 8000.00215ad1f0b2 no eth0
veth108.0
veth110.0
veth116.0
veth125.0
vmtab103i0
vmtab104i0
vmtab117i0
vmtab123i0
vmbr1 8000.001f295d1a65 no eth2
vmtab103i1
 

kth5

New Member
Nov 18, 2008
11
0
1
I guess the guest config might also be of interest:

name: monitor
ide2: debian-500-i386-businesscard.iso,media=cdrom
sockets: 1
bootdisk: virtio0
virtio0: vm-117-disk.qcow2
ostype: l26
memory: 1536
onboot: 1
boot: cad
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
cores: 2
vlan0: e1000=DE:48:97:BF:2E:74
 
Last edited:

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,803
408
103
Austria
www.proxmox.com
where is the 'pve-qemu-kvm' package - seems to be missing?

Also - i just updated the 'pvetest' repository. Please can you test with the latest version and 2.6.32 kernel (which includes OpenVZ support now)
 

kth5

New Member
Nov 18, 2008
11
0
1
Odd indeed, pve-qemu-kvm wasn't listed but it is installed:

host:~# pveversion -v
pve-manager: 1.5-10 (pve-manager/1.5/4822)
running kernel: 2.6.24-12-pve
pve-kernel-2.6.24-7-pve: 2.6.24-11
pve-kernel-2.6.24-12-pve: 2.6.24-24
pve-kernel-2.6.24-5-pve: 2.6.24-6
qemu-server: 1.1-16
pve-firmware: 1.0-5
libpve-storage-perl: 1.0-13
vncterm: 0.9-2
vzctl: 3.0.24-1pve2
vzdump: 1.2-7
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1

host:~# which kvm
/usr/bin/kvm

host:~# dpkg -S /usr/bin/kvm
pve-qemu-kvm: /usr/bin/kvm
I'll try the new kernel in the next maintenance window this night (CEST) and report back.
 
Last edited:

kth5

New Member
Nov 18, 2008
11
0
1
Alright, also followed your advice and as we're still running .24 because we use OpenVZ for some applications went ahead to do the following after an apt-get update assuming .32 finally superseded the .24 branch of the kernel I tried the following next:

host:~# apt-get install proxmox-ve-2.6.32
<snip>
Setting up proxmox-ve-2.6.32 (1.6-11) ...
installing proxmox release key: OK
This seems to also have updated pveversion -v. I'm certain pve-qemu-kvm was up to date just because I checked that against the repository. Now I'm crossing my fingers for the trial.
 
Last edited:

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,803
408
103
Austria
www.proxmox.com
Odd indeed, pve-qemu-kvm wasn't listed but it is installed:

It is not installed if not listed! (You are using the wrong version of kvm). Please use the correct packages, else it makes no sense to report bugs.
 

kth5

New Member
Nov 18, 2008
11
0
1
Pardon me, but I'm sure you overlooked the dpkg -S output? I didn't see anything replacing my "broken" pve-qemu-kvm either by upgrading to the new meta package.
 

kth5

New Member
Nov 18, 2008
11
0
1
On closer inspection it turned out the meta package wasn't replaced either, just installed an configured. proxmox-ve-2.6.24 was not installed before apparently, looking at what pveversion does it may have not added pve-qemu-kvm to the list and thus not show it just because that is an operation happening only when mentioned meta-package is installed.

So yes, I suspect that this was a problem on my part somewhere. Except for the meta-package everything was there indeed and thus also worked together flawlessly if it wasn't for the issue we're having now. Either way, I'm making triple sure now I have the proxmox-ve-2.6.32 completely before I reboot tomorrow morning.

PS: I understand pveversion is provided by pve-manager. Wouldn't it make sense to distribute it with the meta-pakage since it has dependencies on everything needed anyway?
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,803
408
103
Austria
www.proxmox.com
PS: I understand pveversion is provided by pve-manager. Wouldn't it make sense to distribute it with the meta-pakage since it has dependencies on everything needed anyway?

no, because not everyone wants to install the meta-package.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!