Java process (Jenkins) in VM stops host

[shake]

Hi guys,

I have a huge problem here. I have Proxmox with 4 VMs (KVM). Everything worked fine until Saturday when my server suddenly froze.
Now I can reproduce this behavior:

- 3 VMs work as expected
- 1 VM is problematic. Whenever I start this VM the load starts to climb (on the host, not so much within the VM) until it reaches about 150. The server and all VMs became unresponsive at this point. Ping does still work. Open SSH connections still work on the host. But I can no longer start any commands.

I started to identify the problematic process on the VM: Whenever I start Jenkins (http://jenkins-ci.org/), the load start to climb.
I don't know what exactly might trigger the problem. Any ideas how I could solve that issue?
I reduced CPU Sockets / Cores and CPU Units. Didn't change anything.

Jenkins does a lot of IO operations on startup. It also starts an embedded web server. So my best guess is, that it might be related to IO operations...

Host:
Linux gina 2.6.32-6-pve #1 SMP Fri Nov 4 06:54:05 CET 2011 x86_64 GNU/Linux
pveversion -v
pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-6-pve
proxmox-ve-2.6.32: 1.9-50
pve-kernel-2.6.32-6-pve: 2.6.32-50
qemu-server: 1.1-32
pve-firmware: 1.0-14
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2.6-1
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 0.15.0-1
ksm-control-daemon: 1.0-6



Problematic VM:
Linux dev 3.0.0-13-server #22-Ubuntu SMP Wed Nov 2 15:09:08 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

 

[tom]

run 'pveperf', post results. make sure your host is idle.

also post the VMID.conf file of your VM (cat /etc/qemu-server/VMID.conf).

 

[shake]

config file:

name: dev
bootdisk: virtio0
ostype: l26
memory: 8192
onboot: 0
sockets: 4
vlan0: virtio=00:50:56:00:0C:46
virtio0: data:vm-101-disk-1
cores: 2
boot: c
freeze: 0
cpuunits: 10
acpi: 1
kvm: 1


Currently all 4 guests have a load of 0.01 or lower. The host itself has a load of 0.81...



run pveperf 3 times:

pveperf
CPU BOGOMIPS: 54583.46
REGEX/SECOND: 1584075
HD SIZE: 99.21 GB (/dev/mapper/vg0-root)
BUFFERED READS: 144.56 MB/sec
AVERAGE SEEK TIME: 8.94 ms
FSYNCS/SECOND: 53.94
DNS EXT: 60.33 ms
----------------------
CPU BOGOMIPS: 54583.46
REGEX/SECOND: 1573804
HD SIZE: 99.21 GB (/dev/mapper/vg0-root)
BUFFERED READS: 141.78 MB/sec
AVERAGE SEEK TIME: 8.24 ms
FSYNCS/SECOND: 45.19
DNS EXT: 65.37 ms
----------------------
CPU BOGOMIPS: 54583.46
REGEX/SECOND: 1618028
HD SIZE: 99.21 GB (/dev/mapper/vg0-root)
BUFFERED READS: 147.41 MB/sec
AVERAGE SEEK TIME: 8.76 ms
FSYNCS/SECOND: 54.23
DNS EXT: 66.85 ms

 

[tom]

you fsyncs/second are much too low, indicates that you do not have a hardware raid controller with write cache (enabled).

pls give details about your hardware, storage controller and disks, also settings.

 

[shake]

Yes. I use software RAID here... Is that the problem?

 

[tom]

yes, your fsyncs/sec are too slow, there are many threads in this forum dealing with this issue.

 

[shake]

Thanks for your fast reponse... I will give it a try.