CPU Issue

[CobraHost]

Recently after switching from our VMware box over onto a proxmox setup.

we are facing major CPU problem, we have attempted to put every VM onto 8 cores instead of limited per one but still the same issue occurs.

CPU BOGOMIPS: 67195.80
REGEX/SECOND: 98528
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 101.90 MB/sec
AVERAGE SEEK TIME: 11.81 ms
FSYNCS/SECOND: 1039.47
DNS EXT: 57.19 ms
DNS INT: 33.35 ms

pve-manager: 2.0-57 (pve-manager/2.0/ff6cd700)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-65
pve-kernel-2.6.32-11-pve: 2.6.32-65
lvm2: 2.02.88-2pve2
clvm: 2.02.88-2pve2
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-37
pve-firmware: 1.0-15
libpve-common-perl: 1.0-25
libpve-access-control: 1.0-17
libpve-storage-perl: 2.0-17
vncterm: 1.0-2
vzctl: 3.0.30-2pve2
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1


top - 08:53:04 up 18:22, 1 user, load average: 62.23, 62.11, 48.29
Tasks: 452 total, 15 running, 437 sleeping, 0 stopped, 0 zombie
Cpu(s): 61.6%us, 33.1%sy, 0.0%ni, 4.5%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 65956172k total, 51602736k used, 14353436k free, 29812k buffers
Swap: 65011704k total, 710852k used, 64300852k free, 70996k cached


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
709591 root 20 0 1209m 1.0g 2008 S 148 1.6 13:57.37 kvm
13554 root 20 0 2347m 2.0g 1768 R 122 3.2 638:23.50 kvm
14515 root 20 0 1836m 1.5g 1548 R 97 2.4 571:38.29 kvm
19327 root 20 0 1835m 1.5g 1520 R 90 2.4 373:09.83 kvm
18914 root 20 0 2355m 2.0g 1556 R 76 3.2 549:15.30 kvm
17570 root 20 0 4333m 3.9g 1524 R 74 6.2 159:02.82 kvm
6765 root 20 0 3382m 3.0g 1792 R 73 4.7 272:22.71 kvm
16603 root 20 0 3381m 3.0g 1516 S 69 4.7 714:47.16 kvm
18235 root 20 0 4396m 3.9g 1520 R 69 6.2 672:21.08 kvm
25830 root 20 0 1833m 1.5g 3056 S 68 2.4 506:25.01 kvm
20136 root 20 0 1332m 1.0g 1512 S 63 1.6 616:31.15 kvm
17011 root 20 0 1333m 1.0g 1520 S 55 1.6 664:13.57 kvm
17275 root 20 0 1333m 1.0g 1552 S 52 1.6 709:59.40 kvm
20008 root 20 0 1845m 1.5g 1512 R 49 2.4 147:39.52 kvm
45435 root 20 0 1192m 1.0g 2020 R 46 1.6 84:24.29 kvm
13022 root 20 0 1313m 1.0g 1520 S 43 1.6 457:01.86 kvm
20637 root 20 0 1835m 1.5g 1512 S 38 2.4 581:33.43 kvm
182 root 25 5 0 0 0 S 35 0.0 110:02.35 ksmd

 

[udo]

Hi,
an load of over 60 is very ugly. The cpu-time of the processes (kvm and ksmd) in relation to the uptime (only 18 hours) looks like an very overcommited host. Esp. that pveperf looks like you have much cores with an not so high cpu-freq.

Is the cpu-time "real" cpu from the VM, or due virtualization (IO)? Also you have swap used, looks like that first you have use your full memory and after that ksm free a lot of mem.

Is the storage where the VMs lives, the same as the root (which you measured with pveperf)?

What kind of VMs do you use? (can you post an standard config-file?)

I would use less VMs on the hosts, try also to reduce bottlenecks (use virtio-driver, try to run without ksm).

Udo

 

[CobraHost]

Hi,
an load of over 60 is very ugly. The cpu-time of the processes (kvm and ksmd) in relation to the uptime (only 18 hours) looks like an very overcommited host. Esp. that pveperf looks like you have much cores with an not so high cpu-freq.

Is the cpu-time "real" cpu from the VM, or due virtualization (IO)? Also you have swap used, looks like that first you have use your full memory and after that ksm free a lot of mem.

Is the storage where the VMs lives, the same as the root (which you measured with pveperf)?

What kind of VMs do you use? (can you post an standard config-file?)

I would use less VMs on the hosts, try also to reduce bottlenecks (use virtio-driver, try to run without ksm).

Udo

Everything is running on full virtio drivers including networking.

Normally we would use VMware, we only chose proxmox because we could not get a ISO loaded remotely.
The CPU is the following AMD Opteron 6272 x16 2.0Ghz

Got more memory & drives on order for this certain server to handle the extra capacity. thinking of moving it back to VMware.

http://screensnapr.com/v/p3Fqa4.jpg following image is a VMware server with 2x Dual E5620's not having any issues like this.

 

[udo]

Everything is running on full virtio drivers including networking.

Normally we would use VMware, we only chose proxmox because we could not get a ISO loaded remotely.

some time ago i had done an test with vmware and pve (1.x) on the same hardware. vmware was not the winner ;-) so i guess you have also trouble on vmware with this load.

The CPU is the following AMD Opteron 6272 x16 2.0Ghz

Got more memory & drives on order for this certain server to handle the extra capacity. thinking of moving it back to VMware.

http://screensnapr.com/v/p3Fqa4.jpg following image is a VMware server with 2x Dual E5620's not having any issues like this.

Hmm,
have two hosts with 4 * opteron 6136 - never get critical loads. Much depends on fast IO, but your top-output don't show wait... let see what happens with more memory.

Udo

 

[CobraHost]

some time ago i had done an test with vmware and pve (1.x) on the same hardware. vmware was not the winner ;-) so i guess you have also trouble on vmware with this load.

Hmm,
have two hosts with 4 * opteron 6136 - never get critical loads. Much depends on fast IO, but your top-output don't show wait... let see what happens with more memory.

Udo

IO speeds are superb

How ever it's mainly due to not being able to limit anyone on the CPU usage part, had the same issue any other time we tested promox.
Going to reload the node and load VMware onto it.