VM CPUs are slow compared to PVE hosts

stefws

stefws

Renowned Member
Jan 29, 2015
302
4
83
Denmark
siimnet.dk
I'm wondering why I seem to loose so much power when comparing my VMs vCPU to the physical PVE CPU:

Got a PoC PVE 3.4 test cluster, node1-4 runs Ceph Giant with RBD pool for VM images, node5-7 runs VMs.
Node 1,2,7 are 2x dual core AMD Opteron 2218, 24GB,
Node5 is Intel Xeon based, 2x quad core E5440 @ 2.83GHz 32GB,
Node 6 is Intel Xeon based, 1x dual core 5160 @ 3GHz 12GB,

# visit pve 'date; i=0; while (( i < 1000000 )); do (( i ++ )); done; date'
calling node1...
Sat Mar 7 14:59:57 CET 2015
Sat Mar 7 15:00:06 CET 2015
calling node2...
Sat Mar 7 15:00:06 CET 2015
Sat Mar 7 15:00:15 CET 2015
calling node3...
Sat Mar 7 15:00:15 CET 2015
Sat Mar 7 15:00:28 CET 2015
calling node4...
Sat Mar 7 15:00:28 CET 2015
Sat Mar 7 15:00:36 CET 2015
calling node5...
Sat Mar 7 15:00:36 CET 2015
Sat Mar 7 15:00:43 CET 2015
calling node6...
Sat Mar 7 15:00:43 CET 2015
Sat Mar 7 15:00:50 CET 2015
calling node7...
Sat Mar 7 15:00:50 CET 2015
Sat Mar 7 15:00:59 CET 2015




With VM mostly idle, testing one VM at time I get these numbers

# visit vms 'date; i=0; while (( i < 1000000 )); do (( i ++ )); done; date'
calling vm1...
Sat Mar 7 15:01:20 CET 2015
Sat Mar 7 15:04:37 CET 2015
calling vm2...
Sat Mar 7 15:04:38 CET 2015
Sat Mar 7 15:08:53 CET 2015
calling vm3...
Sat Mar 7 15:08:56 CET 2015
Sat Mar 7 15:13:34 CET 2015
calling vm4...
Sat Mar 7 15:13:35 CET 2015
Sat Mar 7 15:16:31 CET 2015
calling vm5...
Sat Mar 7 15:16:32 CET 2015
Sat Mar 7 15:20:05 CET 2015

Why are a a vCPU so much slower than a hypervisor host physical CPU?
Are KVM normal expected to slow things down so much, I would assume in the order of < 5% slower.

TIA
 
BTW
all PVE nodes are running debian 7.8 with kernel 2.6.32-37.148
all VMs are running Centos 6.6 with kernel 2.6.32-504.8.1.el6.x86_64
 
I do not know that 'visit' command, and in general a shell loop is not a good benchmark tool. My guess is that 'visit' cause the delay (connect via network?), or you use different 'shell' versions inside the VMs?
 
visit is just a control loop visiting each server by means of ssh.

there's no network latency difference here, as all cli cmd are performed in the same bash shell after connection is established
and the simple integer loop is just a simple measurement to illustrate difference in pure CPU usage/performance.
 
Last edited:
Also the calc of CPU usage seems weird, f.ex. the Datacenter->Search show overview of all cpu usage in percent.

Here it shows the Resource Pool Sprawl as using 88.3% of assigned 32 vCPUs, but 88.3% seems merely the sum of individual VM CPU usage percent numbers and not 88.3% of 32 vCPUs:

PVE34_cpuusage.jpg

VM percent usage also seems wrong, f.ex. VM104 is showing 27.3% usage of 4 vCPUs, when in fact this VM is burning all 4 vCPUs almost 100%:

VM104_top.png

VM104 has been trying to rpm build a new 3.10 kernel for more than 2 days now, hovering around 25% CPU usage of 4 vCPUs, when in fact it has been burning them all mostly in userland of gcc. All our 4vCPU VMs hardly ever shows above 30% usage from PVE web console.

VM109 on the other hand seems more correct as it is burning around 1 of 2 vCPUs showing 53% usage:

VM109_top.png

But still mostly concerned with this PoC that VMs seems to loose a factor 25-30 power compared with bare metal processing. What could possible be wrong as this hopefully can not be normal perf. loss for KVMs...
 
;) seems I had misunderstood this option:

kvm: 0

Thought it meant weather or not that the VM should be allowing VM to emulate virtualization rather than use HW virtualization of PVE host

removing this makes a huge difference, will restart all my VMs and report back again...
 
Last edited:
So much better without kvm=0 :D

# visit vms 'date; i=0; while (( i < 1000000 )); do (( i ++ )); done; date'calling vm1...
Sat Mar 7 21:33:19 CET 2015
Sat Mar 7 21:33:29 CET 2015
calling vm2...
Sat Mar 7 21:33:29 CET 2015
Sat Mar 7 21:33:37 CET 2015
calling vm3...
Sat Mar 7 21:33:37 CET 2015
Sat Mar 7 21:33:49 CET 2015
calling vm4...
Sat Mar 7 21:33:50 CET 2015
Sat Mar 7 21:34:05 CET 2015
calling vm5...
Sat Mar 7 21:34:05 CET 2015
Sat Mar 7 21:34:14 CET 2015
 
You must log in or register to reply here.
Top Bottom