Strange multi-thread performance With LXC on some node

[bash99]

I've 7 node Proxmox Cluster(4.2, just wait a good time to upgrade to 4.3), and on 2 of those nodes, LXC show strange performance when run with multi-thread programs.

a lxc config with 4 cpu quota

root@pve2:~# more /etc/pve/lxc/165.conf
arch: amd64
cpulimit: 4
cpuunits: 1024
hostname: tmpcpu
memory: 8192
net0: name=eth0,bridge=vmbr0,hwaddr=FA:1F:6B:31:D3:6B,type=veth
ostype: centos
rootfs: lizardfs:165/vm-165-disk-1.raw,size=8G
swap: 512

run sysbench in it with 2 4 8 threads I got:

root@pve2:~# pct enter 165
[root@tmpcpu ~]# for i in 2 4 8; do sysbench --test=cpu --num-threads=$i --cpu-max-prime=10000 run | grep "total time" ; done
  total time:  6.1024s
  total time taken by event execution: 12.1959
  total time:  2.7084s
  total time taken by event execution: 10.8183
  total time:  6.3878s
  total time taken by event execution: 51.0799

The cpu time doubles when use 8 threads!

Also I got very bad performance for Java application.

KVM instance on same node is OK.

but if I move vm-165 to other node, problem disappears.

Kernel version is 4.4.13 or 4.4.21, but another node with 4.4.13 is fine.

Hardware configure is same (E5-2670 *2), include HT settings.

Any clues to check?

 

[fabian]

cpulimit: 4

if you limit yourself to 4 "cores" and run more threads, your performance will decrease because of contention - not sure why you get different results on the two nodes (if they are really identical..). the cpulimits option sets a relative fraction of a time slot as a limit - so if you set it to 1, you get at most 1 core worth of CPU time per time slot, if you set it to 2, you get at most 2 cores worth of time, and so on. if you set it to 0, you get unlimited CPU time. all of this is of course again further limited by actual load on the system!

 

[bash99]

if you limit yourself to 4 "cores" and run more threads, your performance will decrease because of contention - not sure why you get different results on the two nodes (if they are really identical..). the cpulimits option sets a relative fraction of a time slot as a limit - so if you set it to 1, you get at most 1 core worth of CPU time per time slot, if you set it to 2, you get at most 2 cores worth of time, and so on. if you set it to 0, you get unlimited CPU time. all of this is of course again further limited by actual load on the system!

I know it will slow because of thread contention, but not so dramatic.
In a normal node:

root@pve3:~# pct enter 165
[root@tmpcpu ~]# for i in 2 4 8; do sysbench --test=cpu --num-threads=$i --cpu-max-prime=10000 run | grep "total time" ; done
  total time:  5.1636s
  total time taken by event execution: 10.3231
  total time:  2.6397s
  total time taken by event execution: 10.5295
  total time:  2.6571s
  total time taken by event execution: 21.2444

 

[bash99]

It's turn out a BIOS setting problem and is sloved.

The BIOS profile is setting to Performance-per-watt(DAPC), after change it to Performance-per-watt(OS), everything is ok.

I guess the reason is DPAC let most cpu running in low freq, so the cfq scheduler prefer to schedule sysbench shift to other cpu rather than limited cpus.