High stealtime on host with only one vm

[MNfromH]

Hi there,

this is a first time here so hi to everyone!

I'm having a strange issue with a new machine i've set up. The setup is:

Host:
2x Xeon E5-2650v3 Hyperthreading enabled
128GB RAM DDR4 2133 ECC
RAID 1 with 480 gig Samsung SSD MZ7LM480HCHP on (Host OS, high IO storage, not used yet)
RAID-10 with 10k SAS drives (data drives for guests)
HW-Raid: MegaRAID MR9361-8i with BBU

root@host:~ # dpkg -l | grep -i proxmox
ii  libpve-access-control          4.0-16                         amd64        Proxmox VE access control library
ii  libpve-common-perl             4.0-60                         all          Proxmox VE base library
ii  libpve-storage-perl            4.0-38                         all          Proxmox VE storage management library
ii  proxmox-ve                     4.1-26                         all          The Proxmox Virtual Environment
ii  pve-cluster                    4.0-29                         amd64        Cluster Infrastructure for Proxmox Virtual Environment
ii  pve-container                  1.0-63                         all          Proxmox VE Container management tool
ii  pve-firewall                   2.0-26                         amd64        Proxmox VE Firewall
ii  pve-ha-manager                 1.0-31                         amd64        Proxmox VE HA Manager
ii  pve-headers-4.2.6-1-pve        4.2.6-36                       amd64        The Proxmox PVE Kernel Headers
ii  pve-kernel-4.2.6-1-pve         4.2.6-36                       amd64        The Proxmox PVE Kernel Image
ii  pve-manager                    4.2-4                          amd64        The Proxmox Virtual Environment

root@host:~ # uname -a
Linux host 4.2.6-1-pve #1 SMP Thu Jan 28 11:25:08 CET 2016 x86_64 GNU/Linux

root@host:~ # pveperf
CPU BOGOMIPS:  184012.60
REGEX/SECOND:  2091403
HD SIZE:  7.75 GB (/dev/dm-0)
BUFFERED READS:  517.67 MB/sec
AVERAGE SEEK TIME: 0.13 ms
FSYNCS/SECOND:  5078.63
DNS EXT:  7.88 ms
DNS INT:  5.35 ms (dns.company.com)

Currently this machine hosts one single VM (planning to add some in the future)

root@guest:~ $ uname -a
Linux guest 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

We put some serious load on the guest from time to time and we really need all those threads.

Guest CPU Util:


Host CPU Util:



Since the host does literally nothing else then serving the guest I have no idea what is causing this amount of stealtime.
Any input is appreciated.

Thanks for your time guys.

Max

 

[fireon]

Yes, it is a little bit strange, but you have also an little mistake in your vmsetup. You has 2 CPU's, so 2 Sockets, every socket have 20cores. So must say und guest VM "2 Sockets and 18 Cores". Maybe this is the only issue, please change this first and test again. And please post what process in your VM have this hight CPUload.

Thanks

 

[MNfromH]

Hi! Thanks for your answer.

I changed the config to 2 sockets, 18 cores each but the issue still persists.
The processes inside the vm are two multithreaded java applications. Any more details needed here?

I really have no idea how I can debug this any further. It's hard to find any documentation about this whole stealtime thing. Does somebody know how the host calculates this stealtime?

 

[MNfromH]

Well, MAYBE I found the answer. Maybe not as the article I found covers an entirely different architecture and hypervisor.

IBM covers a phenomenon in the following article wich basically describes what I am doing here: Using heavily paralleled applications in a hyperthreaded environemnt (IBM calls in SMT on Power-Systems) while in a guest system, see: https://www.ibm.com/developerworks/...446ebc23c550/page/Measuring+stolen+CPU+cycles

In their example they have a host system with SMT (Hyperthreading) enabled and a guest with two virtual CPUs -> one physical core with two threads.

In the IBM example, if you load this two virtual CPUs with multiple threads, keeping both threads fairly busy, the guest OS will report ~50% CPU-Load and ~50% Stealtime. Wich kind of makes sense... somewhat.

I'll do some tests on this.

 

[fireon]

Hmm, ok. Please enable NUMA, i've overseen this. This is an important option. Please read documentation: http://www.admin-magazine.com/Archive/2014/20/Best-practices-for-KVM-on-NUMA-servers Maybe this is helping you too.