Node has very high load but low CPU usage

[Terminatrix]

My nodes CPU is

CPU(s) 96 x AMD EPYC 7642 48-Core Processor (1 Socket)

The load average has been at 150+ now for several hours but the CPU usage doesn't go above 70%.

ps faxl | grep " D "

1     0  3789     2  20   0      0     0 taskq_ D    ?        1164:49  \_ [txg_sync]
0     0 47979  3975  20   0   6140   896 pipe_w S+   pts/0      0:00  |       \_ grep  D
1     0 38487     1  20   0   6708  2168 run_st D    ?         15:57 /bin/bash /usr/sbin/ksmtuned

Is returned could this be the issue?

After reading it could be a network issue too with too many active connections I installed https://github.com/netdata/netdata to monitor the node but I can't see anything obvious. There is 2,000 active ipv4.sockstat_sockets could that be a limiting factor? I have tried adding some network optimisations

net.core.netdev_max_backlog=8192
net.core.optmem_max=8192
net.core.rmem_max=16777216
net.core.somaxconn=8151
net.core.wmem_max=16777216
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.ip_local_port_range=1024 65535
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 999999999
net.ipv4.tcp_fin_timeout=10
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_keepalive_probes=3
net.ipv4.tcp_keepalive_time=240
net.ipv4.tcp_limit_output_bytes=65536
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_rmem=8192 87380 16777216
net.ipv4.tcp_sack=1
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_syn_retries=3
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_wmem=8192 65536 16777216
net.netfilter.nf_conntrack_generic_timeout = 60
net.netfilter.nf_conntrack_helper=0
net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_tcp_timeout_established = 28800
net.unix.max_dgram_qlen = 4096

What else should I check?

 

[apoc]

Slow io also can create interesting side effects (e.g. CPU waits on io completion). Not sure though if this also increases the load of the system.
Still worth checking.

 

[Terminatrix]

Slow io also can create interesting side effects (e.g. CPU waits on io completion). Not sure though if this also increases the load of the system.
Still worth checking.

How would I check that?

 

[Falk R.]

What is your vCPU / pCPU core ratio?

A too high overprovisioning also leads to such effects, especially if single VMs have an extremely high number of cores.

 

[spirit]

you can have more infos with:

cat /proc/pressure/cpu
cat /proc/pressure/io
cat /proc/pressure/memory

 

[apoc]

See the PVE GUI host info. There is a "io delay" metric.
Aside vcpu to pcpu ratio (which is a good point) also check for sleep states of your processor. Most of the time when someone wants to safe energy that is counter productive in virtualization environments

 

[Terminatrix]

you can have more infos with:

cat /proc/pressure/cpu
cat /proc/pressure/io
cat /proc/pressure/memory

Sorry for the delay the usage went down, now the problem is back.

cat /proc/pressure/cpu
some avg10=13.60 avg60=21.69 avg300=18.65 total=559600398658
cat /proc/pressure/io
some avg10=0.00 avg60=0.00 avg300=0.00 total=23012146775
full avg10=0.00 avg60=0.00 avg300=0.00 total=10762526872
cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=3754116746
full avg10=0.00 avg60=0.00 avg300=0.00 total=2187805051

Seems to be CPU pressure but the CPU graph says 60-80% usage

 

[apoc]

Seems to be CPU pressure but the CPU graph says 60-80% usage

Well. 60%-80% CPU load on your 96 Threads and 48 "real" cores means you are pretty much maxing the system out. I think there is your answer.
If you disable HT cores I guess you max your system out at 100%. So to me this looks like your limit / source of "evil".

 

[Terminatrix]

Well. 60%-80% CPU load on your 96 Threads and 48 "real" cores means you are pretty much maxing the system out. I think there is your answer.
If you disable HT cores I guess you max your system out at 100%. So to me this looks like your limit / source of "evil".

This has helped. AMD EPYC 7642 is not a good CPU to have HT enabled on? I need to do some reading up on this but so far all I came across is that AMD cores are noted at "real cores" whatever that means.

 

[Dunuin]

HT can't do magic. You got 48 cores so only 48 threads can use the full performance of the CPU. With HT you get additional 48 threads that the CPU cores can work on if they need to wait for something but that doens't make your CPU to operate faster. It just increases the efficiency how good these 48 cores can be utilized so you only get around 10-30% extra performance.

 

[Falk R.]

How many VMs do you have on the system? How many vCPUs do your VMs have on average?

 

[Terminatrix]

HT can't do magic. You got 48 cores so only 48 threads can use the full performance of the CPU. With HT you get additional 48 threads that the CPU cores can work on if they need to wait for something but that doens't make your CPU to operate faster. It just increases the efficiency how good these 48 cores can be utilized so you only get around 10-30% extra performance.

I see. I disabled hyperthreading by doing the following

echo off >/sys/devices/system/cpu/smt/control
# check if succeeded
cat /sys/devices/system/cpu/smt/control

The virtual machines running now have more CPU steal but the main problem is sort of resolved. The memory and hard disks performance was terrible, do you think the HT caused this? sysbench tests showed results 1/10th of what they should have been with 10x+ latency. The results are still fairly poor but improved significantly.

How many VMs do you have on the system? How many vCPUs do your VMs have on average?

Quite a few https://gyazo.com/6812c34b1b6f89f5d20d4795f8960c65 close to 130 running virtual machines 0 LXC

 

[Falk R.]

Quite a few https://gyazo.com/6812c34b1b6f89f5d20d4795f8960c65 close to 130 running virtual machines 0 LXC

There's your main problem:
Let's ignore hyperthreading.
You have 48 cores and a VM with 24vCPUs, now let's assume the worst case and X VMs occupy 25 cores of your CPU. Then your 24Core VM comes and ends up in the queue, because only 23 cores are free. (We virtualize the CPU and do no emulation). The following VMs end up behind the fat VM in the queue.
The more cores the individual VMs have, the faster the effect increases and the latency of the VMs goes up extremely.
If the 24Core VM does not utilize the cores to 90+%, you should start with this one and reduce the vCPUs.
This should directly bring a noticeable effect. After that, other VMs with 4-8 cores should also be checked to see if the quantities are actually needed.

This is identical for all hypervisors.

Unfortunately, I can't say anything about hyperthreading with KVM. With HyperV, you have the option of tuning and switching the CPU scheduler. For multicore VMs the following mode is the best: Dual vCPU VMs are directed to two threads of a core, this is best from a security point of view and also for the possible density of VMs.
How KVM handles this and if it is possible to customize it, someone from the staff might be able to tell.

 

[apoc]

Bad ratio IMHO.
Sum all your vcpus up
Divide through real cores (48)
What is the ratio?
Greater 4:1 -> likely highly overloaded. But it really depends on your workload... 4:1 is my rule of thumb for mixed infrastructure.

Problem with ht is that the hardware exposes stuff that isn't there.
If you have no load in the VMs, it helps scheduling idle cores. If there is load, you have just created more contention...