KVM - High load on host when linux VM is idle (MiniPC J4105)

MrDigit

New Member
Feb 23, 2020
3
0
1
24
Hello all,

I have an Odroid H2 (Intel J4105 based) with 32 GB RAM. The CPU has VTx/VTd support. So I gave Proxmox a try.
Fortunately it is working quite well regarding the performance class of the CPU.

But I have one issue that I can't solve up to now. I have installed Centos 8.
1) Even with an idling Centos 8 VM the kvm process is consuming a lot of CPU power (> 10%).
2) This also occurs when I just boot the Centos 8 installer CD.
3) But when I boot the Centos 7 installer CD instead, the load is just at 2-3%.

Here some debugging:

(1) When the VM is in the grub bootloader selection screen KVM consumes 100%:

Code:
top - 15:24:32 up  2:53,  1 user,  load average: 1.28, 0.64, 0.62
Tasks: 195 total,   1 running, 194 sleeping,   0 stopped,   0 zombie
%Cpu(s): 22.6 us,  8.0 sy,  0.0 ni, 67.5 id,  0.0 wa,  0.0 hi,  1.9 si,  0.0 st
MiB Mem :  31939.2 total,  15413.1 free,   2138.6 used,  14387.5 buff/cache
MiB Swap:   8192.0 total,   8155.7 free,     36.2 used.  29287.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
14171 root      20   0 4666728  37532   9776 S 100.0   0.1   1:48.28 kvm

(2) When the VM is started and idling the kvm process consumes constantly > 10% CPU:

Code:
top - 15:27:56 up  2:57,  1 user,  load average: 0.73, 0.86, 0.72
Tasks: 193 total,   1 running, 192 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.5 us,  1.8 sy,  0.0 ni, 96.5 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
MiB Mem :  31939.2 total,  11984.7 free,   5545.9 used,  14408.6 buff/cache
MiB Swap:   8192.0 total,   8155.7 free,     36.2 used.  25880.6 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
14171 root      20   0 4719608   3.3g   9800 S  13.9  10.7   4:02.91 kvm

In the client VM load average is somewhere here
top - 09:30:11 up 4 min, 1 user, load average: 0,02, 0,17, 0,09

I've already tried playing with the cache mode of the disk but could not reduce the load.

Some snippets of my VM configuration:
scsi0: local-lvm:vm-100-disk-0, size=80G
net0:virtio=xx:xx:xx:xx:xx:xx,bridge=vmbr0,firewall=1

Any ideas or suggestions how to reduce the idle load ?

Best regards
MrDigit
 
Last edited:
Hello all,

I have an Odroid H2 (Intel J4105 based) with 32 GB RAM. The CPU has VTx/VTd support. So I gave Proxmox a try.
Fortunately it is working quite well regarding the performance class of the CPU.

But I have one issue that I can't solve up to now. I have installed Centos 8.
1) Even with an idling Centos 8 VM the kvm process is consuming a lot of CPU power (> 10%).
2) This also occurs when I just boot the Centos 8 installer CD.
3) But when I boot the Centos 7 installer CD instead, the load is just at 2-3%.

Here some debugging:

(1) When the VM is in the grub bootloader selection screen KVM consumes 100%:

Code:
top - 15:24:32 up  2:53,  1 user,  load average: 1.28, 0.64, 0.62
Tasks: 195 total,   1 running, 194 sleeping,   0 stopped,   0 zombie
%Cpu(s): 22.6 us,  8.0 sy,  0.0 ni, 67.5 id,  0.0 wa,  0.0 hi,  1.9 si,  0.0 st
MiB Mem :  31939.2 total,  15413.1 free,   2138.6 used,  14387.5 buff/cache
MiB Swap:   8192.0 total,   8155.7 free,     36.2 used.  29287.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
14171 root      20   0 4666728  37532   9776 S 100.0   0.1   1:48.28 kvm

(2) When the VM is started and idling the kvm process consumes constantly > 10% CPU:

Code:
top - 15:27:56 up  2:57,  1 user,  load average: 0.73, 0.86, 0.72
Tasks: 193 total,   1 running, 192 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.5 us,  1.8 sy,  0.0 ni, 96.5 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
MiB Mem :  31939.2 total,  11984.7 free,   5545.9 used,  14408.6 buff/cache
MiB Swap:   8192.0 total,   8155.7 free,     36.2 used.  25880.6 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
14171 root      20   0 4719608   3.3g   9800 S  13.9  10.7   4:02.91 kvm

In the client VM load average is somewhere here
top - 09:30:11 up 4 min, 1 user, load average: 0,02, 0,17, 0,09

I've already tried playing with the cache mode of the disk but could not reduce the load.

Some snippets of my VM configuration:
scsi0: local-lvm:vm-100-disk-0, size=80G
net0:virtio=xx:xx:xx:xx:xx:xx,bridge=vmbr0,firewall=1

Any ideas or suggestions how to reduce the idle load ?

Best regards
MrDigit


Yes, I encounter the same, it's a problem of Centos, caused by gnome GUI.
 
I have the same problem on 2 different home servers - Odroid-H2 and i5-2500k based.
I just create a minimal Debian Buster VM without any graphics and it eats about 8-10% of CPU in idle.
I've changed all options for this VM in hardware tab, but to no avail.

Here is a video of Proxmox UI and top on a server, see for yourself: https://up.agrd.eu/VsTUn5.mp4

This is the config of VM:
Code:
agent: 1
balloon: 512
bootdisk: scsi0
cores: 4
cpu: host
cpulimit: 3
ide2: none,media=cdrom
memory: 1024
name: test
net0: virtio=CE:EB:98:D5:B7:EC,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local:104/vm-104-disk-0.qcow2,discard=on,iothread=1,size=3G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=bbbcdf1f-c5f8-4dfb-8bb0-14ee440400cb
sockets: 1

And here is a list of all installed packages:
Code:
acpi-support-base
acpid
adduser
apparmor
apt
apt-listchanges
apt-utils
base-files
base-passwd
bash
bash-completion
bind9-host
bsdmainutils
bsdutils
busybox
bzip2
ca-certificates
console-setup
console-setup-linux
coreutils
cpio
cron
curl
dash
dbus
debconf
debconf-i18n
debian-archive-keyring
debian-faq
debianutils
dictionaries-common
diffutils
discover
discover-data
distro-info-data
dmidecode
dmsetup
doc-debian
dpkg
e2fsprogs
eject
emacsen-common
fdisk
file
findutils
firmware-linux-free
gcc-8-base:amd64
gdbm-l10n
geoip-database
gettext-base
gpgv
grep
groff-base
grub-common
grub-pc
grub-pc-bin
grub2-common
gzip
hdparm
hostname
iamerican
ibritish
ienglish-common
ifupdown
init
init-system-helpers
initramfs-tools
initramfs-tools-core
installation-report
iproute2
iptables
iputils-ping
isc-dhcp-client
isc-dhcp-common
iso-codes
ispell
kbd
keyboard-configuration
klibc-utils
kmod
krb5-locales
laptop-detect
less
libacl1:amd64
libapparmor1:amd64
libapt-inst2.0:amd64
libapt-pkg5.0:amd64
libargon2-1:amd64
libattr1:amd64
libaudit-common
libaudit1:amd64
libbind9-161:amd64
libblkid1:amd64
libbsd0:amd64
libbz2-1.0:amd64
libc-bin
libc-l10n
libc6:amd64
libcap-ng0:amd64
libcap2:amd64
libcap2-bin
libcom-err2:amd64
libcryptsetup12:amd64
libcurl3-gnutls:amd64
libcurl4:amd64
libdb5.3:amd64
libdbus-1-3:amd64
libdebconfclient0:amd64
libdevmapper1.02.1:amd64
libdiscover2
libdns-export1104
libdns1104:amd64
libedit2:amd64
libefiboot1:amd64
libefivar1:amd64
libelf1:amd64
libestr0:amd64
libexpat1:amd64
libext2fs2:amd64
libfastjson4:amd64
libfdisk1:amd64
libffi6:amd64
libfreetype6:amd64
libfstrm0:amd64
libfuse2:amd64
libgcc1:amd64
libgcrypt20:amd64
libgdbm-compat4:amd64
libgdbm6:amd64
libgeoip1:amd64
libglib2.0-0:amd64
libglib2.0-data
libgmp10:amd64
libgnutls30:amd64
libgpg-error0:amd64
libgpm2:amd64
libgssapi-krb5-2:amd64
libhogweed4:amd64
libicu63:amd64
libidn11:amd64
libidn2-0:amd64
libip4tc0:amd64
libip6tc0:amd64
libiptc0:amd64
libisc-export1100:amd64
libisc1100:amd64
libisccc161:amd64
libisccfg163:amd64
libjson-c3:amd64
libk5crypto3:amd64
libkeyutils1:amd64
libklibc:amd64
libkmod2:amd64
libkrb5-3:amd64
libkrb5support0:amd64
libldap-2.4-2:amd64
libldap-common
liblmdb0:amd64
liblocale-gettext-perl
liblockfile-bin
liblognorm5:amd64
liblwres161:amd64
liblz4-1:amd64
liblzma5:amd64
libmagic-mgc
libmagic1:amd64
libmnl0:amd64
libmount1:amd64
libmpdec2:amd64
libncurses6:amd64
libncursesw6:amd64
libnetfilter-conntrack3:amd64
libnettle6:amd64
libnewt0.52:amd64
libnfnetlink0:amd64
libnftnl11:amd64
libnghttp2-14:amd64
libnss-systemd:amd64
libp11-kit0:amd64
libpam-modules:amd64
libpam-modules-bin
libpam-runtime
libpam-systemd:amd64
libpam0g:amd64
libpci3:amd64
libpcre2-8-0:amd64
libpcre3:amd64
libperl5.28:amd64
libpipeline1:amd64
libpng16-16:amd64
libpopt0:amd64
libprocps7:amd64
libprotobuf-c1:amd64
libpsl5:amd64
libpython-stdlib:amd64
libpython2-stdlib:amd64
libpython2.7-minimal:amd64
libpython2.7-stdlib:amd64
libpython3-stdlib:amd64
libpython3.7-minimal:amd64
libpython3.7-stdlib:amd64
libreadline7:amd64
librtmp1:amd64
libsasl2-2:amd64
libsasl2-modules:amd64
libsasl2-modules-db:amd64
libseccomp2:amd64
libselinux1:amd64
libsemanage-common
libsemanage1:amd64
libsepol1:amd64
libslang2:amd64
libsmartcols1:amd64
libsqlite3-0:amd64
libss2:amd64
libssh2-1:amd64
libssl1.1:amd64
libstdc++6:amd64
libsystemd0:amd64
libtasn1-6:amd64
libtext-charwidth-perl
libtext-iconv-perl
libtext-wrapi18n-perl
libtinfo6:amd64
libuchardet0:amd64
libudev1:amd64
libunistring2:amd64
libusb-0.1-4:amd64
libusb-1.0-0:amd64
libuuid1:amd64
libwrap0:amd64
libx11-6:amd64
libx11-data
libxau6:amd64
libxcb1:amd64
libxdmcp6:amd64
libxext6:amd64
libxml2:amd64
libxmuu1:amd64
libxtables12:amd64
libzstd1:amd64
linux-base
linux-image-4.19.0-6-amd64
linux-image-4.19.0-8-amd64
linux-image-amd64
locales
login
logrotate
lsb-base
lsb-release
lsof
man-db
manpages
mawk
mc
mc-data
mime-support
mount
nano
ncurses-base
ncurses-bin
ncurses-term
net-tools
netbase
netcat-traditional
openssh-client
openssh-server
openssh-sftp-server
openssl
os-prober
passwd
pciutils
perl
perl-base
perl-modules-5.28
powermgmt-base
procps
publicsuffix
python
python-apt-common
python-minimal
python2
python2-minimal
python2.7
python2.7-minimal
python3
python3-apt
python3-certifi
python3-chardet
python3-debconf
python3-debian
python3-debianbts
python3-httplib2
python3-idna
python3-minimal
python3-pkg-resources
python3-pycurl
python3-pysimplesoap
python3-reportbug
python3-requests
python3-six
python3-urllib3
python3.7
python3.7-minimal
qemu-guest-agent
readline-common
reportbug
rsyslog
sed
sensible-utils
shared-mime-info
sudo
systemd
systemd-sysv
sysvinit-utils
tar
task-english
task-ssh-server
tasksel
tasksel-data
telnet
traceroute
tzdata
ucf
udev
unzip
usb.ids
usbutils
util-linux
util-linux-locales
vim-common
vim-tiny
wamerican
wget
whiptail
xauth
xdg-user-dirs
xkb-data
xxd
xz-utils
zlib1g:amd64

No GUI at all, as you see :(
 
I have a number of Debian 10 VM's and none of them exhibit that behavior. I have seen it with CentOS when GUI is installed though.

I have noticed in the past that machine=q35 uses lower CPU at idle than the default i440. Also, you use balloon and I don't, you use virtio-scsi-single while I use virtio-scsi-pci.

Maybe some things to look at.

ETA: I'm also running on a Xeon D-1541 host, so somewhat more powerful.
 
I have a number of Debian 10 VM's and none of them exhibit that behavior. I have seen it with CentOS when GUI is installed though.

I have noticed in the past that machine=q35 uses lower CPU at idle than the default i440. Also, you use balloon and I don't, you use virtio-scsi-single while I use virtio-scsi-pci.

Maybe some things to look at.

ETA: I'm also running on a Xeon D-1541 host, so somewhat more powerful.
Yes, @BobhWasatch, changing machine to q35 lowers idle CPU usage by 2-3 times! Thank you!