PVE 100%CPU on all kvm while vms are idle at 0-5% cpu

qinyi said:
Code: 
root@txm-test-cloud230:/tmp# execsnoop-bpfcc
Click to expand...
Gotta keep this one in mind :)
qinyi said:
Find the error process "/bin/eymblcdf".
Click to expand...
A quick internet search shows nothing about this, so might just be a randomly generated string of letters. If you think the VM was compromised, the best course of action is to re-create the VM from scratch, re-obtain the image used to create it from somewhere safe, check logs and keep an eye out for anything else that might be amiss in other VMs and on your host/cluster too.
 
Hello!
I'm experiencing an issue with the KVM thread utilization in virtual machines as well. Currently, I'm in the process of migrating from PVE7 to PVE8 and I'm seeing increased CPU usage on the KVM thread.

Let’s look at the same virtual machine running on a PVE7 hypervisor:

qm config:
Code: 
acpi: 1
agent: 1
autostart: 0
balloon: 0
bios: seabios
boot: c
bootdisk: virtio0
cicustom: 
ciuser: ubuntu
cores: 4
cpu: host
cpuunits: 1000
description: db
hotplug: network,disk,usb
ide2: common:vm-100-cloudinit,media=cdrom
ipconfig0: ip=10.10.10.10/24,gw=10.10.10.1
kvm: 1
memory: 8192
meta: creation-qemu=7.2.10,ctime=1728285531
name: db
net0: virtio=FE:A5:30:7B:B9:3B,bridge=vmbr100
numa: 1
onboot: 1
ostype: l26
protection: 0
scsihw: virtio-scsi-pci
smbios1: uuid=cf9faa60-fd2a-4f13-93e1-d8c58303e731
sockets: 1
sshkeys: ...
tablet: 1
vga: std
virtio0: common:vm-100-disk-0,format=raw,mbps_rd=75,mbps_rd_max=100,mbps_wr=75,mbps_wr_max=100,size=50G
virtio1: common:vm-100-disk-1,format=raw,mbps_rd=75,mbps_rd_max=100,mbps_wr=75,mbps_wr_max=100,size=50G
vmgenid: 34944264-6568-4f10-9d33-27cbed32b0c2
pveversion -v:
Code: 
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-1
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

ps -L -p 3326266 -o pid,tid,psr,pcpu,comm:
Code: 
    PID     TID PSR %CPU COMMAND
3326266 3326266  78 16.6 kvm
3326266 3326267  87  0.0 call_rcu
3326266 3326289  81  8.3 CPU 0/KVM
3326266 3326290  30 14.7 CPU 1/KVM
3326266 3326291  83 15.0 CPU 2/KVM
3326266 3326292  78 15.0 CPU 3/KVM
3326266 3326296  83  0.0 vnc_worker
3326266 3326383  28  0.0 worker
3326266 3326391  86  0.0 worker
3326266 3326580  88  0.0 worker
3326266 3326594  25  0.0 worker
3326266 3327587  75  0.0 worker
3326266 3327590  40  0.0 worker
3326266 3327591  80  0.0 worker
3326266 3329531  73  0.0 iou-wrk-3326266
3326266 3329537  83  0.0 iou-wrk-3326290
3326266 3329538  34  0.0 iou-wrk-3326289
After live migration to a host with identical hardware but PVE8, the CPU usage on the main KVM thread increases dramatically:

ps -L -p 2133284 -o pid,tid,psr,pcpu,comm:
Code: 
    PID     TID PSR %CPU COMMAND
2133284 2133284  84 90.8 kvm
2133284 2133285  74  0.0 call_rcu
2133284 2133287  33  0.0 kvm-nx-lpage-re
2133284 2133307   6  1.3 vhost-2133284
2133284 2133308  94 10.0 CPU 0/KVM
2133284 2133309  92 15.9 CPU 1/KVM
2133284 2133310  90 16.2 CPU 2/KVM
2133284 2133311  86 16.9 CPU 3/KVM
2133284 2133314  24  0.0 vnc_worker

pveversion -v (PVE 8):
Code: 
proxmox-ve: 8.3.0 (running kernel: 6.11.11-1-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.11.11-1-pve: 6.11.11-1
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8: 6.8.12-11
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.0.0-1.1
intel-microcode: 3.20250512.1~deb11u1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.1
libpve-cluster-perl: 8.1.1
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.2-1
proxmox-backup-file-restore: 3.4.2-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.11
pve-cluster: 8.1.1
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-4
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.13
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2

timeout 10 sudo strace -c -p $(sudo cat /var/run/qemu-server/100.pid):
Code: 
strace: Process 2133284 attached
strace: Process 2133284 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 65.89    3.075430         154     19893      1276 sendmsg
 18.98    0.885777         106      8288           io_submit
  5.26    0.245635          24     10177           ppoll
  3.31    0.154460           6     24116      3593 recvmsg
  2.68    0.124968           4     27291           write
  2.65    0.123811           7     16427           setsockopt
  1.14    0.053079           4     11122           read
  0.07    0.003038          14       204           ioctl
  0.01    0.000440          10        44           fcntl
  0.01    0.000426          20        21           accept4
  0.00    0.000206           9        21           close
  0.00    0.000147           7        21         1 futex
  0.00    0.000059           2        21           getsockname
  0.00    0.000003           3         1           mmap
  0.00    0.000003           0         4           rt_sigprocmask
  0.00    0.000002           2         1           clone3
------ ----------- ----------- --------- --------- ----------------
100.00    4.667484          39    117652      4870 total

From previous discussions, I’ve seen that the issue often lies with pve-qemu-kvm. Could you please advise which version is currently considered stable, and whether this is indeed the root cause?
 
Hi,
p-drift said:
I'm experiencing an issue with the KVM thread utilization in virtual machines as well. Currently, I'm in the process of migrating from PVE7 to PVE8 and I'm seeing increased CPU usage on the KVM thread.
Click to expand...
are there any messages on the source or target side in the system logs/journal or in the migration task log? What does lscpu say on both source and target node? Does the issue persist after a shutdown+start of the VM? Do you see increased usage inside the VM too or only outside?

Code: 
proxmox-ve: 8.3.0 (running kernel: 6.11.11-1-pve)
Note that the opt-in kernel 6.11 is not supported anymore since the opt-in 6.14 kernel came along. Please use either the standard 6.8 kernel or the opt-in 6.14 kernel: https://forum.proxmox.com/threads/o...e-8-available-on-test-no-subscription.164497/

Also Proxmox 8.4 is the current version for the Proxmox 8.x release cycle:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates
https://pve.proxmox.com/wiki/Package_Repositories
 
Hi!
fiona said:
Hi,

are there any messages on the source or target side in the system logs/journal or in the migration task log? What does lscpu say on both source and target node? Does the issue persist after a shutdown+start of the VM? Do you see increased usage inside the VM too or only outside?

Code: 
proxmox-ve: 8.3.0 (running kernel: 6.11.11-1-pve)
Note that the opt-in kernel 6.11 is not supported anymore since the opt-in 6.14 kernel came along. Please use either the standard 6.8 kernel or the opt-in 6.14 kernel: https://forum.proxmox.com/threads/o...e-8-available-on-test-no-subscription.164497/

Also Proxmox 8.4 is the current version for the Proxmox 8.x release cycle:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates
https://pve.proxmox.com/wiki/Package_Repositories
Click to expand...
> are there any messages on the source or target side in the system logs/journal or in the migration task log? What does lscpu say on both source and target node?

No, dmesg and journald is clear.

> What does lscpu say on both source and target node?

PVE8:
Code: 
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   96
  On-line CPU(s) list:    0-95
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz
    CPU family:           6
    Model:                85
    Thread(s) per core:   2
    Core(s) per socket:   24
    Socket(s):            2
    Stepping:             7
    BogoMIPS:             4800.00
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs
                           bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt
                          tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid
                           ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1
                           xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts vnmi pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization features: 
  Virtualization:         VT-x
Caches (sum of all):     
  L1d:                    1.5 MiB (48 instances)
  L1i:                    1.5 MiB (48 instances)
  L2:                     48 MiB (48 instances)
  L3:                     71.5 MiB (2 instances)
NUMA:                     
  NUMA node(s):           4
  NUMA node0 CPU(s):      0-11,48-59
  NUMA node1 CPU(s):      12-23,60-71
  NUMA node2 CPU(s):      24-35,72-83
  NUMA node3 CPU(s):      36-47,84-95
Vulnerabilities:         
  Gather data sampling:   Vulnerable
  Itlb multihit:          KVM: Vulnerable
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Vulnerable
  Reg file data sampling: Not affected
  Retbleed:               Vulnerable
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
  Spectre v2:             Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable; BHI: Vulnerable
  Srbds:                  Not affected
  Tsx async abort:        Mitigation; TSX disabled

PVE7:
Code: 
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          96
On-line CPU(s) list:             0-95
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       2
NUMA node(s):                    4
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz
Stepping:                        7
CPU MHz:                         2400.000
BogoMIPS:                        4800.00
Virtualization:                  VT-x
L1d cache:                       1.5 MiB
L1i cache:                       1.5 MiB
L2 cache:                        48 MiB
L3 cache:                        71.5 MiB
NUMA node0 CPU(s):               0-11,48-59
NUMA node1 CPU(s):               12-23,60-71
NUMA node2 CPU(s):               24-35,72-83
NUMA node3 CPU(s):               36-47,84-95
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Vulnerable
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Vulnerable
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfm
                                 on pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic
                                  movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhan
                                 ced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt av
                                 x512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_
                                 capabilities

> Does the issue persist after a shutdown+start of the VM?
Yes

> Do you see increased usage inside the VM too or only outside?
Only outside. CPU utillization inside is not more then 50%.

Снимок экрана 2025-07-29 в 13.28.52.png

I'll try to downgrade kernel version to 6.8, I hope it will help.
Thank's!
 
fiona said:
Hi,

are there any messages on the source or target side in the system logs/journal or in the migration task log? What does lscpu say on both source and target node? Does the issue persist after a shutdown+start of the VM? Do you see increased usage inside the VM too or only outside?

Code: 
proxmox-ve: 8.3.0 (running kernel: 6.11.11-1-pve)
Note that the opt-in kernel 6.11 is not supported anymore since the opt-in 6.14 kernel came along. Please use either the standard 6.8 kernel or the opt-in 6.14 kernel: https://forum.proxmox.com/threads/o...e-8-available-on-test-no-subscription.164497/

Also Proxmox 8.4 is the current version for the Proxmox 8.x release cycle:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates
https://pve.proxmox.com/wiki/Package_Repositories
Click to expand...

So, I've downgraded to 6.8.12-13-pve, but nothing changed, CPU utilization of KVM thread is the same.

pveversion:
Code: 
proxmox-ve: 8.3.0 (running kernel: 6.8.12-13-pve)
pve-manager: 8.4.0 (running version: 8.4.0/ec58e45e1bcdf2ac)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-9-pve: 6.8.12-9
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.0.0-1.1
intel-microcode: 3.20250211.1~deb11u1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.3.7-1
proxmox-backup-file-restore: 3.3.7-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 0.7.3
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2
 
@p-drift does enabling the IO thread setting on the VM disks help? Does changing the CPU type from host to something else like x86-64-v2-AES make a difference?

p-drift said:
> Do you see increased usage inside the VM too or only outside?
Only outside. CPU utillization inside is not more then 50%.
Click to expand...
Did you look at the idle % from top or how did you check it?

p-drift said:
timeout 10 sudo strace -c -p $(sudo cat /var/run/qemu-server/100.pid):
Code: 
strace: Process 2133284 attached
strace: Process 2133284 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 65.89    3.075430         154     19893      1276 sendmsg
 18.98    0.885777         106      8288           io_submit
  5.26    0.245635          24     10177           ppoll
  3.31    0.154460           6     24116      3593 recvmsg
  2.68    0.124968           4     27291           write
  2.65    0.123811           7     16427           setsockopt
  1.14    0.053079           4     11122           read
  0.07    0.003038          14       204           ioctl
  0.01    0.000440          10        44           fcntl
  0.01    0.000426          20        21           accept4
  0.00    0.000206           9        21           close
  0.00    0.000147           7        21         1 futex
  0.00    0.000059           2        21           getsockname
  0.00    0.000003           3         1           mmap
  0.00    0.000003           0         4           rt_sigprocmask
  0.00    0.000002           2         1           clone3
------ ----------- ----------- --------- --------- ----------------
100.00    4.667484          39    117652      4870 total
Click to expand...
What stands out is the long times for sendmsg and the high error counts for sendmsg and recvmsg syscalls. Could be the VNC display communication, could be the QMP communication, but maybe also something else.

How big is the general CPU load/pressore on the Proxmox VE node itself? What OS/kernel/workload is running in the VM?
 
So, I calculated the utilization and found that the CPU usage of the main KVM thread is actually the sum of the utilization of all other VM threads on a PVE8 hypervisor.
Code: 
$ sudo ps -L -p 1750794 -o pid,tid,psr,pcpu,comm
    PID     TID PSR %CPU COMMAND
1750794 1750794  71 90.5 kvm
1750794 1750795   0  0.0 call_rcu
1750794 1750797   7  0.0 kvm-nx-lpage-re
1750794 1750837  74  1.8 vhost-1750794
1750794 1750838  28 15.1 CPU 0/KVM
1750794 1750839  16 23.7 CPU 1/KVM
1750794 1750840  80 24.1 CPU 2/KVM
1750794 1750841  25 25.0 CPU 3/KVM
1750794 1750844  57  0.0 vnc_worker
1750794 1751386   5  0.0 worker
1750794 1751421  19  0.0 worker
1750794 1752047  49  0.0 worker
1750794 1945974  70  0.0 worker
1750794 2357810  33  0.0 worker
1750794 3394448  79  0.0 worker

90.5% for the main KVM thread and 89.7% in total for the other threads. This behavior is consistent across all other VMs on PVE8, whether they're under load or idle.

However, this is not the case on PVE7:
Code: 
$ sudo ps -L -p 3326266 -o pid,tid,psr,pcpu,comm
    PID     TID PSR %CPU COMMAND
3326266 3326266  78 16.6 kvm
3326266 3326267  87  0.0 call_rcu
3326266 3326289  81  8.3 CPU 0/KVM
3326266 3326290  30 14.7 CPU 1/KVM
3326266 3326291  83 15.0 CPU 2/KVM
3326266 3326292  78 15.0 CPU 3/KVM
3326266 3326296  83  0.0 vnc_worker
3326266 3326383  28  0.0 worker
3326266 3326391  86  0.0 worker
3326266 3326580  88  0.0 worker
3326266 3326594  25  0.0 worker
3326266 3327587  75  0.0 worker
3326266 3327590  40  0.0 worker
3326266 3327591  80  0.0 worker
3326266 3329531  73  0.0 iou-wrk-3326266
3326266 3329537  83  0.0 iou-wrk-3326290
3326266 3329538  34  0.0 iou-wrk-3326289

When IO thread is enabled, the utilization from the dedicated IO thread is also counted towards the main KVM thread.

Could this behavior be correct? Or is it a measurement/reporting issue?
Am I right in assuming that if the main KVM thread utilization goes over 100%, it could lead to delays and freezes inside the virtual machine?
 
p-drift said:
So, I calculated the utilization and found that the CPU usage of the main KVM thread is actually the sum of the utilization of all other VM threads on a PVE8 hypervisor.
Code: 
$ sudo ps -L -p 1750794 -o pid,tid,psr,pcpu,comm
    PID     TID PSR %CPU COMMAND
1750794 1750794  71 90.5 kvm
1750794 1750795   0  0.0 call_rcu
1750794 1750797   7  0.0 kvm-nx-lpage-re
1750794 1750837  74  1.8 vhost-1750794
1750794 1750838  28 15.1 CPU 0/KVM
1750794 1750839  16 23.7 CPU 1/KVM
1750794 1750840  80 24.1 CPU 2/KVM
1750794 1750841  25 25.0 CPU 3/KVM
1750794 1750844  57  0.0 vnc_worker
1750794 1751386   5  0.0 worker
1750794 1751421  19  0.0 worker
1750794 1752047  49  0.0 worker
1750794 1945974  70  0.0 worker
1750794 2357810  33  0.0 worker
1750794 3394448  79  0.0 worker

90.5% for the main KVM thread and 89.7% in total for the other threads. This behavior is consistent across all other VMs on PVE8, whether they're under load or idle.

However, this is not the case on PVE7:
Code: 
$ sudo ps -L -p 3326266 -o pid,tid,psr,pcpu,comm
    PID     TID PSR %CPU COMMAND
3326266 3326266  78 16.6 kvm
3326266 3326267  87  0.0 call_rcu
3326266 3326289  81  8.3 CPU 0/KVM
3326266 3326290  30 14.7 CPU 1/KVM
3326266 3326291  83 15.0 CPU 2/KVM
3326266 3326292  78 15.0 CPU 3/KVM
3326266 3326296  83  0.0 vnc_worker
3326266 3326383  28  0.0 worker
3326266 3326391  86  0.0 worker
3326266 3326580  88  0.0 worker
3326266 3326594  25  0.0 worker
3326266 3327587  75  0.0 worker
3326266 3327590  40  0.0 worker
3326266 3327591  80  0.0 worker
3326266 3329531  73  0.0 iou-wrk-3326266
3326266 3329537  83  0.0 iou-wrk-3326290
3326266 3329538  34  0.0 iou-wrk-3326289

When IO thread is enabled, the utilization from the dedicated IO thread is also counted towards the main KVM thread.

Could this behavior be correct? Or is it a measurement/reporting issue?
Am I right in assuming that if the main KVM thread utilization goes over 100%, it could lead to delays and freezes inside the virtual machine?
Click to expand...
Hmm, maybe there was a change in how the value for the main PID is reported.

In any case, the value given there is not the current usage, but according to man ps:
CPU usage is currently expressed as the percentage of time spent running during the entire lifetime of a process. This is not ideal, and it does not conform to the standards that ps otherwise conforms to. CPU usage is unlikely to add up to exactly 100%.
Click to expand...
 
You must log in or register to reply here.
Top Bottom