One Of Twelve CPU Core Usage is Over 90%

[Duktek]

Hello

Currently we have some question regarding the algorithm for distributing core usage on VM. So we have one VM that has 12 CPU cores with CPU usage of around 70%. And after checking via the htop command, there is a CPU core that always has high usage, on CPU core 12.

Can anyone help explain why only CPU core 12 usage is always above 90%? Is it rellevant with irq imbalance?

Thank you for the help

 

[LnxBil]

Can anyone help explain why only CPU core 12 usage is always above 90%? Is it rellevant with irq imbalance?

Not without more information. Is this always the same process? HAve you pinned processes to a core? What CPU is this?

 

[Duktek]

Not without more information. Is this always the same process? HAve you pinned processes to a core? What CPU is this?

Thanks for your reply,

Im sorry becaouse didn't provide information of our proxmox version and cpu model that we used before. Here is the detail proxmox version and the cpu model:

# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.39-4-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-9
pve-kernel-helper: 7.2-9
pve-kernel-5.4: 6.4-18
pve-kernel-5.3: 6.1-6
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.4.189-2-pve: 5.4.189-2
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1

# lscpu |grep "Model name"
Model name:                      Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz

I will answer your question in the following points:

1. Sorry for privacy reasons we cannot provide the details of the process on this forum, but the process ID always changed and this is consistent from 2 application *Celery* and *Gunicorn*. We don't know why the cpu core distribution is not balanced

2. We do not pinned any processes to specific CPU cores on the vm or proxmox server

3. In this VM we use type CPU 'host' for the processors

 

[LnxBil]

In this VM we use type CPU 'host' for the processors

In 1 you said it's celery and gunicon processes, so no VM?

 

[Duktek]

In 1 you said it's celery and gunicon processes, so no VM?

I mean, celery and gunicorn is application running in the vm

 

[LnxBil]

I mean, celery and gunicorn is the application running in the vm

I don't get what privacy reasons are there in the ouput of a PVE qemu process besides the name? So has this qemu process different process ids or are you talking about the process IDs inside of the VM?

 

[Duktek]

I don't get what privacy reasons are there in the ouput of a PVE qemu process besides the name? So has this qemu process different process ids or are you talking about the process IDs inside of the VM?

Sorry, yes we're talking about the process IDs inside of the vm. The screenshot im attach before is output htop command inside the vm

 

[LnxBil]

Sorry, yes we're talking about the process IDs inside of the vm. The screenshot im attach before is output htop command inside the vm

Ah, inside .... hm ... so it's not a scheduling problem of PVE itself, but the VM's. Is the same core affinity visible on the PVE host AND inside the host?

 

[Duktek]

Sorry, we're no expert on this issue, can you help us inform how to see the details about the "core affinity" you're mentioning earlier? So that we can determine that it's visible both on the PVE host and inside the host. P.S. is "inside the host" means the VM itself? Thanks in advance.

 

[LnxBil]

how to see the details about the "core affinity" you're mentioning earlier?

If you haven't set it manually, it won't be set automatically.

Check with taskset --all-tasks -p <pid>

 

[Duktek]

If you haven't set it manually, it won't be set automatically.

Check with taskset --all-tasks -p <pid>

Thanks for your reply,

Here are the command results from one PID inside the vm:

~ taskset --all-tasks -p 22964
pid 22964's current affinity mask: fff
pid 23026's current affinity mask: fff
pid 23027's current affinity mask: fff
pid 23028's current affinity mask: fff
pid 23029's current affinity mask: fff
pid 23030's current affinity mask: fff
pid 23031's current affinity mask: fff
pid 23032's current affinity mask: fff
pid 23033's current affinity mask: fff
pid 23034's current affinity mask: fff
pid 23035's current affinity mask: fff
pid 23036's current affinity mask: fff
pid 23037's current affinity mask: fff

~ taskset --all-tasks -cp 22964
pid 22964's current affinity list: 0-11
pid 23026's current affinity list: 0-11
pid 23027's current affinity list: 0-11
pid 23028's current affinity list: 0-11
pid 23029's current affinity list: 0-11
pid 23030's current affinity list: 0-11
pid 23031's current affinity list: 0-11
pid 23032's current affinity list: 0-11
pid 23033's current affinity list: 0-11
pid 23034's current affinity list: 0-11
pid 23035's current affinity list: 0-11
pid 23036's current affinity list: 0-11
pid 23037's current affinity list: 0-11

Does the result of the above command means that the process with PID "22964" will be run to all CPU cores from 0-11?

 

[LnxBil]

Does the result of the above command means that the process with PID "22964" will be run to all CPU cores from 0-11?

Yes. The fff on othe other hand means everywhere.

 

[Duktek]

Yes. The fff on othe other hand means everywhere.

Okayy, so with the result of command taskset before, process with PID "22964" has been allocated to all CPUs, and then the OS or VM itself handling the distributing it to the core. Thats its correct?

 

[leesteken]

Okayy, so with the result of command taskset before, process with PID "22964" has been allocated to all CPUs, and then the OS or VM itself handling the distributing it to the core. Thats its correct?

Yes, and you'll need to ask the makers of celery and gunicorn and the operating system inside the VM why this is happening.

 

[Duktek]

Yes, and you'll need to ask the makers of celery and gunicorn and the operating system inside the VM why this is happening.

Thank you for the answers LnxBil and leesteken, I will continue this case with my team. this case is solved for now.​