chrt (process priority) and cpulimit and system stability

Denis Kulikov

Member
Feb 28, 2018
22
0
6
38
Hello everyone!

When i was running some tests with privileged container (based on old centos distribution) - i`m find that cpulimit doesn`t work if i use chrt command.
I think that it`s correct, but cannot find proper way to deal with this (CAP_SYS_NICE give all capabilities).
Can anyone know - how correctly 'cgroups real time budget' should be set for container (lxc.cgroup.cpu.rt_runtime_us - it`s hard to understand and i'm don`t know hot it work then # CONFIG_RT_GROUP_SCHED is not set by default in pve kernels. )?


For example:
1. stress --cpu 1 --timeout 30 with cores: 1 and cpulimit: 0.5 work as expected - one core utilization up to 50% (if this core dedicated to one container).
2. chrt 10 stress --cpu 1 --timeout 30 with cores: 1 and cpulimit: 0.5 doesn`t work as expected - one core utilization up to 100% (if this core dedicated to one container).

If this core ( is used for long time for stress utility) and also used for MSI-X (network card - cat /proc/interrupts) - i`m see a backtrace in network driver (bnx2) and other (more scary things), such as:

[Fr 12:13:52 2020] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew is too large:
[Fr 12:13:52 2020] clocksource: 'hpet' wd_now: d5edd84d wd_last: a92fb674 mask: ffffffff
[Fr 12:13:52 2020] clocksource: 'tsc' cs_now: 6da143faca03 cs_last: 6c34b7fbdb24 mask: ffffffffffffffff
[Fr 12:13:52 2020] tsc: Marking TSC unstable due to clocksource watchdog
[Fr 12:13:52 2020] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
[Fr 12:13:52 2020] sched_clock: Marking unstable (50047901196233, -395795977)<-(50047546796970, -47413078)
[Fr 12:13:52 2020] clocksource: Switched to clocksource hpet
[Fr 12:13:52 2020] bnx2 0000:03:00.0 enp3s0f0: NIC Copper Link is Down

This shows that the process in the container (with CAP_SYS_NICE) can affect the entire host.

Any advice would be appreciated.

proxmox-ve: 6.2-1 (running kernel: 5.3.18-3-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-19
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.10.17-2-pve: 4.10.17-20
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-1
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
pve-zsync: 2.0-3
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!