chrt (process priority) and cpulimit and system stability

[Denis Kulikov]

Hello everyone!

When i was running some tests with privileged container (based on old centos distribution) - i`m find that cpulimit doesn`t work if i use chrt command.
I think that it`s correct, but cannot find proper way to deal with this (CAP_SYS_NICE give all capabilities).
Can anyone know - how correctly 'cgroups real time budget' should be set for container (lxc.cgroup.cpu.rt_runtime_us - it`s hard to understand and i'm don`t know hot it work then # CONFIG_RT_GROUP_SCHED is not set by default in pve kernels. )?


For example:
1. stress --cpu 1 --timeout 30 with cores: 1 and cpulimit: 0.5 work as expected - one core utilization up to 50% (if this core dedicated to one container).
2. chrt 10 stress --cpu 1 --timeout 30 with cores: 1 and cpulimit: 0.5 doesn`t work as expected - one core utilization up to 100% (if this core dedicated to one container).

If this core ( is used for long time for stress utility) and also used for MSI-X (network card - cat /proc/interrupts) - i`m see a backtrace in network driver (bnx2) and other (more scary things), such as:

[Fr 12:13:52 2020] clocksource: timekeeping watchdog on CPU2: Marking clocksource 'tsc' as unstable because the skew is too large:
[Fr 12:13:52 2020] clocksource: 'hpet' wd_now: d5edd84d wd_last: a92fb674 mask: ffffffff
[Fr 12:13:52 2020] clocksource: 'tsc' cs_now: 6da143faca03 cs_last: 6c34b7fbdb24 mask: ffffffffffffffff
[Fr 12:13:52 2020] tsc: Marking TSC unstable due to clocksource watchdog
[Fr 12:13:52 2020] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
[Fr 12:13:52 2020] sched_clock: Marking unstable (50047901196233, -395795977)<-(50047546796970, -47413078)
[Fr 12:13:52 2020] clocksource: Switched to clocksource hpet
[Fr 12:13:52 2020] bnx2 0000:03:00.0 enp3s0f0: NIC Copper Link is Down

This shows that the process in the container (with CAP_SYS_NICE) can affect the entire host.

Any advice would be appreciated.

Spoiler: pveversion

proxmox-ve: 6.2-1 (running kernel: 5.3.18-3-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-19
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-23-pve: 4.15.18-51
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.10.17-2-pve: 4.10.17-20
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-1
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
pve-zsync: 2.0-3
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1