LXC restart creates kworker CPU 100%

@Alwin

Does the latest kernel as of this comment in the pve no sub repo include the fixes described previously?

4.15.18-2-pve #1 SMP PVE 4.15.18-20 (Thu, 16 Aug 2018 11:06:35 +0200) x86_64 GNU/Linux
 
I unfortunilty have to report that kernel 4.15.18-5-pve is also faulty and has the same issues. I have tested on two different nodes and the same kworker issue happens.
 
  • Like
Reactions: Marius_B
Thanks for bumping this and the report - would you mind opening a bug-report at https://bugzilla.proxmox.com/ ?

Since we cannot currently reproduce the issue locally, it would be helpful if you provide us with a stacktrace of the kworker process which runs at 100%:
Code:
cat /proc/<pid>/stack
where <pid> is the kworkers pid.

Apart from that a perf report of the process and the whole system could also provide some insight:
Code:
# Gets the perf output for the whole system
perf record -F 99 -a -g -- sleep 60
perf script > out.perf

# Capturing PID X (the kworker):
perf record -F 99 -p <pid> -g -- sleep 60
perf script > out.perf

(probably best to attach the outputs in bugzilla)

EDIT: if possible please also provide the output of pvereport - it would help in maybe isolating the issue to certain setups
 
Last edited:
@Stoiko Ivanov

Thank you.

I have filed the following bug:

https://bugzilla.proxmox.com/show_bug.cgi?id=1943

I am trying to get the perf reports for you, but I am seeing this error when trying to run perf:

Code:
root@node3:~# perf
/usr/bin/perf: line 13: exec: perf_4.15: not found
E: linux-perf-4.15 is not installed.

I am not seeing a package for perf-4.15. I'll research this and attach the reports once this is resolved.

Please let us know if we can provide any further details. Thanks, again.
 
in pve the package is called `linux-tools-4.15`
 
Have the same bug here with a ProxMox kernel "PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 GNU/Linux"

cat of the pid stacktrace says " [<0>] 0xffffffffffffffff "

Storage is LVM-Thin and no iptables rules are used.

Will reset the server in the evening, is there a fix for this problem?
 
Last edited:
I've been plagued with this bug for many kernel versions, but I think I might have corrected my issue. I'm currently running 4.15.18-10-pve. I use LVM-Thin, no ZFS.

I could reproduce this issue on demand. After resetting my node to fix the issue, I could get away with one (maybe two) container restarts before a kworker was at 100% again and the node required another hard reset.

Recently, I found this other thread suggesting to update the BIOS. I looked at mine and found it to be several versions behind.

Since updating the BIOS, I have been able to restart containers without issues for a few days - something that has never been possible before with my configuration. I'm not ready to declare total victory yet, but the BIOS update suggestion has at least improved the situation.
 
I have replicated this problem on a supermicro chassis on command over the last two days. Updated BIOS and pve versions have made no difference. All I have to do is shut a lxc down and 1 core is pegged by kworker.

pveversion is 5.3-9

planar is X10SDV-8C-TLN4F

intel 10G and 1G on the mb.
 
Sharing this possible workaround posted on a related thread in case it helps others out: https://forum.proxmox.com/threads/lxc-container-reboot-fails-lxc-becomes-unusable.41264/

Proxmox's support unfortunately hasn't been any help with this. We've been trying to at least find a workaround on our Proxmox servers. Simply as a test, referencing this bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1765980, we setup ip6tables on one of our nodes to block all IPv6 traffic. Since the change, we haven't been able to reproduce the issue on our test server with automated LXC shutdown/startup of containers (this server previously exhibited the issue). I'd be curious to hear if anyone else is able to try a similar approach and see if they can reproduce the issue afterwards.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!