LXC restart creates kworker CPU 100%

[fabian]

For information: https://pve.proxmox.com/pipermail/pve-devel/2018-August/033288.html fix will be available soon in testingrepo.

packages with that fix included are available on pvetest.

 

[Jean-Pierre]

For information: https://pve.proxmox.com/pipermail/pve-devel/2018-August/033288.html fix will be available soon in testingrepo.

I see that fix is relating to ZFS specifically - not making use of ZFS here and seeing same results, on 2 nodes so far. Both 5.2 with kernel 4.15.18-1. Have rolled back to 4.15.10-1 as this was the last working version before I upgraded. Not sure how stable anything between is.

 

[Alwin]

We uploaded a Kernel for testing. This Kernel includes the next upstream release from Ubuntu with our patches applied.
http://download.proxmox.com/temp/pve-kernel-4.15.18-2-pve_4.15.18-20~test1_amd64.deb

Kernel header if needed.
http://download.proxmox.com/temp/pve-headers-4.15.18-2-pve_4.15.18-20~test1_amd64.deb

 

[qubozik]

@Alwin

Does the latest kernel as of this comment in the pve no sub repo include the fixes described previously?

4.15.18-2-pve #1 SMP PVE 4.15.18-20 (Thu, 16 Aug 2018 11:06:35 +0200) x86_64 GNU/Linux

 

[Jean-Pierre]

I unfortunilty have to report that kernel 4.15.18-5-pve is also faulty and has the same issues. I have tested on two different nodes and the same kworker issue happens.

 

[Marius_B]

I unfortunilty have to report that kernel 4.15.18-5-pve is also faulty and has the same issues. I have tested on two different nodes and the same kworker issue happens.

I agree, after updating to latest version recently we've seen a recurrence of this problem ...

 

[Stoiko Ivanov]

Thanks for bumping this and the report - would you mind opening a bug-report at https://bugzilla.proxmox.com/ ?

Since we cannot currently reproduce the issue locally, it would be helpful if you provide us with a stacktrace of the kworker process which runs at 100%:

cat /proc/<pid>/stack

where <pid> is the kworkers pid.

Apart from that a perf report of the process and the whole system could also provide some insight:

# Gets the perf output for the whole system
perf record -F 99 -a -g -- sleep 60
perf script > out.perf

# Capturing PID X (the kworker):
perf record -F 99 -p <pid> -g -- sleep 60
perf script > out.perf

(probably best to attach the outputs in bugzilla)

EDIT: if possible please also provide the output of pvereport - it would help in maybe isolating the issue to certain setups

 

[qubozik]

@Stoiko Ivanov

Thank you.

I have filed the following bug:

https://bugzilla.proxmox.com/show_bug.cgi?id=1943

I am trying to get the perf reports for you, but I am seeing this error when trying to run perf:

root@node3:~# perf
/usr/bin/perf: line 13: exec: perf_4.15: not found
E: linux-perf-4.15 is not installed.

I am not seeing a package for perf-4.15. I'll research this and attach the reports once this is resolved.

Please let us know if we can provide any further details. Thanks, again.

 

[Stoiko Ivanov]

in pve the package is called `linux-tools-4.15`

 

[qubozik]

Got it. Thank you.

Replied to bug with links to perf reports.

 

[Sorcerer]

Have the same bug here with a ProxMox kernel "PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 GNU/Linux"

cat of the pid stacktrace says " [<0>] 0xffffffffffffffff "

Storage is LVM-Thin and no iptables rules are used.

Will reset the server in the evening, is there a fix for this problem?

 

[sfg19]

I've been plagued with this bug for many kernel versions, but I think I might have corrected my issue. I'm currently running 4.15.18-10-pve. I use LVM-Thin, no ZFS.

I could reproduce this issue on demand. After resetting my node to fix the issue, I could get away with one (maybe two) container restarts before a kworker was at 100% again and the node required another hard reset.

Recently, I found this other thread suggesting to update the BIOS. I looked at mine and found it to be several versions behind.

Since updating the BIOS, I have been able to restart containers without issues for a few days - something that has never been possible before with my configuration. I'm not ready to declare total victory yet, but the BIOS update suggestion has at least improved the situation.

 

[foobar73]

I have replicated this problem on a supermicro chassis on command over the last two days. Updated BIOS and pve versions have made no difference. All I have to do is shut a lxc down and 1 core is pegged by kworker.

pveversion is 5.3-9

planar is X10SDV-8C-TLN4F

intel 10G and 1G on the mb.

 

[seneca214]

Sharing this possible workaround posted on a related thread in case it helps others out: https://forum.proxmox.com/threads/lxc-container-reboot-fails-lxc-becomes-unusable.41264/

Proxmox's support unfortunately hasn't been any help with this. We've been trying to at least find a workaround on our Proxmox servers. Simply as a test, referencing this bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1765980, we setup ip6tables on one of our nodes to block all IPv6 traffic. Since the change, we haven't been able to reproduce the issue on our test server with automated LXC shutdown/startup of containers (this server previously exhibited the issue). I'd be curious to hear if anyone else is able to try a similar approach and see if they can reproduce the issue afterwards.