kernel:[ 9203.691567] watchdog: BUG: soft lockup - CPU#15 stuck for 6802s! [systemd-timesyn:639]

D

dengolius

Well-Known Member
Dec 28, 2016
46
8
48
34
Ukraine
t.me
I've just upgrade PVE server from 8.1.3 to the latest 8.2.2 and got this:
Code: 
sudo apt autoremove
[sudo] password for dengolius:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  pve-kernel-5.15 pve-kernel-5.15.143-1-pve
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 400 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 85017 files and directories currently installed.)
Removing pve-kernel-5.15 (7.4-11) ...
Removing pve-kernel-5.15.143-1-pve (5.15.143-1) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 5.15.143-1-pve /boot/vmlinuz-5.15.143-1-pve
update-initramfs: Deleting /boot/initrd.img-5.15.143-1-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 5.15.143-1-pve /boot/vmlinuz-5.15.143-1-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 5.15.143-1-pve /boot/vmlinuz-5.15.143-1-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 5.15.143-1-pve /boot/vmlinuz-5.15.143-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.8.4-2-pve
Found initrd image: /boot/initrd.img-6.8.4-2-pve
Found linux image: /boot/vmlinuz-6.5.13-5-pve
Found initrd image: /boot/initrd.img-6.5.13-5-pve
Found linux image: /boot/vmlinuz-6.5.13-1-pve
Found initrd image: /boot/initrd.img-6.5.13-1-pve
Found linux image: /boot/vmlinuz-5.13.19-2-pve
Found initrd image: /boot/initrd.img-5.13.19-2-pve
done
dengolius@hub1:~$
Message from syslogd@hub1 at May  3 18:42:33 ...
 kernel:[ 1734.574973] watchdog: Watchdog detected hard LOCKUP on cpu 12

Message from syslogd@hub1 at May  3 18:44:56 ...
 kernel:[ 1923.987764] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:45:24 ...
 kernel:[ 1951.986701] watchdog: BUG: soft lockup - CPU#15 stuck for 48s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:45:52 ...
 kernel:[ 1979.985629] watchdog: BUG: soft lockup - CPU#15 stuck for 75s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:46:20 ...
 kernel:[ 2007.984550] watchdog: BUG: soft lockup - CPU#15 stuck for 101s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:46:48 ...
 kernel:[ 2035.983465] watchdog: BUG: soft lockup - CPU#15 stuck for 127s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:47:16 ...
 kernel:[ 2063.982374] watchdog: BUG: soft lockup - CPU#15 stuck for 153s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:47:44 ...
 kernel:[ 2091.981277] watchdog: BUG: soft lockup - CPU#15 stuck for 179s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:48:12 ...
 kernel:[ 2119.980176] watchdog: BUG: soft lockup - CPU#15 stuck for 205s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:48:40 ...
 kernel:[ 2147.979070] watchdog: BUG: soft lockup - CPU#15 stuck for 231s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:49:08 ...
 kernel:[ 2175.977961] watchdog: BUG: soft lockup - CPU#15 stuck for 257s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:49:36 ...
 kernel:[ 2203.976848] watchdog: BUG: soft lockup - CPU#15 stuck for 283s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:50:04 ...
 kernel:[ 2231.975733] watchdog: BUG: soft lockup - CPU#15 stuck for 309s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:50:32 ...
 kernel:[ 2259.974615] watchdog: BUG: soft lockup - CPU#15 stuck for 335s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:51:00 ...
 kernel:[ 2287.973493] watchdog: BUG: soft lockup - CPU#15 stuck for 361s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:51:28 ...
 kernel:[ 2315.972370] watchdog: BUG: soft lockup - CPU#15 stuck for 387s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:51:56 ...
 kernel:[ 2343.971245] watchdog: BUG: soft lockup - CPU#15 stuck for 413s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:52:24 ...
 kernel:[ 2371.970118] watchdog: BUG: soft lockup - CPU#15 stuck for 440s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:52:52 ...
 kernel:[ 2399.968990] watchdog: BUG: soft lockup - CPU#15 stuck for 466s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:53:20 ...
 kernel:[ 2427.967859] watchdog: BUG: soft lockup - CPU#15 stuck for 492s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:53:48 ...
 kernel:[ 2455.966728] watchdog: BUG: soft lockup - CPU#15 stuck for 518s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:54:16 ...
 kernel:[ 2483.965596] watchdog: BUG: soft lockup - CPU#15 stuck for 544s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:54:44 ...
 kernel:[ 2511.964463] watchdog: BUG: soft lockup - CPU#15 stuck for 570s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:55:12 ...
 kernel:[ 2539.963328] watchdog: BUG: soft lockup - CPU#15 stuck for 596s! [systemd-timesyn:639]

Message from syslogd@hub1 at May  3 18:55:40 ...
 kernel:[ 2567.962193] watchdog: BUG: soft lockup - CPU#15 stuck for 622s! [systemd-timesyn:639]


See my bug report https://bugzilla.proxmox.com/show_bug.cgi?id=3821#c11
 
Last edited:
Looks like I've solved it by disabling systemd-timesyncd service:

sudo systemctl disable systemd-timesyncd

and enabling sudo timedatectl set-ntp true+ uncomment line with
Code: 
FallbackNTP

Bash: 
cat /etc/systemd/timesyncd.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the timesyncd.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=ntp.hetzner.com
FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048
#ConnectionRetrySec=30
#SaveIntervalSec=60
 
Have similar issue on Promox 8.2.2 hosted at Hetzner with a single VM debian having these CPU soft lockup's when running database backups.. Have you found any solution to the problem ?

We already changed hardware twice, I suspect it's a software issue (running AMD EPYC 9454, 512GB RAM 4800Mt/s, NVME DISKS)
 
  • Like
Reactions: dengolius
JustThat said:
Have similar issue on Promox 8.2.2 hosted at Hetzner with a single VM debian having these CPU soft lockup's when running database backups.. Have you found any solution to the problem ?

We already changed hardware twice, I suspect it's a software issue (running AMD EPYC 9454, 512GB RAM 4800Mt/s, NVME DISKS)
Click to expand...
I've applied recommendations from https://forum.proxmox.com/threads/k...6802s-systemd-timesyn-639.146368/#post-665192 so my server is running smoothly for the latest month.

PS: I also failed in issue with Kernel 6.8 on my laptop on Sunday, but it was on Garuda linux so I just reinstalled OS to Xubuntu 24.04, because didn't have time for resolving issues and experiments with Arch :D
 
  • Like
Reactions: JustThat
Ok our problem seem to be solved with Kernel 6.5 indeed.

Default 6.8 kernel has issues in our configuration (AMD EPYC, we also have an intel i9-10940X CPU @ 3.30GHz, that one doesn't seemt o have the issue on 6.8)
 
Spoke too soon, the issue is back

We are using kernel 6.5 + Disabled C-State, and issue still happens...any ideas?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back