VM CPU issues: watchdog: BUG: soft lockup - CPU#7 stuck for 22s!

[cquest]

Got the same error on my Proxmox v8 host, without VM running on it.

 

[vcloud]

I am getting "watchdog: BUG: soft lockup - CPU#x stuck..." message on a host with local-only storage, no VMs, no ZFS, running Proxmox 9.0.1. It's just Proxmox running alone off local NVMe SSD.

 

[RolandK]

please install all updates first before reporting issues, as things may have already been fixed.

 

[vcloud]

please install all updates first before reporting issues, as things may have already been fixed.

There were no available updates at the time I observed this message (no-subscription repositories).

 

[RolandK]

>running Proxmox 9.0.1

so, why are you running proxmox 9.0.1 instead of 9.0.11 ?

 

[vcloud]

>running Proxmox 9.0.1

so, why are you running proxmox 9.0.1 instead of 9.0.11 ?

Sorry, typo:
PU(s) 16 x 13th Gen Intel(R) Core(TM) i5-13500H (1 Socket)
Kernel Version Linux 6.14.11-4-pve (2025-10-10T08:04Z)
Boot Mode EFI (Secure Boot)
Manager Version pve-manager/9.0.11/3bf5476b8a4699e2

 

[falcon4fun]

I am getting "watchdog: BUG: soft lockup - CPU#x stuck..." message on a host with local-only storage, no VMs, no ZFS, running Proxmox 9.0.1. It's just Proxmox running alone off local NVMe SSD.

Same. Yesterday got. But have CIFS to another machine from host side mapped to one LXC (PBS). Latest version too.

System Information Manufacturer: MINIX | Product Name: NEO Z150-0dB
Kernel Version Linux 6.14.11-4-pve (2025-10-10T08:04Z)
Boot Mode EFI (Secure Boot)
Manager Version pve-manager/9.0.11/3bf5476b8a4699e2

Nov 03 22:01:55 pve pvestatd[979]: proxmox-backup-client failed: Error: http request timed out
Nov 03 22:01:55 pve pvestatd[979]: status update time (120.242 seconds)
Nov 03 22:02:56 pve kernel: CIFS: VFS: \\192.168.0.100 has not responded in 180 seconds. Reconnecting...
Nov 03 22:02:56 pve kernel: CIFS: VFS: close cached dir rc -11
Nov 03 22:03:06 pve pvestatd[979]: proxmox-backup-client failed: Error: EHOSTDOWN: Host is down
Nov 03 22:03:07 pve pvestatd[979]: status update time (71.617 seconds)
Nov 03 22:03:43 pve kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [tokio-runtime-w:622612]
Nov 03 22:03:43 pve kernel: Modules linked in: dm_snapshot bluetooth nf_conntrack_netlink xt_nat xt_tcpudp macvlan xt_conntrack xt_MASQUERADE xt_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype nft_compat xfrm_user xfrm_algo overlay cmac nls_utf8 cifs cifs_arc4 nls_ucs2_utils rdma_cm >
Nov 03 22:03:43 pve kernel:  x86_pkg_temp_thermal soundwire_bus intel_powerclamp snd_soc_sdca coretemp snd_soc_avs kvm_intel snd_soc_hda_codec snd_hda_ext_core i915 snd_soc_core kvm snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel sch_fq_codel snd_intel_dspcfg irqbypass snd_intel_sdw_acpi polyval_clmulni polyv>
Nov 03 22:03:43 pve kernel: CPU: 2 UID: 34 PID: 622612 Comm: tokio-runtime-w Tainted: P        W  O       6.14.11-4-pve #1

 

[lawipac]

The io_thread method cannot help with OS installation, since the ISO was used and the system became unresponsive during the process. I'm on versoin 9.

The only way to bypass this soft freeze is to stop all other VMs, which is very annoying. Otherwise, no new VM can be 'installed'.

 

[techglitch42]

I'm on Proxmox 9.0.11 with a similar problem. For storage they were using local, LVM-Thin storage. It should be pretty fast storage, it's just LVM-Thin running on some NVME disks.

I already had IO thread checked (aka iothread=1)

And my 2 VM's were using 'VirtIO SCSI single' for the SCSI Controller. But the CPU was set to something other than "host"... so I just changed the CPU to 'host', and while I was at it I set the Async IO (aka aio) to "threads" instead of the default of 'io_uring'.

I don't know if reducing the cores, changing CPU to host, and setting Async IO to 'threads' (aka aio=threads) will fix the issue but it's all I could come up with after going over this forum post and comments.

----

EDIT: Nope, that did not do the trick. i don't see the error in the kernel logs in the VM's anymore but there's still clearly something going on with resources, any type of activity in one VM will cause the other VM to become unresponsive until i stop the activity in the 1st VM. This is very odd.

 

[Geg0r]

I have exactly the same problems since a view days...
Not all VMs are effected but some (randomly) and resetting is the only chance to get them back.



Proxmox 9.1.2
Linux proxmox 6.17.2-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.2-2 (2025-11-26T12:33Z) x86_64
Boot Mode EFI (Secure Boot)
pve-manager/9.1.2/9d436f37a0ac4172 (running kernel: 6.17.2-2-pve)


Any ideas?

 

[proxman4]

Same here : in a vm. It happens at backup time :

déc. 06 21:04:20 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 06 21:04:32 nextcloud kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 149s! [kcompactd0:67]
déc. 06 21:04:32 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 06 21:04:44 nextcloud kernel: watchdog: BUG: soft lockup - CPU#7 stuck for 101s! [kworker/7:1:963237]
déc. 06 21:04:44 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 06 21:04:48 nextcloud kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 179s! [kworker/2:3:981254]
déc. 06 21:04:48 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 06 21:04:51 nextcloud kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 174s! [php-fpm8.3:981202]
déc. 06 21:04:51 nextcloud kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 148s! [php-fpm8.3:982878]
déc. 06 21:04:51 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 06 21:04:51 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 11 21:01:00 nextcloud kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:3719795]
déc. 11 21:01:00 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 12 21:02:40 nextcloud kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [kcompactd0:67]
déc. 12 21:02:40 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 12 21:03:08 nextcloud kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 48s! [kcompactd0:67]
déc. 12 21:03:09 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 12 21:03:22 nextcloud kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 72s! [swapper/4:0]
déc. 12 21:03:22 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 12 21:03:22 nextcloud kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 69s! [kworker/u16:0:42652]
déc. 12 21:03:22 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 12 21:03:56 nextcloud kernel: watchdog: BUG: soft lockup - CPU#7 stuck for 26s! [kworker/7:1:30049]
déc. 12 21:03:56 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 18 21:00:56 nextcloud kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:3:2805252]
déc. 18 21:00:56 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 18 21:01:16 nextcloud kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kcompactd0:67]
déc. 18 21:01:16 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 18 21:01:24 nextcloud kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 52s! [kworker/3:3:2805252]
déc. 18 21:01:24 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
déc. 18 21:01:39 nextcloud kernel: watchdog: BUG: soft lockup - CPU#7 stuck for 43s! [postgres:2823746]
déc. 18 21:01:39 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
janv. 31 21:01:32 nextcloud kernel: watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/5:1:3754200]
janv. 31 21:01:32 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
févr. 07 21:01:48 nextcloud kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:2696771]
févr. 07 21:01:48 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
févr. 07 21:01:52 nextcloud kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kcompactd0:67]
févr. 07 21:01:52 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
févr. 07 21:02:16 nextcloud kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 48s! [kworker/3:1:2696771]
févr. 07 21:02:16 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
févr. 07 21:02:20 nextcloud kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 48s! [kcompactd0:67]
févr. 07 21:02:20 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
févr. 07 21:02:24 nextcloud kernel: watchdog: BUG: soft lockup - CPU#7 stuck for 51s! [php-fpm8.3:2719309]
févr. 07 21:02:24 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60
févr. 07 21:02:24 nextcloud kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 51s! [php-fpm8.3:2719226]
févr. 07 21:02:24 nextcloud kernel:  ? lockup_detector_update_enable+0x60/0x60

 

[RolandK]

do you have backup fleecing enabled @proxman4 ?