[SOLVED] VMs freeze with 100% CPU

jens-maus said:
What we haven't tried yet is using the mitigations=off kernel commandline option like mentioned by @Whatever in the previous post. But what we can postulate to for is that this seems to be not related to Qemu v7 vs. v8 since a simple kernel 5.15 downgrade is enough to make sure the VMs are working smoothly again.
Click to expand...

Please note (e.g. @fiona) as mentioned here (https://forum.proxmox.com/threads/p...ows-server-2019-vms.130727/page-2#post-574617) we have now applied the mitigations=off kernel option to some of our affected Proxmox8/kernel6.2 nodes where we had issues with our Windows2019 RDS Terminal Server VMs going wild into 100%CPU spikes and partly stalling.

After having added that option and booting the kernel 6.2 again (we could only previously solve the issue by downgrading to kernel 5.15) the affected Windows VMs seem to work smoothly again and we could not see any more 100%CPU spikes yet. So for us adding the mitigations=off kernel option seem to have solved the issue (like @Whatever proposed).

However, I would of course be curious to see if other ppl in here which have similar 100%CPU issues since kernel > 5.15 can also solve their issues by adding that kernel option as well. And if this is the case then at least @fiona and the Proxmox developers should have a slight hint where to look at for identifying the root cause.
 
  • Like
Reactions: coenvl
Sorry that I have to revert my latest statement that just using mitigations=off seem to solve our CPU spike/ICMP ping/proxmox console lockup issues with our windows 2019 vm. This alone does not seem to completely solve the issue as we have just learned with increasing number of users logging into these RDS server. See here for more info:

https://forum.proxmox.com/threads/p...ows-server-2019-vms.130727/page-2#post-574637

However, please note that using mitigations=off does seem to help at least to some extend because in our usual tests which we do to reproduce the issue also with only one user connected we did not spot the issue. but now with putting more load on the VMs we could see 100%CPU spikes + ICMP ping request increases appearing again.... So sorry @fiona but the issue seem to be not only related to CPU mitigations handling in newer kernels...
 
  • Like
Reactions: coenvl and Universal4
Hi again, as just stated in my initial problem post (https://forum.proxmox.com/threads/p...cpu-issue-with-windows-server-2019-vms.130727) we could now finally solve our 100%CPU spikes, noVNC console stalling and high ICMP requests against our Windows2019 RDS cluster VMs by using the mitigations=off kernel command option, but more importantly after having completely disabled KSM sharing on our Proxmox8 nodes. It seems that especially the KSM sharing result in temporary unresponsiveness of VMs which are under high memory pressure so that the KSM sharing daemon starts to play an important role. In our case we have serveral Proxmox nodes which host only a single Windows2019 VM which are assigned to occupy about 95% of the RAM of its host (>120GB). And in these cases KSM sharing starts to pop in after a while and seem to result in temporary CPU spikes, noVNC console stalling and high ICMP ping request times against these VMs. Rebooting the hosts into kernel 5.15 seem to immediately solve these issue and the KSM sharing is then in our case able to assign about 100GB of RAM into the KSM sharing mem page. After booting back into kernel 6.2 the KSM daemon is also high on CPU load, but is somehow not able to assign that much amount of memory for KSM sharing and seem to continue endlessly in trying to do so, which could explain the CPU spikes&co which are also visible on the Proxmox hosts itself at the time they happen on the VMs.

However, as said, in our case using mitigations=off kernel command option in addition to having KSM sharing disabled completely also immediately solved our 100%CPU spike and stalling issues and we are now happily using Proxmox8 together with kernel 6.2.

See here for more info:
https://forum.proxmox.com/threads/p...ows-server-2019-vms.130727/page-4#post-575886
 
Thanks jens-maus for your findings. I will definitely try setting `mitigations=off` the next time I have another hanging VM. I think I have to reboot the host, so it involves migrating all other running VMs first. I hope it solves my problem as well, since we have never seen the issue on any of our windows VMs, but only on linux ones (ubuntu and debian). I think I had already tried switching off KSM, but that seemed to have little to no effect. However, I see that it is currently enabled again, I may not have disabled it permanently.

I haven't seen any hanging VMs for about two weeks now (11 days to be precise), so I expect one any time now.
 
Hi @fiona ,

Had another case of this problem. This time, the VM has been stuck for several days...
Usual data:
root@xenon:~# strace -c -p $(cat /var/run/qemu-server/105.pid)
strace: Process 2780175 attached
^Cstrace: Process 2780175 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.03 7.500966 5357 1400 ppoll
0.47 0.035584 6 5352 write
0.19 0.014114 10 1376 read
0.15 0.011657 971 12 fcntl
0.12 0.008742 6 1310 recvmsg
0.03 0.002421 93 26 sendmsg
0.01 0.000950 158 6 accept4
0.00 0.000029 4 6 close
0.00 0.000009 1 6 getsockname
------ ----------- ----------- --------- --------- ----------------
100.00 7.574472 797 9494 total

root@xenon:~# gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/105.pid)[New LWP 2780176]
[New LWP 2780220]
[New LWP 2780221]
[New LWP 2780222]
[New LWP 2780225]
[New LWP 2780274]
[New LWP 1339380]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa19403c0f6 in __ppoll (fds=0x55eb4686d200, nfds=147, timeout=<optimized out>, timeout@entry=0x7ffc805a6ce0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
42 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.

Thread 8 (Thread 0x7fa189114280 (LWP 1339380) "iou-wrk-2780175"):
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 7 (Thread 0x7f9f635f96c0 (LWP 2780274) "vnc_worker"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55eb46889b98) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x55eb46889b98, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007fa193fc5d9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55eb46889b98, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007fa193fc83f8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55eb46889ba8, cond=0x55eb46889b70) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=cond@entry=0x55eb46889b70, mutex=mutex@entry=0x55eb46889ba8) at ./nptl/pthread_cond_wait.c:618
#5 0x000055eb44d506fb in qemu_cond_wait_impl (cond=0x55eb46889b70, mutex=0x55eb46889ba8, file=0x55eb44dddbb4 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:225
#6 0x000055eb447b6fdd in vnc_worker_thread_loop (queue=queue@entry=0x55eb46889b70) at ../ui/vnc-jobs.c:248
#7 0x000055eb447b7ce8 in vnc_worker_thread (arg=arg@entry=0x55eb46889b70) at ../ui/vnc-jobs.c:361
#8 0x000055eb44d4fbe8 in qemu_thread_start (args=0x55eb468a66b0) at ../util/qemu-thread-posix.c:541
#9 0x00007fa193fc8fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fa1940495bc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f9f7bdff6c0 (LWP 2780225) "SPICE Worker"):
#0 0x00007fa19403bfff in __GI___poll (fds=0x7f9f5c0014f0, nfds=2, timeout=2147483647) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007fa1958df9ae in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007fa1958dfcef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007fa196004fa9 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#4 0x00007fa193fc8fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5 0x00007fa1940495bc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7fa1821ff6c0 (LWP 2780222) "CPU 2/KVM"):
#0 __GI___ioctl (fd=33, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1 0x000055eb44bbc55f in kvm_vcpu_ioctl (cpu=cpu@entry=0x55eb463e0230, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2 0x000055eb44bbc6b5 in kvm_cpu_exec (cpu=cpu@entry=0x55eb463e0230) at ../accel/kvm/kvm-all.c:2939
#3 0x000055eb44bbdcfd in kvm_vcpu_thread_fn (arg=arg@entry=0x55eb463e0230) at ../accel/kvm/kvm-accel-ops.c:51
#4 0x000055eb44d4fbe8 in qemu_thread_start (args=0x55eb463e8c80) at ../util/qemu-thread-posix.c:541
#5 0x00007fa193fc8fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6 0x00007fa1940495bc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7fa182dff6c0 (LWP 2780221) "CPU 1/KVM"):
#0 __GI___ioctl (fd=32, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1 0x000055eb44bbc55f in kvm_vcpu_ioctl (cpu=cpu@entry=0x55eb463d6cd0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2 0x000055eb44bbc6b5 in kvm_cpu_exec (cpu=cpu@entry=0x55eb463d6cd0) at ../accel/kvm/kvm-all.c:2939
#3 0x000055eb44bbdcfd in kvm_vcpu_thread_fn (arg=arg@entry=0x55eb463d6cd0) at ../accel/kvm/kvm-accel-ops.c:51
#4 0x000055eb44d4fbe8 in qemu_thread_start (args=0x55eb463df890) at ../util/qemu-thread-posix.c:541
#5 0x00007fa193fc8fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6 0x00007fa1940495bc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7fa183fff6c0 (LWP 2780220) "CPU 0/KVM"):
#0 __GI___ioctl (fd=31, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1 0x000055eb44bbc55f in kvm_vcpu_ioctl (cpu=cpu@entry=0x55eb463a61c0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2 0x000055eb44bbc6b5 in kvm_cpu_exec (cpu=cpu@entry=0x55eb463a61c0) at ../accel/kvm/kvm-all.c:2939
#3 0x000055eb44bbdcfd in kvm_vcpu_thread_fn (arg=arg@entry=0x55eb463a61c0) at ../accel/kvm/kvm-accel-ops.c:51
#4 0x000055eb44d4fbe8 in qemu_thread_start (args=0x55eb463ace10) at ../util/qemu-thread-posix.c:541
#5 0x00007fa193fc8fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6 0x00007fa1940495bc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fa188fa66c0 (LWP 2780176) "call_rcu"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x000055eb44d50d6a in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at ./include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0x55eb45660ce8 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:464
#3 0x000055eb44d5a5c2 in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:261
#4 0x000055eb44d4fbe8 in qemu_thread_start (args=0x55eb45fdfcf0) at ../util/qemu-thread-posix.c:541
#5 0x00007fa193fc8fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6 0x00007fa1940495bc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7fa189114280 (LWP 2780175) "kvm"):
#0 0x00007fa19403c0f6 in __ppoll (fds=0x55eb4686d200, nfds=147, timeout=<optimized out>, timeout@entry=0x7ffc805a6ce0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#1 0x000055eb44d65bee in ppoll (__ss=0x0, __timeout=0x7ffc805a6ce0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:64
#2 qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=2999229611) at ../util/qemu-timer.c:351
#3 0x000055eb44d634ee in os_host_main_loop_wait (timeout=2999229611) at ../util/main-loop.c:308
#4 main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:592
#5 0x000055eb4497faf7 in qemu_main_loop () at ../softmmu/runstate.c:731
#6 0x000055eb44bc6a46 in qemu_default_main () at ../softmmu/main.c:37
#7 0x00007fa193f6718a in __libc_start_call_main (main=main@entry=0x55eb4478c390 <main>, argc=argc@entry=98, argv=argv@entry=0x7ffc805a6ef8) at ../sysdeps/nptl/libc_start_call_main.h:58
#8 0x00007fa193f67245 in __libc_start_main_impl (main=0x55eb4478c390 <main>, argc=98, argv=0x7ffc805a6ef8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc805a6ee8) at ../csu/libc-start.c:381
#9 0x000055eb4478de71 in _start ()
[Inferior 1 (process 2780175) detached]

root@xenon:~# qm config 105
agent: 1
audio0: device=ich9-intel-hda,driver=spice
balloon: 512
bios: ovmf
bootdisk: scsi0
cores: 3
cpu: kvm64
efidisk0: local-lvm:vm-105-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
ide0: none,media=cdrom
ide2: none,media=cdrom
machine: pc-q35-7.2
memory: 8192
name: win10
net0: virtio=9A:00:B8:64:D0:29,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: local-lvm:vm-105-disk-0,discard=on,size=128G
scsihw: virtio-scsi-pci
smbios1: uuid=81e5d251-b177-4bc0-aba0-c0df15073315
sockets: 1
tpmstate0: local-lvm:vm-105-disk-2,size=4M,version=v2.0
vga: qxl
vmgenid: 098c9138-7808-48bb-9d4c-c7987143cabf
root@xenon:~# pveversion -v
proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.3
pve-kernel-5.15: 7.4-4
pve-kernel-6.2.16-4-pve: 6.2.16-5
pve-kernel-6.2.16-3-pve: 6.2.16-3
pve-kernel-5.15.108-1-pve: 5.15.108-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.6
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1
Click to expand...
 
Hi @fiona ,

Just noticed that an Ubuntu VM had also crashed in the same way - this is rare because normally I have this problem only with Windows guests.
root@xenon:~# strace -c -p $(cat /var/run/qemu-server/101.pid) strace: Process 1539 attached
^Cstrace: Process 1539 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
97.19 4.956876 1546 3206 ppoll
1.44 0.073655 5 12368 write
0.62 0.031579 9 3182 read
0.32 0.016397 5 3026 recvmsg
0.26 0.013509 225 60 sendmsg
0.16 0.008017 668 12 close
0.00 0.000029 2 12 accept4
0.00 0.000011 0 24 fcntl
0.00 0.000010 0 12 getsockname
------ ----------- ----------- --------- --------- ----------------
100.00 5.100083 232 21902 total

root@xenon:~# gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/101.pid)[New LWP 1540]
[New LWP 1565]
[New LWP 1566]
[New LWP 1712]
[New LWP 1205502]

warning: File "/usr/lib/x86_64-linux-gnu/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /usr/lib/x86_64-linux-gnu/libthread_db.so.1
line to your configuration file "/root/.config/gdb/gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
0x00007f6c8496b0f6 in __ppoll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, timeout@entry=0x7fff649f77f0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:64
64 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.

Thread 6 (LWP 1205502 "iou-wrk-1539"):
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 5 (LWP 1712 "vnc_worker"):
#0 0x00007f6c848f4d36 in __futex_abstimed_wait_common (futex_word=<optimized out>, expected=0, clockid=<optimized out>, abstime=<optimized out>, private=<optimized out>, cancel=<optimized out>) at ./nptl/futex-internal.c:103
#1 0x00007f6c848f73f8 in futex_fatal_error () at ../sysdeps/nptl/futex-internal.h:87
#2 futex_wake (private=<optimized out>, processes_to_wake=<optimized out>, futex_word=<optimized out>) at ../sysdeps/nptl/futex-internal.h:225
#3 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55b6abbf6028, cond=0x55b6abbf5ff0) at ./nptl/pthread_cond_wait.c:590
#4 ___pthread_cond_wait (cond=cond@entry=0x55b6abbf5ff0, mutex=mutex@entry=0x55b6abbf6028) at ./nptl/pthread_cond_wait.c:618
#5 0x000055b6a984a6fb in qemu_cond_wait_impl (cond=0x55b6abbf5ff0, mutex=0x55b6abbf6028, file=0x55b6a98d7bb4 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:225
#6 0x000055b6a92b0fdd in vnc_worker_thread_loop (queue=queue@entry=0x55b6abbf5ff0) at ../ui/vnc-jobs.c:248
#7 0x000055b6a92b1ce8 in vnc_worker_thread (arg=arg@entry=0x55b6abbf5ff0) at ../ui/vnc-jobs.c:361
#8 0x000055b6a9849be8 in qemu_thread_start (args=0x55b6abbf5ae0) at ../util/qemu-thread-posix.c:541
#9 0x00007f6c848f7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:571
#10 0x00007f6c849785bc in __closefrom_fallback (from=0, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:52
#11 0x0000000000000000 in ?? ()

Thread 4 (LWP 1566 "CPU 1/KVM"):
#0 0x00007f6c8496cafb in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6c00000010 in ?? ()
#2 0x00007f6c73ffa1d0 in ?? ()
#3 0x00007f6c73ffa190 in ?? ()
#4 0x0e7099e96d8ae100 in ?? ()
#5 0x0000000000000000 in ?? ()

Thread 3 (LWP 1565 "CPU 0/KVM"):
#0 0x00007f6c8496cafb in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6c00000010 in ?? ()
#2 0x00007f6c78f9a1d0 in ?? ()
#3 0x00007f6c78f9a190 in ?? ()
#4 0x0e7099e96d8ae100 in ?? ()
#5 0x0000000000000000 in ?? ()

Thread 2 (LWP 1540 "call_rcu"):
#0 0x00007f6c849704f9 in setlogmask (pmask=-1441420056) at ./misc/syslog.c:382
#1 0x0000000000000000 in ?? ()

Thread 1 (LWP 1539 "kvm"):
#0 0x00007f6c8496b0f6 in __ppoll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, timeout@entry=0x7fff649f77f0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:64
#1 0x000055b6a985fbee in ppoll (__ss=0x0, __timeout=0x7fff649f77f0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:64
#2 qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=1799098203) at ../util/qemu-timer.c:351
#3 0x000055b6a985d4ee in os_host_main_loop_wait (timeout=1799098203) at ../util/main-loop.c:308
#4 main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:592
#5 0x000055b6a9479af7 in qemu_main_loop () at ../softmmu/runstate.c:731
#6 0x000055b6a96c0a46 in qemu_default_main () at ../softmmu/main.c:37
#7 0x00007f6c8489618a in __libc_start_call_main (main=0x55b6a9286390 <main>, argc=66, argv=0x7fff649f7a08) at ../sysdeps/nptl/libc_start_call_main.h:51
#8 0x00007f6c84896245 in __libc_start_main_impl (main=0x55b6a9286390 <main>, argc=66, argv=0x7fff649f7a08, init=0x7f6c86bf9020 <_rtld_global>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff649f79f8) at ../csu/libc-start.c:337
#9 0x000055b6a9287e71 in _start ()
[Inferior 1 (process 1539) detached]
root@xenon:~# qm config 101
agent: 1
balloon: 1024
boot: dcn
bootdisk: scsi0
cores: 2
cpu: host
ide2: none,media=cdrom
memory: 8192
name: ubuntu-desktop
net0: virtio=1E:61:DB:CE:63:B3,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local-lvm:vm-101-disk-1,discard=on,size=48G
scsihw: virtio-scsi-pci
smbios1: uuid=3eb3c166-27c1-4585-87cc-d66198b5f57e
sockets: 1
tablet: 0
vga: std
Click to expand...
 
Hi,
VivienM said:
Hi @fiona ,

Had another case of this problem. This time, the VM has been stuck for several days...
Usual data:
Click to expand...
that one looks like earlier ones.

VivienM said:
Hi @fiona ,

Just noticed that an Ubuntu VM had also crashed in the same way - this is rare because normally I have this problem only with Windows guests.
Click to expand...
but this one is new
Code: 
Thread 2 (LWP 1540 "call_rcu"):
#0  0x00007f6c849704f9 in setlogmask (pmask=-1441420056) at ./misc/syslog.c:382
#1  0x0000000000000000 in ?? ()
The pmask=-1441420056 parameter seems wrong at a glance and I can't find any direct calls of this in the QEMU source code. It's also something I'd expect to be called during initialization. So maybe this is the PLT corruption scenario again, but with a different kind of corruption.

Did you already try switching away from the default aio=io_uring for the VMs drives? We had one user where the corruption didn't happen afterwards, but the issue is rare so it's not clear it was actually because of that.

Did you already try the workaround with mitigations=off? EDIT: Of course this will not be a long-term solution depending on how untrusted your VMs are, but it would be interesting to see if it helps.

Unfortunately, we still haven't been able to reproduce the issue.
 
Last edited:
fiona said:
but this one is new
Code: 
Thread 2 (LWP 1540 "call_rcu"):
#0  0x00007f6c849704f9 in setlogmask (pmask=-1441420056) at ./misc/syslog.c:382
#1  0x0000000000000000 in ?? ()
The pmask=-1441420056 parameter seems wrong at a glance and I can't find any direct calls of this in the QEMU source code. It's also something I'd expect to be called during initialization. So maybe this is the PLT corruption scenario again, but with a different kind of corruption.

Did you already try switching away from the default aio=io_uring for the VMs drives? We had one user where the corruption didn't happen afterwards, but the issue is rare so it's not clear it was actually because of that.

Did you already try the workaround with mitigations=off? EDIT: Of course this will not be a long-term solution depending on how untrusted your VMs are, but it would be interesting to see if it helps.

Unfortunately, we still haven't been able to reproduce the issue.
Click to expand...
Hi @fiona,

I will try aio=native now. Might as well switch this to a version 8.0 machine now too.

Haven't tried the mitigations=off. May try that next time I reboot the host...

One other thing I would add: I've seen this issue with Windows guests (frequently), at least one Ubuntu guest (much less frequently), but have never, ever, ever seen it with a FreeBSD guest. This may be a coincidence or it may not be (it's just a random observation I had), but the FreeBSD guest (and the Ubuntu guest that I don't think has ever had this problem) has a text-mode console, while Windows and the Ubuntu desktop guest that that second set of data came from both would have graphical consoles.
 
VivienM said:
One other thing I would add: I've seen this issue with Windows guests (frequently), at least one Ubuntu guest (much less frequently), but have never, ever, ever seen it with a FreeBSD guest. This may be a coincidence or it may not be (it's just a random observation I had), but the FreeBSD guest (and the Ubuntu guest that I don't think has ever had this problem) has a text-mode console, while Windows and the Ubuntu desktop guest that that second set of data came from both would have graphical consoles.
Click to expand...

I do have the issue on Debian guests (of varying versions from Debian 10 to Debian 12) and none of them have a GUI installed.

I'll try to upgrade my cluster to Proxmox 8, turn off KSM and use mitigations=off and see if it improves things but rebooting the whole cluster is lots of work and I need to inform our customers before of potential performance degradation while VMs are being shuffled all around the cluster.
 
I believe I was having the same issue. Kept seeing:

Watchdog: BUG: soft lockup - CPU#XX

SSH session closed out, an external uptime monitor wasn't able to ping, the Proxmox console freezes up and CPU shoots through the roof. Tried a lot of settings changes. Even tried a complete fresh reinstall of the VM. After 9 hours of various things and pulling my hair out I managed to boot into Kernel 5.15 under Proxmox 8.0.3 and it has been running solid ever since.
 
lukemax said:
I believe I was having the same issue. Kept seeing:

Watchdog: BUG: soft lockup - CPU#XX

SSH session closed out, an external uptime monitor wasn't able to ping, the Proxmox console freezes up and CPU shoots through the roof. Tried a lot of settings changes. Even tried a complete fresh reinstall of the VM. After 9 hours of various things and pulling my hair out I managed to boot into Kernel 5.15 under Proxmox 8.0.3 and it has been running solid ever since.
Click to expand...

Reinstall ? So I could recover every crashed VM just by doing a hard power off and then restarting it, and it seems to be the same for other people affected.
 
kilobug said:
Reinstall ? So I could recover every crashed VM just by doing a hard power off and then restarting it, and it seems to be the same for other people affected.
Click to expand...
I realized after reading this thread that I could recover every crashed VM by hibernating it and then powering it back on. No hard power off actually necessary...
 
udotirol said:
ok, then I will open an support ticket the next time one of our VMs freezes. Not sure how fast it happens, but it will, sooner or later ... I'll keep you posted.
Click to expand...
Coming back from vacation, I indeed found a stuck Win10 VM. Unfortunately I only noticed it now and I also cannot say how long it has been stuck, because this is an unmonitored VM.

The VM doesn't have the proposed mitigations=off configured, because this has come up only during my vacation.

@fiona I shall open a support ticket now.

And just for completeness, here's the debug info:

Code: 
strace -c -p $(cat /var/run/qemu-server/150.pid)
strace: Process 4980 attached
^Cstrace: Process 4980 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.56   32.207030       22745      1416           ppoll
  0.35    0.112088          20      5360           write
  0.05    0.015724          11      1312           recvmsg
  0.04    0.013870          10      1378           read
  0.00    0.001018          39        26           sendmsg
  0.00    0.000028           4         6           accept4
  0.00    0.000023           3         6           close
  0.00    0.000006           0        12           fcntl
  0.00    0.000005           0         6           getsockname
------ ----------- ----------- --------- --------- ----------------
100.00   32.349792        3397      9522           total

root@herkules:~# gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/150.pid)
[New LWP 4981]
[New LWP 4982]
[New LWP 4984]
[New LWP 5007]
[New LWP 5008]
[New LWP 5009]
[New LWP 5010]
[New LWP 5011]
[New LWP 5013]
[New LWP 5014]
[New LWP 5015]
[New LWP 5577]
[New LWP 171513]
[New LWP 171612]
[New LWP 1695625]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f660cc55a66 in __ppoll (fds=0x562476093340, nfds=141, timeout=<optimized out>, timeout@entry=0x7ffc13cc45b0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
44    ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.

Thread 16 (Thread 0x7f6601808700 (LWP 1695625) "iou-wrk-4982"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 15 (Thread 0x7f5ff23ff700 (LWP 171612) "iou-wrk-5010"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 14 (Thread 0x7f5ff3bff700 (LWP 171513) "iou-wrk-5008"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 13 (Thread 0x7f5ff17ff700 (LWP 5577) "iou-wrk-5011"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 12 (Thread 0x7f6600e47700 (LWP 5015) "iou-wrk-5007"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 11 (Thread 0x7f5fe2dbf700 (LWP 5014) "vnc_worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x562475fbd248) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x562475fbd258, cond=0x562475fbd220) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x562475fbd220, mutex=mutex@entry=0x562475fbd258) at pthread_cond_wait.c:638
#3  0x00005624748469cb in qemu_cond_wait_impl (cond=0x562475fbd220, mutex=0x562475fbd258, file=0x5624748bd434 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:220
#4  0x00005624742d55c3 in vnc_worker_thread_loop (queue=0x562475fbd220) at ../ui/vnc-jobs.c:248
#5  0x00005624742d6288 in vnc_worker_thread (arg=arg@entry=0x562475fbd220) at ../ui/vnc-jobs.c:361
#6  0x0000562474845e89 in qemu_thread_start (args=0x7f5fe2dba3f0) at ../util/qemu-thread-posix.c:505
#7  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f5fe3fff700 (LWP 5013) "SPICE Worker"):
#0  0x00007f660cc5596f in __GI___poll (fds=0x7f5fcc001360, nfds=2, timeout=2147483647) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f660e0cb0ae in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f660e0cb40b in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f660e774fe7 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#4  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f5ff17ff700 (LWP 5011) "CPU 4/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475fb2c00, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475fb2c00) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475fb2c00) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff17fa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f5ff23ff700 (LWP 5010) "CPU 3/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475faae10, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475faae10) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475faae10) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff23fa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f5ff2fff700 (LWP 5009) "CPU 2/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475fa3160, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475fa3160) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475fa3160) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff2ffa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f5ff3bff700 (LWP 5008) "CPU 1/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475f9b340, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475f9b340) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475f9b340) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff3bfa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f6600e47700 (LWP 5007) "CPU 0/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475f6b7c0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475f6b7c0) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475f6b7c0) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f6600e423f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f66022751c0 (LWP 4984) "iou-wrk-4980"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 3 (Thread 0x7f6601808700 (LWP 4982) "kvm"):
#0  0x00007f660cc55a66 in __ppoll (fds=0x7f65f4002800, nfds=5, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
#1  0x000056247485ae6d in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
#2  0x0000562474843999 in fdmon_poll_wait (ctx=0x562475cc76f0, ready_list=0x7f6601803368, timeout=-1) at ../util/fdmon-poll.c:80
#3  0x0000562474843076 in aio_poll (ctx=0x562475cc76f0, blocking=blocking@entry=true) at ../util/aio-posix.c:660
#4  0x00005624746fc946 in iothread_run (opaque=opaque@entry=0x562475b96300) at ../iothread.c:67
#5  0x0000562474845e89 in qemu_thread_start (args=0x7f66018033f0) at ../util/qemu-thread-posix.c:505
#6  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f660210a700 (LWP 4981) "call_rcu"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000056247484704a in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at /build/pve-qemu/pve-qemu-kvm-7.2.0/include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x5624750a8328 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:430
#3  0x000056247484f94a in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:261
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f66021053f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f66022751c0 (LWP 4980) "kvm"):
#0  0x00007f660cc55a66 in __ppoll (fds=0x562476093340, nfds=141, timeout=<optimized out>, timeout@entry=0x7ffc13cc45b0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
#1  0x000056247485ae11 in ppoll (__ss=0x0, __timeout=0x7ffc13cc45b0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=2209966553) at ../util/qemu-timer.c:351
#3  0x0000562474858675 in os_host_main_loop_wait (timeout=2209966553) at ../util/main-loop.c:315
#4  main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:606
#5  0x0000562474475191 in qemu_main_loop () at ../softmmu/runstate.c:739
#6  0x00005624742aeaa7 in qemu_default_main () at ../softmmu/main.c:37
#7  0x00007f660cb89d0a in __libc_start_main (main=0x5624742a9c60 <main>, argc=81, argv=0x7ffc13cc4778, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc13cc4768) at ../csu/libc-start.c:308
#8  0x00005624742ae9da in _start ()
[Inferior 1 (process 4980) detached]
 
Last edited:
been having this issue since upgrading to V8 from v7, mainly on Win VMs with large amount of RAM.
 
Statusupdate:
I have disabled KSM on some nodes where we have trouble with freezed VMs (windows + ubuntu). On some VMs the ballooning is disabled, but not on all. Most VMs use aio=threads.

Since disabled KSM we have no freezed VM on this nodes (without set mitigations=off).

We updated from 5.15 to 5.19/6.2 due the issue, that an migration of an VM can produce an node where all VMs are dead.
This issue was fixed in 5.19 but not in 5.15 - or is it fixed now?

So we have the choice between some freezed VMs with KSM, dead nodes on migration or disabling KSM?!

Unfortunality some nodes are a little bit overcommited, so KSM can't disabled on all nodes without memory-upgrade.
Due this i'm not able to update the nodes, which are running an older kernel yet.

Udo
 
we have booted all nodes in 5.15.108-1 so far no issues but will see how it goes, we have left KSM enabled and not set the mitigations=off
 
udo said:
Statusupdate:
I have disabled KSM on some nodes where we have trouble with freezed VMs (windows + ubuntu). On some VMs the ballooning is disabled, but not on all. Most VMs use aio=threads.

Since disabled KSM we have no freezed VM on this nodes (without set mitigations=off).

We updated from 5.15 to 5.19/6.2 due the issue, that an migration of an VM can produce an node where all VMs are dead.
This issue was fixed in 5.19 but not in 5.15 - or is it fixed now?

So we have the choice between some freezed VMs with KSM, dead nodes on migration or disabling KSM?!

Unfortunality some nodes are a little bit overcommited, so KSM can't disabled on all nodes without memory-upgrade.
Due this i'm not able to update the nodes, which are running an older kernel yet.

Udo
Click to expand...
I remember having tried with both KSM and/or ballooning disabled months ago, to no avail. Maybe it improves the situation, but at least for us it did not make a real difference.
 
  • Like
Reactions: pandada8
My current solution is to stop and restart each VM once a day through a backup-job. Since the previous freezes only appeared after several weeks of uptime, I hope that I can now prevent this.
 
udo said:
Statusupdate:
I have disabled KSM on some nodes where we have trouble with freezed VMs (windows + ubuntu). On some VMs the ballooning is disabled, but not on all. Most VMs use aio=threads.

Since disabled KSM we have no freezed VM on this nodes (without set mitigations=off).
Click to expand...
Interesting. Thank you for the update. Maybe KSM causes the PLT corruption issue you had, so I'd suggest other people with that issue (you should be able to see if it's that one in your GDB backtrace) give that workaround a spin too.

udotirol said:
I remember having tried with both KSM and/or ballooning disabled months ago, to no avail. Maybe it improves the situation, but at least for us it did not make a real difference.
Click to expand...
In your case it doesn't seem to be the PLT corruption issue. It's likely that there are multiple issues with similar symptoms reported in this thread.

udo said:
We updated from 5.15 to 5.19/6.2 due the issue, that an migration of an VM can produce an node where all VMs are dead.
This issue was fixed in 5.19 but not in 5.15 - or is it fixed now?
Click to expand...
Do you mean the issue with AMD CPUs where other VMs would freeze/see time jumps? Yes, that has been fixed since a long time in kernels >= 5.15.74-1.

udotirol said:
Coming back from vacation, I indeed found a stuck Win10 VM. Unfortunately I only noticed it now and I also cannot say how long it has been stuck, because this is an unmonitored VM.

The VM doesn't have the proposed mitigations=off configured, because this has come up only during my vacation.

@fiona I shall open a support ticket now.

And just for completeness, here's the debug info:

Code: 
strace -c -p $(cat /var/run/qemu-server/150.pid)
strace: Process 4980 attached
^Cstrace: Process 4980 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.56   32.207030       22745      1416           ppoll
  0.35    0.112088          20      5360           write
  0.05    0.015724          11      1312           recvmsg
  0.04    0.013870          10      1378           read
  0.00    0.001018          39        26           sendmsg
  0.00    0.000028           4         6           accept4
  0.00    0.000023           3         6           close
  0.00    0.000006           0        12           fcntl
  0.00    0.000005           0         6           getsockname
------ ----------- ----------- --------- --------- ----------------
100.00   32.349792        3397      9522           total

root@herkules:~# gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/150.pid)
[New LWP 4981]
[New LWP 4982]
[New LWP 4984]
[New LWP 5007]
[New LWP 5008]
[New LWP 5009]
[New LWP 5010]
[New LWP 5011]
[New LWP 5013]
[New LWP 5014]
[New LWP 5015]
[New LWP 5577]
[New LWP 171513]
[New LWP 171612]
[New LWP 1695625]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f660cc55a66 in __ppoll (fds=0x562476093340, nfds=141, timeout=<optimized out>, timeout@entry=0x7ffc13cc45b0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
44    ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.

Thread 16 (Thread 0x7f6601808700 (LWP 1695625) "iou-wrk-4982"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 15 (Thread 0x7f5ff23ff700 (LWP 171612) "iou-wrk-5010"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 14 (Thread 0x7f5ff3bff700 (LWP 171513) "iou-wrk-5008"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 13 (Thread 0x7f5ff17ff700 (LWP 5577) "iou-wrk-5011"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 12 (Thread 0x7f6600e47700 (LWP 5015) "iou-wrk-5007"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 11 (Thread 0x7f5fe2dbf700 (LWP 5014) "vnc_worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x562475fbd248) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x562475fbd258, cond=0x562475fbd220) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x562475fbd220, mutex=mutex@entry=0x562475fbd258) at pthread_cond_wait.c:638
#3  0x00005624748469cb in qemu_cond_wait_impl (cond=0x562475fbd220, mutex=0x562475fbd258, file=0x5624748bd434 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:220
#4  0x00005624742d55c3 in vnc_worker_thread_loop (queue=0x562475fbd220) at ../ui/vnc-jobs.c:248
#5  0x00005624742d6288 in vnc_worker_thread (arg=arg@entry=0x562475fbd220) at ../ui/vnc-jobs.c:361
#6  0x0000562474845e89 in qemu_thread_start (args=0x7f5fe2dba3f0) at ../util/qemu-thread-posix.c:505
#7  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f5fe3fff700 (LWP 5013) "SPICE Worker"):
#0  0x00007f660cc5596f in __GI___poll (fds=0x7f5fcc001360, nfds=2, timeout=2147483647) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f660e0cb0ae in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f660e0cb40b in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f660e774fe7 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#4  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f5ff17ff700 (LWP 5011) "CPU 4/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475fb2c00, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475fb2c00) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475fb2c00) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff17fa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f5ff23ff700 (LWP 5010) "CPU 3/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475faae10, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475faae10) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475faae10) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff23fa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f5ff2fff700 (LWP 5009) "CPU 2/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475fa3160, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475fa3160) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475fa3160) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff2ffa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f5ff3bff700 (LWP 5008) "CPU 1/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475f9b340, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475f9b340) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475f9b340) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f5ff3bfa3f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f6600e47700 (LWP 5007) "CPU 0/KVM"):
#0  0x00007f660cc57237 in ioctl () at ../sysdeps/unix/syscall-template.S:120
#1  0x00005624746be997 in kvm_vcpu_ioctl (cpu=cpu@entry=0x562475f6b7c0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3035
#2  0x00005624746beb01 in kvm_cpu_exec (cpu=cpu@entry=0x562475f6b7c0) at ../accel/kvm/kvm-all.c:2850
#3  0x00005624746c017d in kvm_vcpu_thread_fn (arg=arg@entry=0x562475f6b7c0) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f6600e423f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f66022751c0 (LWP 4984) "iou-wrk-4980"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 3 (Thread 0x7f6601808700 (LWP 4982) "kvm"):
#0  0x00007f660cc55a66 in __ppoll (fds=0x7f65f4002800, nfds=5, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
#1  0x000056247485ae6d in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
#2  0x0000562474843999 in fdmon_poll_wait (ctx=0x562475cc76f0, ready_list=0x7f6601803368, timeout=-1) at ../util/fdmon-poll.c:80
#3  0x0000562474843076 in aio_poll (ctx=0x562475cc76f0, blocking=blocking@entry=true) at ../util/aio-posix.c:660
#4  0x00005624746fc946 in iothread_run (opaque=opaque@entry=0x562475b96300) at ../iothread.c:67
#5  0x0000562474845e89 in qemu_thread_start (args=0x7f66018033f0) at ../util/qemu-thread-posix.c:505
#6  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f660210a700 (LWP 4981) "call_rcu"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000056247484704a in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at /build/pve-qemu/pve-qemu-kvm-7.2.0/include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x5624750a8328 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:430
#3  0x000056247484f94a in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:261
#4  0x0000562474845e89 in qemu_thread_start (args=0x7f66021053f0) at ../util/qemu-thread-posix.c:505
#5  0x00007f660cd41ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f660cc61a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f66022751c0 (LWP 4980) "kvm"):
#0  0x00007f660cc55a66 in __ppoll (fds=0x562476093340, nfds=141, timeout=<optimized out>, timeout@entry=0x7ffc13cc45b0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
#1  0x000056247485ae11 in ppoll (__ss=0x0, __timeout=0x7ffc13cc45b0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=2209966553) at ../util/qemu-timer.c:351
#3  0x0000562474858675 in os_host_main_loop_wait (timeout=2209966553) at ../util/main-loop.c:315
#4  main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:606
#5  0x0000562474475191 in qemu_main_loop () at ../softmmu/runstate.c:739
#6  0x00005624742aeaa7 in qemu_default_main () at ../softmmu/main.c:37
#7  0x00007f660cb89d0a in __libc_start_main (main=0x5624742a9c60 <main>, argc=81, argv=0x7ffc13cc4778, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc13cc4768) at ../csu/libc-start.c:308
#8  0x00005624742ae9da in _start ()
[Inferior 1 (process 4980) detached]
Click to expand...
Thank you! We (@fweber and me) took a good look at that VM, but unfortunately were not able to find the root cause yet. Observations:
  • There are rather few events in the main loop compared to a functioning VM.
  • The vCPUs threads are all running busy with 100% and never exit out of KVM. They usually would for interrupts/IO/etc. It's not clear why unfortunately and it's not really feasible to look into what they are doing. The only thing I could think of is doing some kernel tracing for KVM functions to get a better picture. And it's also not clear if the root cause lies here of it's just another symptom.
  • There is no IO going on AFAWCS.
This is why the ppoll calls take up most of the time in the strace, since there are essentially no other syscalls and it just waits for those fewer than usual events.
 
Thanks @fiona and @fweber for your efforts here, this seems to be a particularly nasty bug ...

VivienM said:
I realized after reading this thread that I could recover every crashed VM by hibernating it and then powering it back on. No hard power off actually necessary...
Click to expand...
I can confirm this, too, I could indeed revive such a stuck VM by hibernating and then resuming it:

Code: 
# qm suspend 150 --todisk 1
  Logical volume "vm-150-state-suspend-2023-08-03" created.
State saved, quitting
#
# qm resume 150
Resuming suspended VM
activating and using 'herkules_lvm_nvme:vm-150-state-suspend-2023-08-03' as vmstate
Resumed VM, removing state
  Logical volume "vm-150-state-suspend-2023-08-03" successfully removed

That's at least something.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back