Random VM crashes with a SPICE vm: QEMU free(): corrupted unsorted chunks

[DeeMaas]

apt install libspice-server1-dbgsym

The coredumpctl gdb -1 output is equal to the above.

(gdb) thread apply all backtrace

Thread 10 (Thread 0x7fb73bfff6c0 (LWP 1561872)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fb73bff9fe0, op=393, expected=0, futex_word=0x55c10bb64880) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x55c10bb64880, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fb73bff9fe0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007fb888396efb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c10bb64880, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fb73bff9fe0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fb88839983c in __pthread_cond_wait_common (abstime=0x7fb73bff9fe0, clockid=0, mutex=0x55c10bb647f0, cond=0x55c10bb64858) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x55c10bb64858, mutex=mutex@entry=0x55c10bb647f0, abstime=abstime@entry=0x7fb73bff9fe0) at ./nptl/pthread_cond_wait.c:643
#5  0x000055c10a3f2b31 in qemu_cond_timedwait_ts (cond=cond@entry=0x55c10bb64858, mutex=mutex@entry=0x55c10bb647f0, ts=ts@entry=0x7fb73bff9fe0, file=file@entry=0x55c10a64ef78 "../util/thread-pool.c", line=line@entry=90) at ../util/qemu-thread-posix.c:239
#6  0x000055c10a3f36d0 in qemu_cond_timedwait_impl (cond=0x55c10bb64858, mutex=0x55c10bb647f0, ms=<optimized out>, file=0x55c10a64ef78 "../util/thread-pool.c", line=90) at ../util/qemu-thread-posix.c:253
#7  0x000055c10a407f04 in worker_thread (opaque=opaque@entry=0x55c10bb647e0) at ../util/thread-pool.c:90
#8  0x000055c10a3f29a8 in qemu_thread_start (args=0x55c10c16f640) at ../util/qemu-thread-posix.c:541
#9  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7fb7753ff6c0 (LWP 7936)):
#0  0x00007fb88840d15f in __GI___poll (fds=0x7fb74c027420, nfds=4, timeout=939) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fb889be79ae in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007fb889be7cef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007fb88a2e6fa9 in red_worker_main (arg=0x55c10d67fb90) at ../server/red-worker.cpp:1021
#4  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5  0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7fb8852026c0 (LWP 7900)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055c10a3f3b2a in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at ./include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x55c10ad469c8 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:464
#3  0x000055c10a3fd432 in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:278
#4  0x000055c10a3f29a8 in qemu_thread_start (args=0x55c10b89a720) at ../util/qemu-thread-posix.c:541
#5  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7fb75f3bf6c0 (LWP 7938)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55c10d92e67c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x55c10d92e67c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007fb888396efb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c10d92e67c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fb888399558 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c10d92e688, cond=0x55c10d92e650) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x55c10d92e650, mutex=mutex@entry=0x55c10d92e688) at ./nptl/pthread_cond_wait.c:618
#5  0x000055c10a3f34bb in qemu_cond_wait_impl (cond=0x55c10d92e650, mutex=0x55c10d92e688, file=0x55c10a4b7cf4 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:225
#6  0x000055c109e7ff0b in vnc_worker_thread_loop (queue=queue@entry=0x55c10d92e650) at ../ui/vnc-jobs.c:248
#7  0x000055c109e80ba8 in vnc_worker_thread (arg=arg@entry=0x55c10d92e650) at ../ui/vnc-jobs.c:362
#8  0x000055c10a3f29a8 in qemu_thread_start (args=0x55c10c134b90) at ../util/qemu-thread-posix.c:541
#9  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7fb87e7ff6c0 (LWP 7932)):
#0  __GI___ioctl (fd=33, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c10a2596bf in kvm_vcpu_ioctl (cpu=cpu@entry=0x55c10bc86740, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x000055c10a259b95 in kvm_cpu_exec (cpu=cpu@entry=0x55c10bc86740) at ../accel/kvm/kvm-all.c:2991
#3  0x000055c10a25b07d in kvm_vcpu_thread_fn (arg=arg@entry=0x55c10bc86740) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x000055c10a3f29a8 in qemu_thread_start (args=0x55c10bc8f5c0) at ../util/qemu-thread-posix.c:541
#5  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

--Type <RET> for more, q to quit, c to continue without paging--
Thread 5 (Thread 0x7fb87f3ff6c0 (LWP 7931)):
#0  __GI___ioctl (fd=31, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c10a2596bf in kvm_vcpu_ioctl (cpu=cpu@entry=0x55c10bc7cda0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x000055c10a259b95 in kvm_cpu_exec (cpu=cpu@entry=0x55c10bc7cda0) at ../accel/kvm/kvm-all.c:2991
#3  0x000055c10a25b07d in kvm_vcpu_thread_fn (arg=arg@entry=0x55c10bc7cda0) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x000055c10a3f29a8 in qemu_thread_start (args=0x55c10bc85d90) at ../util/qemu-thread-posix.c:541
#5  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7fb87ffff6c0 (LWP 7930)):
#0  __GI___ioctl (fd=29, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c10a2596bf in kvm_vcpu_ioctl (cpu=cpu@entry=0x55c10bc4ba90, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x000055c10a259b95 in kvm_cpu_exec (cpu=cpu@entry=0x55c10bc4ba90) at ../accel/kvm/kvm-all.c:2991
#3  0x000055c10a25b07d in kvm_vcpu_thread_fn (arg=arg@entry=0x55c10bc4ba90) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x000055c10a3f29a8 in qemu_thread_start (args=0x55c10b892560) at ../util/qemu-thread-posix.c:541
#5  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7fb87dbff6c0 (LWP 7933)):
#0  __GI___ioctl (fd=35, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c10a2596bf in kvm_vcpu_ioctl (cpu=cpu@entry=0x55c10bc8ff70, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x000055c10a259b95 in kvm_cpu_exec (cpu=cpu@entry=0x55c10bc8ff70) at ../accel/kvm/kvm-all.c:2991
#3  0x000055c10a25b07d in kvm_vcpu_thread_fn (arg=arg@entry=0x55c10bc8ff70) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x000055c10a3f29a8 in qemu_thread_start (args=0x55c10bc99740) at ../util/qemu-thread-posix.c:541
#5  0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fb88546b500 (LWP 7899)):
#0  __GI___libc_write (nbytes=8, buf=0x55c10a64a7a8 <value>, fd=48) at ../sysdeps/unix/sysv/linux/write.c:26
#1  __GI___libc_write (fd=48, buf=buf@entry=0x55c10a64a7a8 <value>, nbytes=nbytes@entry=8) at ../sysdeps/unix/sysv/linux/write.c:24
#2  0x000055c10a3f0fbc in event_notifier_set (e=0x55c10d938340) at ../util/event_notifier-posix.c:114
#3  0x000055c10a1d585a in virtio_notify_irqfd (vdev=<optimized out>, vq=<optimized out>) at ../hw/virtio/virtio.c:2455
#4  0x000055c109ec839f in virtio_blk_data_plane_notify (s=<optimized out>, vq=<optimized out>) at ../hw/block/dataplane/virtio-blk.c:54
#5  0x000055c10a19bb03 in virtio_blk_req_complete (req=req@entry=0x55c10c774c20, status=status@entry=0 '\000') at ../hw/block/virtio-blk.c:68
#6  0x000055c10a19c12e in virtio_blk_rw_complete (opaque=<optimized out>, ret=0) at ../hw/block/virtio-blk.c:133
#7  0x000055c10a2cca9d in blk_aio_complete (acb=0x55c10bcc7700) at ../block/block-backend.c:1563
#8  blk_aio_complete (acb=0x55c10bcc7700) at ../block/block-backend.c:1560
#9  blk_aio_write_entry (opaque=0x55c10bcc7700) at ../block/block-backend.c:1630
#10 0x000055c10a407cdb in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at ../util/coroutine-ucontext.c:177
#11 0x00007fb8883629c0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x00007fb87fff93e0 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fb777dff6c0 (LWP 7935)):
#0  _int_malloc (av=av@entry=0x7fb758000030, bytes=bytes@entry=240) at ./malloc/malloc.c:4004
#1  0x00007fb8883aa6e2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at ./malloc/malloc.c:3674
#2  0x00007fb889bed6d1 in g_malloc0 () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007fb88a2d50fc in red::simple_ptr_counted<RedDrawable>::operator new (size=240) at ../server/utils.hpp:465
#4  red::make_shared<RedDrawable> () at ../server/utils.hpp:247
#5  red_drawable_new (qxl=0x55c10d35a1d8, slots=slots@entry=0x55c10d491920, group_id=1, addr=72057594054792568, flags=0) at ../server/red-parse-qxl.cpp:1263
#6  0x00007fb88a2e7a2c in red_process_display (worker=worker@entry=0x55c10d4918c0, ring_is_empty=ring_is_empty@entry=0x7fb777df9fd4) at ../server/red-worker.cpp:195
#7  0x00007fb88a2e7cb7 in worker_source_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at ../server/red-worker.cpp:924
#8  0x00007fb889be77a9 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#9  0x00007fb889be7a38 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#10 0x00007fb889be7cef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#11 0x00007fb88a2e6fa9 in red_worker_main (arg=0x55c10d4918c0) at ../server/red-worker.cpp:1021
#12 0x00007fb88839a134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007fb88841a7dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

 

[fiona]

So the segfault seems to happen while the server-side SPICE code is allocating some shared pointer. But it's hard to tell what exactly goes wrong. Can you also share the output of dpkg --list libc6? Are you using virt-viewer or something else as a client? What version?

 

[garbled]

I know that stacktrace isn't mine, but I'm using:

root@jabbah:~# dpkg -l | grep virt-viewer
ii virt-viewer 7.0-2 amd64 Displaying the graphical console of a virtual machine
root@jabbah:~# cat /etc/debian_version
11.3


I have not changed the version of virt-viewer on the client end. It's still the same version I was running before upgrading proxmox from 7 to 8.X

 

[DeeMaas]

Client side

dpkg -l | grep virt-viewer
ii  virt-viewer                             11.0-2                              amd64        Displaying the graphical console of a virtual machine

ii  libc6:amd64    2.36-9+deb12u3 amd64        GNU C Library: Shared libraries

cat /etc/debian_version
12.2

 

[DeeMaas]

server side (proxmox)

# dpkg --list libc6
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Architecture Description
+++-==============-==============-============-=================================
ii  libc6:amd64    2.36-9+deb12u4 amd64        GNU C Library: Shared libraries

 

[DeeMaas]

Do you have any news for us?

 

[fiona]

Unfortunately not. I was not yet able to reproduce the issue. It might be a heap corruption and for those, a trace like you provided just points to the effect unfortunately, not the root cause. Additional traces from other crashes could help to narrow it down. I also checked for any fixes upstream, but nothing caught my eye.

 

[DeeMaas]

When VMs crashed I found that system storage latancy were over 7000 ms. Also every VM had 4Gb memory.
I delete storage replica and latency down below 20 ms. Also I increased memory to 6Gb. There are no crashes after this steps. I'm continue monitoring the situation.

 

[DeeMaas]

New crash

# coredumpctl gdb -1
           PID: 2403438 (kvm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Wed 2024-03-06 12:30:37 MSK (12min ago)
  Command Line: /usr/bin/kvm -id 104 -name VM2,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/104.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/104.pid -daemonize -smbios type=1,uuid=f60adc36-313a-400f-be51-7eab74a82a86 -smp 6,sockets=1,cores=6,maxcpus=6 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/104.vnc,password=on -cpu host,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt -m 16384 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device vmgenid,guid=003259a7-0163-48a7-ad41-fc296d9750bf -device qemu-xhci,p2=15,p3=15,id=xhci,bus=pci.1,addr=0x1b -chardev spicevmc,id=usbredirchardev0,name=usbredir -device usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=xhci.0,port=1 -chardev spicevmc,id=usbredirchardev1,name=usbredir -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=xhci.0,port=2 -device ich9-intel-hda,id=audiodev0,bus=pci.2,addr=0xc -device hda-micro,id=audiodev0-codec0,bus=audiodev0.0,cad=0,audiodev=spice-backend0 -device hda-duplex,id=audiodev0-codec1,bus=audiodev0.0,cad=1,audiodev=spice-backend0 -audiodev spice,id=spice-backend0 -device qxl-vga,id=vga,bus=pcie.0,addr=0x1 -chardev socket,path=/var/run/qemu-server/104.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device qxl,id=vga1,ram_size=67108864,vram_size=33554432,bus=pci.0,addr=0x18 -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice tls-port=61004,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -iscsi initiator-name=iqn.1993-08.org.debian:01:fd45f7ff9e93 -drive file=/dev/vdi/vm-104-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:2F:24:B3,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256 -rtc driftfix=slew,base=localtime -machine hpet=off,type=pc-q35-8.1+pve0 -global kvm-pit.lost_tick_policy=discard
    Executable: /usr/bin/qemu-system-x86_64
 Control Group: /qemu.slice/104.scope
          Unit: 104.scope
         Slice: qemu.slice
       Boot ID: a3c583db0f604f80a58e77c7c27b7e28
    Machine ID: ac51467659484c5ab485c87c559ab459
      Hostname: vdi3
       Storage: /var/lib/systemd/coredump/core.kvm.0.a3c583db0f604f80a58e77c7c27b7e28.2403438.1709717437000000.zst (present)
  Size on Disk: 6.3G
       Message: Process 2403438 (kvm) of user 0 dumped core.

                Module libsystemd.so.0 from deb systemd-252.22-1~deb12u1.amd64
                Module libudev.so.1 from deb systemd-252.22-1~deb12u1.amd64
                Stack trace of thread 2403479:
                #0  0x00007fb9fe6ccfcd n/a (libc.so.6 + 0x94fcd)
                #1  0x00007fb9fe6cfdcd n/a (libc.so.6 + 0x97dcd)
                #2  0x00007fb9fe6d16e2 __libc_calloc (libc.so.6 + 0x996e2)
                #3  0x00007fb9fff146d1 g_malloc0 (libglib-2.0.so.0 + 0x5a6d1)
                #4  0x00005628f096fc90 n/a (qemu-system-x86_64 + 0x331c90)
                #5  0x00005628f0be782b n/a (qemu-system-x86_64 + 0x5a982b)
                #6  0x00005628f0be658b n/a (qemu-system-x86_64 + 0x5a858b)
                #7  0x00007fba0060e1bc n/a (libspice-server.so.1 + 0x521bc)
                #8  0x00007fba0060ecac n/a (libspice-server.so.1 + 0x52cac)
                #9  0x00007fb9fff0e7a9 g_main_context_dispatch (libglib-2.0.so.0 + 0x547a9)
                #10 0x00007fb9fff0ea38 n/a (libglib-2.0.so.0 + 0x54a38)
                #11 0x00007fb9fff0ecef g_main_loop_run (libglib-2.0.so.0 + 0x54cef)
                #12 0x00007fba0060dfa9 n/a (libspice-server.so.1 + 0x51fa9)
                #13 0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #14 0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403472:
                #0  0x00007fb9fe735c5b ioctl (libc.so.6 + 0xfdc5b)
                #1  0x00005628f0d696bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x00005628f0d69b95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x00005628f0d6b07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #6  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403439:
                #0  0x00007fb9fe739719 syscall (libc.so.6 + 0x101719)
                #1  0x00005628f0f03b2a n/a (qemu-system-x86_64 + 0x8c5b2a)
                #2  0x00005628f0f0d432 n/a (qemu-system-x86_64 + 0x8cf432)
                #3  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #4  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #5  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403473:
                #0  0x00007fb9fe735c5b ioctl (libc.so.6 + 0xfdc5b)
                #1  0x00005628f0d696bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x00005628f0d69b95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x00005628f0d6b07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #6  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403475:
                #0  0x00007fb9fe735c5b ioctl (libc.so.6 + 0xfdc5b)
                #1  0x00005628f0d696bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x00005628f0d69b95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x00005628f0d6b07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #6  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403474:
                #0  0x00007fb9fe735c5b ioctl (libc.so.6 + 0xfdc5b)
                #1  0x00005628f0d696bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x00005628f0d69b95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x00005628f0d6b07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #6  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403477:
                #0  0x00007fb9fe735c5b ioctl (libc.so.6 + 0xfdc5b)
                #1  0x00005628f0d696bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x00005628f0d69b95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x00005628f0d6b07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #6  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403480:
                #0  0x00007fb9fe73415f __poll (libc.so.6 + 0xfc15f)
                #1  0x00007fb9fff0e9ae n/a (libglib-2.0.so.0 + 0x549ae)
                #2  0x00007fb9fff0ecef g_main_loop_run (libglib-2.0.so.0 + 0x54cef)
                #3  0x00007fba0060dfa9 n/a (libspice-server.so.1 + 0x51fa9)
                #4  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #5  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403476:
                #0  0x00007fb9fe735c5b ioctl (libc.so.6 + 0xfdc5b)
                #1  0x00005628f0d696bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x00005628f0d69b95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x00005628f0d6b07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #6  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 2403438:
                #0  0x00007fb9fe734256 ppoll (libc.so.6 + 0xfc256)
                #1  0x00005628f0f18c2e n/a (qemu-system-x86_64 + 0x8dac2e)
                #2  0x00005628f0f1651e n/a (qemu-system-x86_64 + 0x8d851e)
                #3  0x00005628f0b73a87 n/a (qemu-system-x86_64 + 0x535a87)
                #4  0x00005628f0d73f36 n/a (qemu-system-x86_64 + 0x735f36)
                #5  0x00007fb9fe65f24a n/a (libc.so.6 + 0x2724a)
                #6  0x00007fb9fe65f305 __libc_start_main (libc.so.6 + 0x27305)
                #7  0x00005628f0966081 n/a (qemu-system-x86_64 + 0x328081)

                Stack trace of thread 2403482:
                #0  0x00007fb9fe6bde96 n/a (libc.so.6 + 0x85e96)
                #1  0x00007fb9fe6c0558 pthread_cond_wait (libc.so.6 + 0x88558)
                #2  0x00005628f0f034bb n/a (qemu-system-x86_64 + 0x8c54bb)
                #3  0x00005628f098ff0b n/a (qemu-system-x86_64 + 0x351f0b)
                #4  0x00005628f0990ba8 n/a (qemu-system-x86_64 + 0x352ba8)
                #5  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #6  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #7  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)

                Stack trace of thread 847242:
                #0  0x00007fb9fe6bde96 n/a (libc.so.6 + 0x85e96)
                #1  0x00007fb9fe6c083c pthread_cond_timedwait (libc.so.6 + 0x8883c)
                #2  0x00005628f0f02b31 n/a (qemu-system-x86_64 + 0x8c4b31)
                #3  0x00005628f0f036d0 n/a (qemu-system-x86_64 + 0x8c56d0)
                #4  0x00005628f0f17f04 n/a (qemu-system-x86_64 + 0x8d9f04)
                #5  0x00005628f0f029a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #6  0x00007fb9fe6c1134 n/a (libc.so.6 + 0x89134)
                #7  0x00007fb9fe7417dc n/a (libc.so.6 + 0x1097dc)
                ELF object binary architecture: AMD x86-64

GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/qemu-system-x86_64...
(No debugging symbols found in /usr/bin/qemu-system-x86_64)

warning: Can't open file anon_inode:kvm-vcpu:5 which was expanded to anon_inode:kvm-vcpu:5 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:4 which was expanded to anon_inode:kvm-vcpu:4 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:3 which was expanded to anon_inode:kvm-vcpu:3 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:2 which was expanded to anon_inode:kvm-vcpu:2 during file-backed mapping note processing

warning: Can't open file /[aio] (deleted) during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:1 which was expanded to anon_inode:kvm-vcpu:1 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:0 which was expanded to anon_inode:kvm-vcpu:0 during file-backed mapping note processing

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 2403479]
[New LWP 2403472]
[New LWP 2403439]
[New LWP 2403473]
[New LWP 2403475]
[New LWP 2403474]
[New LWP 2403477]
[New LWP 2403480]
[New LWP 2403476]
[New LWP 2403438]
[New LWP 2403482]
[New LWP 847242]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/kvm -id 104 -name VM2,debug-threads=on -no-shutdown -chardev'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb9fe6ccfcd in unlink_chunk (p=p@entry=0x7fb5d8021ed0, av=0x7fb5d8000030) at ./malloc/malloc.c:1628
1628    ./malloc/malloc.c: No such file or directory.
[Current thread is 1 (Thread 0x7fb5e91ff6c0 (LWP 2403479))]

 

[DeeMaas]

(gdb) thread apply all backtrace



Thread 12 (Thread 0x7fb57bfff6c0 (LWP 847242)):

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fb57bff9fe0, op=393, expected=0, futex_word=0x5628f23a25c0) at ./nptl/futex-internal.c:57

#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5628f23a25c0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fb57bff9fe0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87

#2  0x00007fb9fe6bdefb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5628f23a25c0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fb57bff9fe0, private=private@entry=0) at ./nptl/futex-internal.c:139

#3  0x00007fb9fe6c083c in __pthread_cond_wait_common (abstime=0x7fb57bff9fe0, clockid=0, mutex=0x5628f23a2530, cond=0x5628f23a2598) at ./nptl/pthread_cond_wait.c:503

#4  ___pthread_cond_timedwait64 (cond=0x5628f23a2598, mutex=0x5628f23a2530, abstime=0x7fb57bff9fe0) at ./nptl/pthread_cond_wait.c:643

#5  0x00005628f0f02b31 in ?? ()

#6  0x00005628f0f036d0 in ?? ()

#7  0x00005628f0f17f04 in ?? ()

#8  0x00005628f0f029a8 in ?? ()

#9  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#10 0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 11 (Thread 0x7fb5bffff6c0 (LWP 2403482)):

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5628f256062c) at ./nptl/futex-internal.c:57

#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5628f256062c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87

#2  0x00007fb9fe6bdefb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5628f256062c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139

#3  0x00007fb9fe6c0558 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5628f2560638, cond=0x5628f2560600) at ./nptl/pthread_cond_wait.c:503

#4  ___pthread_cond_wait (cond=0x5628f2560600, mutex=0x5628f2560638) at ./nptl/pthread_cond_wait.c:618

#5  0x00005628f0f034bb in ?? ()

#6  0x00005628f098ff0b in ?? ()

#7  0x00005628f0990ba8 in ?? ()

#8  0x00005628f0f029a8 in ?? ()

#9  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#10 0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 10 (Thread 0x7fb9fba0f500 (LWP 2403438)):

#0  0x00007fb9fe734256 in __ppoll (fds=0x5628f2abc000, nfds=153, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42

#1  0x00005628f0f18c2e in ?? ()

#2  0x00005628f0f1651e in ?? ()

#3  0x00005628f0b73a87 in ?? ()

#4  0x00005628f0d73f36 in ?? ()

#5  0x00007fb9fe65f24a in __libc_start_call_main (main=main@entry=0x5628f0964460 <main>, argc=argc@entry=86, argv=argv@entry=0x7fffeb07cb68) at ../sysdeps/nptl/libc_start_call_main.h:58

#6  0x00007fb9fe65f305 in __libc_start_main_impl (main=0x5628f0964460 <main>, argc=86, argv=0x7fffeb07cb68, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffeb07cb58) at ../csu/libc-start.c:360

#7  0x00005628f0966081 in ?? ()



Thread 9 (Thread 0x7fb5eb1ff6c0 (LWP 2403476)):

#0  __GI___ioctl (fd=34, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36

#1  0x00005628f0d696bf in ?? ()

#2  0x00005628f0d69b95 in ?? ()

#3  0x00005628f0d6b07d in ?? ()

#4  0x00005628f0f029a8 in ?? ()

#5  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#6  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 8 (Thread 0x7fb5c57ff6c0 (LWP 2403480)):

#0  0x00007fb9fe73415f in __GI___poll (fds=0x7fb5c0027420, nfds=4, timeout=2374) at ../sysdeps/unix/sysv/linux/poll.c:29

#1  0x00007fb9fff0e9ae in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0

#2  0x00007fb9fff0ecef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0

#3  0x00007fba0060dfa9 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1

#4  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#5  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



--Type <RET> for more, q to quit, c to continue without paging--

Thread 7 (Thread 0x7fb5ea5ff6c0 (LWP 2403477)):

#0  __GI___ioctl (fd=36, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36

#1  0x00005628f0d696bf in ?? ()

#2  0x00005628f0d69b95 in ?? ()

#3  0x00005628f0d6b07d in ?? ()

#4  0x00005628f0f029a8 in ?? ()

#5  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#6  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 6 (Thread 0x7fb9f8dff6c0 (LWP 2403474)):

#0  __GI___ioctl (fd=30, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36

#1  0x00005628f0d696bf in ?? ()

#2  0x00005628f0d69b95 in ?? ()

#3  0x00005628f0d6b07d in ?? ()

#4  0x00005628f0f029a8 in ?? ()

#5  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#6  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 5 (Thread 0x7fb5ebdff6c0 (LWP 2403475)):

#0  __GI___ioctl (fd=32, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36

#1  0x00005628f0d696bf in ?? ()

#2  0x00005628f0d69b95 in ?? ()

#3  0x00005628f0d6b07d in ?? ()

#4  0x00005628f0f029a8 in ?? ()

#5  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#6  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 4 (Thread 0x7fb9f99ff6c0 (LWP 2403473)):

#0  __GI___ioctl (fd=28, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36

#1  0x00005628f0d696bf in ?? ()

#2  0x00005628f0d69b95 in ?? ()

#3  0x00005628f0d6b07d in ?? ()

#4  0x00005628f0f029a8 in ?? ()

#5  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#6  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 3 (Thread 0x7fb9fb5276c0 (LWP 2403439)):

#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

#1  0x00005628f0f03b2a in ?? ()

#2  0x00005628f0f0d432 in ?? ()

#3  0x00005628f0f029a8 in ?? ()

#4  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#5  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 2 (Thread 0x7fb9faa636c0 (LWP 2403472)):

#0  __GI___ioctl (fd=26, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36

#1  0x00005628f0d696bf in ?? ()

#2  0x00005628f0d69b95 in ?? ()

#3  0x00005628f0d6b07d in ?? ()

#4  0x00005628f0f029a8 in ?? ()

#5  0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#6  0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



Thread 1 (Thread 0x7fb5e91ff6c0 (LWP 2403479)):

#0  0x00007fb9fe6ccfcd in unlink_chunk (p=p@entry=0x7fb5d8021ed0, av=0x7fb5d8000030) at ./malloc/malloc.c:1628

#1  0x00007fb9fe6cfdcd in _int_malloc (av=av@entry=0x7fb5d8000030, bytes=bytes@entry=6412) at ./malloc/malloc.c:4303

#2  0x00007fb9fe6d16e2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at ./malloc/malloc.c:3674

#3  0x00007fb9fff146d1 in g_malloc0 () from /lib/x86_64-linux-gnu/libglib-2.0.so.0

#4  0x00005628f096fc90 in ?? ()

--Type <RET> for more, q to quit, c to continue without paging--

#5  0x00005628f0be782b in ?? ()

#6  0x00005628f0be658b in ?? ()

#7  0x00007fba0060e1bc in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1

#8  0x00007fba0060ecac in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1

#9  0x00007fb9fff0e7a9 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0

#10 0x00007fb9fff0ea38 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0

#11 0x00007fb9fff0ecef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0

#12 0x00007fba0060dfa9 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1

#13 0x00007fb9fe6c1134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#14 0x00007fb9fe7417dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

 

[DeeMaas]

New crash

# coredumpctl gdb -1
           PID: 1801697 (kvm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Mon 2024-03-11 10:18:55 MSK (3min 2s ago)
  Command Line: /usr/bin/kvm -id 123 -name VM3,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/123.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/123.pid -daemonize -smbios type=1,uuid=40e0b4e8-ce32-497b-b6dd-9485c8a32ae4 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/123.vnc,password=on -cpu host,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt -m 8192 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device vmgenid,guid=a528eabd-0046-4946-a412-0ecf9457e684 -device qemu-xhci,p2=15,p3=15,id=xhci,bus=pci.1,addr=0x1b -chardev spicevmc,id=usbredirchardev0,name=usbredir -device usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=xhci.0,port=1 -chardev spicevmc,id=usbredirchardev1,name=usbredir -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=xhci.0,port=2 -device ich9-intel-hda,id=audiodev0,bus=pci.2,addr=0xc -device hda-micro,id=audiodev0-codec0,bus=audiodev0.0,cad=0,audiodev=spice-backend0 -device hda-duplex,id=audiodev0-codec1,bus=audiodev0.0,cad=1,audiodev=spice-backend0 -audiodev spice,id=spice-backend0 -device qxl-vga,id=vga,bus=pcie.0,addr=0x1 -chardev socket,path=/var/run/qemu-server/123.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device qxl,id=vga1,ram_size=67108864,vram_size=33554432,bus=pci.0,addr=0x18 -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice tls-port=61004,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -iscsi initiator-name=iqn.1993-08.org.debian:01:48c37d4b1ce2 -drive file=/dev/vdi/vm-123-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap123i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:EB:B1:5F,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256 -rtc driftfix=slew,base=localtime -machine hpet=off,type=pc-q35-8.1+pve0 -global kvm-pit.lost_tick_policy=discard
    Executable: /usr/bin/qemu-system-x86_64
 Control Group: /qemu.slice/123.scope
          Unit: 123.scope
         Slice: qemu.slice
       Boot ID: bc4893b1093a43dfa8b22db90af69e28
    Machine ID: 13c00927de604a6c917b600921106592
      Hostname: vdi2
       Storage: /var/lib/systemd/coredump/core.kvm.0.bc4893b1093a43dfa8b22db90af69e28.1801697.1710141535000000.zst (present)
  Size on Disk: 2.6G
       Message: Process 1801697 (kvm) of user 0 dumped core.

                Module libsystemd.so.0 from deb systemd-252.22-1~deb12u1.amd64
                Module libudev.so.1 from deb systemd-252.22-1~deb12u1.amd64
                Stack trace of thread 1801732:
                #0  0x00007f6566db6e2c __pthread_kill_implementation (libc.so.6 + 0x8ae2c)
                #1  0x00007f6566d67fb2 __GI_raise (libc.so.6 + 0x3bfb2)
                #2  0x00007f6566d52472 __GI_abort (libc.so.6 + 0x26472)
                #3  0x00007f6566dab430 __libc_message (libc.so.6 + 0x7f430)
                #4  0x00007f6566dc07aa malloc_printerr (libc.so.6 + 0x947aa)
                #5  0x00007f6566dc289c _int_free (libc.so.6 + 0x9689c)
                #6  0x00007f6566dc4e8f __GI___libc_free (libc.so.6 + 0x98e8f)
                #7  0x00007f6568cf1089 n/a (libspice-server.so.1 + 0x41089)
                #8  0x00007f6568cc8338 n/a (libspice-server.so.1 + 0x18338)
                #9  0x00007f6568cec5e3 n/a (libspice-server.so.1 + 0x3c5e3)
                #10 0x00007f6568cec700 n/a (libspice-server.so.1 + 0x3c700)
                #11 0x00007f6568cd679d n/a (libspice-server.so.1 + 0x2679d)
                #12 0x00007f656860267f g_main_context_dispatch (libglib-2.0.so.0 + 0x5467f)
                #13 0x00007f6568602a38 n/a (libglib-2.0.so.0 + 0x54a38)
                #14 0x00007f6568602cef g_main_loop_run (libglib-2.0.so.0 + 0x54cef)
                #15 0x00007f6568d01fa9 n/a (libspice-server.so.1 + 0x51fa9)
                #16 0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #17 0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1801729:
                #0  0x00007f6566e29c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x000055c2fdcdd6bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x000055c2fdcddb95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x000055c2fdcdf07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x000055c2fde769a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1801727:
                #0  0x00007f6566e29c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x000055c2fdcdd6bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x000055c2fdcddb95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x000055c2fdcdf07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x000055c2fde769a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1801728:
                #0  0x00007f6566e29c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x000055c2fdcdd6bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x000055c2fdcddb95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x000055c2fdcdf07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x000055c2fde769a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1801730:
                #0  0x00007f6566e29c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x000055c2fdcdd6bf n/a (qemu-system-x86_64 + 0x72b6bf)
                #2  0x000055c2fdcddb95 n/a (qemu-system-x86_64 + 0x72bb95)
                #3  0x000055c2fdcdf07d n/a (qemu-system-x86_64 + 0x72d07d)
                #4  0x000055c2fde769a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #5  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1801735:
                #0  0x00007f6566db1e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
                #1  0x00007f6566db4558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
                #2  0x000055c2fde774bb n/a (qemu-system-x86_64 + 0x8c54bb)
                #3  0x000055c2fd903f0b n/a (qemu-system-x86_64 + 0x351f0b)
                #4  0x000055c2fd904ba8 n/a (qemu-system-x86_64 + 0x352ba8)
                #5  0x000055c2fde769a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #6  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #7  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1801697:
                #0  0x00007f6566e28256 __ppoll (libc.so.6 + 0xfc256)
                #1  0x000055c2fde8cc2e n/a (qemu-system-x86_64 + 0x8dac2e)
                #2  0x000055c2fde8a51e n/a (qemu-system-x86_64 + 0x8d851e)
                #3  0x000055c2fdae7a87 n/a (qemu-system-x86_64 + 0x535a87)
                #4  0x000055c2fdce7f36 n/a (qemu-system-x86_64 + 0x735f36)
                #5  0x00007f6566d5324a __libc_start_call_main (libc.so.6 + 0x2724a)
                #6  0x00007f6566d53305 __libc_start_main_impl (libc.so.6 + 0x27305)
                #7  0x000055c2fd8da081 n/a (qemu-system-x86_64 + 0x328081)

                Stack trace of thread 1801733:
                #0  0x00007f6566e2815f __GI___poll (libc.so.6 + 0xfc15f)
                #1  0x00007f65686029ae n/a (libglib-2.0.so.0 + 0x549ae)
                #2  0x00007f6568602cef g_main_loop_run (libglib-2.0.so.0 + 0x54cef)
                #3  0x00007f6568d01fa9 n/a (libspice-server.so.1 + 0x51fa9)
                #4  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #5  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1801698:
                #0  0x00007f6566e2d719 syscall (libc.so.6 + 0x101719)
                #1  0x000055c2fde77b2a n/a (qemu-system-x86_64 + 0x8c5b2a)
                #2  0x000055c2fde81432 n/a (qemu-system-x86_64 + 0x8cf432)
                #3  0x000055c2fde769a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #4  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #5  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 1233149:
                #0  0x00007f6566db1e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
                #1  0x00007f6566db483c __pthread_cond_wait_common (libc.so.6 + 0x8883c)
                #2  0x000055c2fde76b31 n/a (qemu-system-x86_64 + 0x8c4b31)
                #3  0x000055c2fde776d0 n/a (qemu-system-x86_64 + 0x8c56d0)
                #4  0x000055c2fde8bf04 n/a (qemu-system-x86_64 + 0x8d9f04)
                #5  0x000055c2fde769a8 n/a (qemu-system-x86_64 + 0x8c49a8)
                #6  0x00007f6566db5134 start_thread (libc.so.6 + 0x89134)
                #7  0x00007f6566e357dc __clone3 (libc.so.6 + 0x1097dc)
                ELF object binary architecture: AMD x86-64

GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/qemu-system-x86_64...
(No debugging symbols found in /usr/bin/qemu-system-x86_64)

warning: Can't open file anon_inode:kvm-vcpu:3 which was expanded to anon_inode:kvm-vcpu:3 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:2 which was expanded to anon_inode:kvm-vcpu:2 during file-backed mapping note processing

warning: Can't open file /[aio] (deleted) during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:1 which was expanded to anon_inode:kvm-vcpu:1 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:0 which was expanded to anon_inode:kvm-vcpu:0 during file-backed mapping note processing

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 1801732]
[New LWP 1801729]
[New LWP 1801727]
[New LWP 1801728]
[New LWP 1801730]
[New LWP 1801735]
[New LWP 1801697]
[New LWP 1801733]
[New LWP 1801698]
[New LWP 1233149]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/kvm -id 123 -name VM3,debug-threads=on -no-shutdown -chardev'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f63531ff6c0 (LWP 1801732))]

 

[DeeMaas]

(gdb) thread apply all backtrace

Thread 10 (Thread 0x7f63371fe6c0 (LWP 1233149)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f63371f8fe0, op=393, expected=0, futex_word=0x55c3000c15c4) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x55c3000c15c4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f63371f8fe0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f6566db1efb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c3000c15c4, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f63371f8fe0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f6566db483c in __pthread_cond_wait_common (abstime=0x7f63371f8fe0, clockid=0, mutex=0x55c3000c1530, cond=0x55c3000c1598) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x55c3000c1598, mutex=0x55c3000c1530, abstime=0x7f63371f8fe0) at ./nptl/pthread_cond_wait.c:643
#5  0x000055c2fde76b31 in ?? ()
#6  0x000055c2fde776d0 in ?? ()
#7  0x000055c2fde8bf04 in ?? ()
#8  0x000055c2fde769a8 in ?? ()
#9  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7f6563c1c6c0 (LWP 1801698)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055c2fde77b2a in ?? ()
#2  0x000055c2fde81432 in ?? ()
#3  0x000055c2fde769a8 in ?? ()
#4  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5  0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7f63379ff6c0 (LWP 1801733)):
#0  0x00007f6566e2815f in __GI___poll (fds=0x7f6330027420, nfds=4, timeout=8879) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f65686029ae in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f6568602cef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f6568d01fa9 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#4  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5  0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7f6563e80500 (LWP 1801697)):
#0  0x00007f6566e28256 in __ppoll (fds=0x55c3005efa00, nfds=151, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#1  0x000055c2fde8cc2e in ?? ()
#2  0x000055c2fde8a51e in ?? ()
#3  0x000055c2fdae7a87 in ?? ()
#4  0x000055c2fdce7f36 in ?? ()
#5  0x00007f6566d5324a in __libc_start_call_main (main=main@entry=0x55c2fd8d8460 <main>, argc=argc@entry=86, argv=argv@entry=0x7ffeb05f5318) at ../sysdeps/nptl/libc_start_call_main.h:58
#6  0x00007f6566d53305 in __libc_start_main_impl (main=0x55c2fd8d8460 <main>, argc=86, argv=0x7ffeb05f5318, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffeb05f5308) at ../csu/libc-start.c:360
#7  0x000055c2fd8da081 in ?? ()

Thread 6 (Thread 0x7f6335fbf6c0 (LWP 1801735)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55c300403adc) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x55c300403adc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f6566db1efb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55c300403adc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f6566db4558 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55c300403ae8, cond=0x55c300403ab0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x55c300403ab0, mutex=0x55c300403ae8) at ./nptl/pthread_cond_wait.c:618
#5  0x000055c2fde774bb in ?? ()
#6  0x000055c2fd903f0b in ?? ()
#7  0x000055c2fd904ba8 in ?? ()
#8  0x000055c2fde769a8 in ?? ()
#9  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f65609ff6c0 (LWP 1801730)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  __GI___ioctl (fd=32, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c2fdcdd6bf in ?? ()
#2  0x000055c2fdcddb95 in ?? ()
#3  0x000055c2fdcdf07d in ?? ()
#4  0x000055c2fde769a8 in ?? ()
#5  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f65621ff6c0 (LWP 1801728)):
#0  __GI___ioctl (fd=28, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c2fdcdd6bf in ?? ()
#2  0x000055c2fdcddb95 in ?? ()
#3  0x000055c2fdcdf07d in ?? ()
#4  0x000055c2fde769a8 in ?? ()
#5  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f65631586c0 (LWP 1801727)):
#0  __GI___ioctl (fd=26, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c2fdcdd6bf in ?? ()
#2  0x000055c2fdcddb95 in ?? ()
#3  0x000055c2fdcdf07d in ?? ()
#4  0x000055c2fde769a8 in ?? ()
#5  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f65615ff6c0 (LWP 1801729)):
#0  __GI___ioctl (fd=30, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000055c2fdcdd6bf in ?? ()
#2  0x000055c2fdcddb95 in ?? ()
#3  0x000055c2fdcdf07d in ?? ()
#4  0x000055c2fde769a8 in ?? ()
#5  0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f63531ff6c0 (LWP 1801732)):
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007f6566db6e8f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007f6566d67fb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f6566d52472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007f6566dab430 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f6566ec5459 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007f6566dc07aa in malloc_printerr (str=str@entry=0x7f6566ec81e0 "free(): corrupted unsorted chunks") at ./malloc/malloc.c:5660
#6  0x00007f6566dc289c in _int_free (av=0x7f6348000030, p=0x7f634808e5b0, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4626
#7  0x00007f6566dc4e8f in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385
#8  0x00007f6568cf1089 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#9  0x00007f6568cc8338 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#10 0x00007f6568cec5e3 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#11 0x00007f6568cec700 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#12 0x00007f6568cd679d in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#13 0x00007f656860267f in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007f6568602a38 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#15 0x00007f6568602cef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x00007f6568d01fa9 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#17 0x00007f6566db5134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#18 0x00007f6566e357dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

 

[DeeMaas]

Maybe this help

#4  0x00007f6566dab430 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f6566ec5459 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007f6566dc07aa in malloc_printerr (str=str@entry=0x7f6566ec81e0 "free(): corrupted unsorted chunks") at ./malloc/malloc.c:5660
#6  0x00007f6566dc289c in _int_free (av=0x7f6348000030, p=0x7f634808e5b0, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4626
#7  0x00007f6566dc4e8f in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385

 

[DeeMaas]

 

[fiona]

Unfortunately, it seems like you didn't have the debug symbols for libspice-server1 (and neither for pve-qemu-kvm) available anymore for the latest two traces. So there's only question marks there.

Hmm, it seems like starting at least a few minutes earlier the guest-ping command was failing. Can you tell what was happening in the guest around that time, what was the load on the system as a whole?

 

[garbled]

On mine, which has the same symptoms as above (need to reinstall the core dump capture), the load is pretty minimal on the host. Maybe 4-5% cpu, dual-6140 server. As for what was happening on the guest, I was just scrolling up and down in a web page on firefox (not a complex one, pretty simple page actually).

The one thing I have noticed, is that my old system, was an X5570, whereas, I started noticing this problem when I migrated the VM to a faster host. All other VM's are perfectly stable on same host, or all of my hosts really.

 

[DeeMaas]

# coredumpctl gdb -1
           PID: 668339 (kvm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2024-04-01 10:40:25 MSK (6h ago)
  Command Line: /usr/bin/kvm -id 123 -name VM,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/123.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/123.pid -daemonize -smbios type=1,uuid=40e0b4e8-ce32-497b-b6dd-9485c8a32ae4 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/123.vnc,password=on -cpu host,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt -m 8192 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device vmgenid,guid=a528eabd-0046-4946-a412-0ecf9457e684 -device qemu-xhci,p2=15,p3=15,id=xhci,bus=pci.1,addr=0x1b -chardev spicevmc,id=usbredirchardev0,name=usbredir -device usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=xhci.0,port=1 -chardev spicevmc,id=usbredirchardev1,name=usbredir -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=xhci.0,port=2 -device ich9-intel-hda,id=audiodev0,bus=pci.2,addr=0xc -device hda-micro,id=audiodev0-codec0,bus=audiodev0.0,cad=0,audiodev=spice-backend0 -device hda-duplex,id=audiodev0-codec1,bus=audiodev0.0,cad=1,audiodev=spice-backend0 -audiodev spice,id=spice-backend0 -device qxl-vga,id=vga,bus=pcie.0,addr=0x1 -chardev socket,path=/var/run/qemu-server/123.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device qxl,id=vga1,ram_size=67108864,vram_size=33554432,bus=pci.0,addr=0x18 -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice tls-port=61004,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -iscsi initiator-name=iqn.1993-08.org.debian:01:48c37d4b1ce2 -drive file=/dev/vdi/vm-123-disk-0,if=none,id=drive-virtio0,throttling.bps-read=209715200,throttling.bps-write=209715200,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap123i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:EB:B1:5F,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256 -rtc driftfix=slew,base=localtime -machine hpet=off,type=pc-q35-8.1+pve0 -global kvm-pit.lost_tick_policy=discard
    Executable: /usr/bin/qemu-system-x86_64
 Control Group: /qemu.slice/123.scope
          Unit: 123.scope
         Slice: qemu.slice
       Boot ID: bc4893b1093a43dfa8b22db90af69e28
    Machine ID: 13c00927de604a6c917b600921106592
      Hostname: vdi2
       Storage: /var/lib/systemd/coredump/core.kvm.0.bc4893b1093a43dfa8b22db90af69e28.668339.1711957225000000.zst (present)
  Size on Disk: 3.0G
       Message: Process 668339 (kvm) of user 0 dumped core.

                Module libsystemd.so.0 from deb systemd-252.22-1~deb12u1.amd64
                Module libudev.so.1 from deb systemd-252.22-1~deb12u1.amd64
                Stack trace of thread 668375:
                #0  0x00007fa6faa3cfcd unlink_chunk (libc.so.6 + 0x94fcd)
                #1  0x00007fa6faa3ff4d _int_malloc (libc.so.6 + 0x97f4d)
                #2  0x00007fa6faa416e2 __libc_calloc (libc.so.6 + 0x996e2)
                #3  0x00007fa6fc2846d1 g_malloc0 (libglib-2.0.so.0 + 0x5a6d1)
                #4  0x0000558bee8abcb0 n/a (qemu-system-x86_64 + 0x331cb0)
                #5  0x0000558beeb2384b n/a (qemu-system-x86_64 + 0x5a984b)
                #6  0x0000558beeb225ab n/a (qemu-system-x86_64 + 0x5a85ab)
                #7  0x00007fa6fc97e1bc n/a (libspice-server.so.1 + 0x521bc)
                #8  0x00007fa6fc97ecac n/a (libspice-server.so.1 + 0x52cac)
                #9  0x00007fa6fc27e7a9 g_main_context_dispatch (libglib-2.0.so.0 + 0x547a9)
                #10 0x00007fa6fc27ea38 n/a (libglib-2.0.so.0 + 0x54a38)
                #11 0x00007fa6fc27ecef g_main_loop_run (libglib-2.0.so.0 + 0x54cef)
                #12 0x00007fa6fc97dfa9 n/a (libspice-server.so.1 + 0x51fa9)
                #13 0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #14 0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 668370:
                #0  0x00007fa6faaa5c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x0000558beeca56cf n/a (qemu-system-x86_64 + 0x72b6cf)
                #2  0x0000558beeca5ba5 n/a (qemu-system-x86_64 + 0x72bba5)
                #3  0x0000558beeca708d n/a (qemu-system-x86_64 + 0x72d08d)
                #4  0x0000558beee3eb78 n/a (qemu-system-x86_64 + 0x8c4b78)
                #5  0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 668371:
                #0  0x00007fa6faaa5c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x0000558beeca56cf n/a (qemu-system-x86_64 + 0x72b6cf)
                #2  0x0000558beeca5ba5 n/a (qemu-system-x86_64 + 0x72bba5)
                #3  0x0000558beeca708d n/a (qemu-system-x86_64 + 0x72d08d)
                #4  0x0000558beee3eb78 n/a (qemu-system-x86_64 + 0x8c4b78)
                #5  0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 668376:
                #0  0x00007fa6fa950350 n/a (libssl.so.3 + 0x59350)
                #1  0x00007fa6fa94ad80 n/a (libssl.so.3 + 0x53d80)
                #2  0x00007fa6fa94b267 n/a (libssl.so.3 + 0x54267)
                #3  0x00007fa6fa92ccc3 SSL_write (libssl.so.3 + 0x35cc3)
                #4  0x00007fa6fc97bfb0 n/a (libspice-server.so.1 + 0x4ffb0)
                #5  0x00007fa6fc97c92c n/a (libspice-server.so.1 + 0x5092c)
                #6  0x00007fa6fc9681d3 n/a (libspice-server.so.1 + 0x3c1d3)
                #7  0x00007fa6fc968347 n/a (libspice-server.so.1 + 0x3c347)
                #8  0x00007fa6fc9685d3 n/a (libspice-server.so.1 + 0x3c5d3)
                #9  0x00007fa6fc968700 n/a (libspice-server.so.1 + 0x3c700)
                #10 0x00007fa6fc95279d n/a (libspice-server.so.1 + 0x2679d)
                #11 0x00007fa6fc27e67f g_main_context_dispatch (libglib-2.0.so.0 + 0x5467f)
                #12 0x00007fa6fc27ea38 n/a (libglib-2.0.so.0 + 0x54a38)
                #13 0x00007fa6fc27ecef g_main_loop_run (libglib-2.0.so.0 + 0x54cef)
                #14 0x00007fa6fc97dfa9 n/a (libspice-server.so.1 + 0x51fa9)
                #15 0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #16 0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 668373:
                #0  0x00007fa6faaa5c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x0000558beeca56cf n/a (qemu-system-x86_64 + 0x72b6cf)
                #2  0x0000558beeca5ba5 n/a (qemu-system-x86_64 + 0x72bba5)
                #3  0x0000558beeca708d n/a (qemu-system-x86_64 + 0x72d08d)
                #4  0x0000558beee3eb78 n/a (qemu-system-x86_64 + 0x8c4b78)
                #5  0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 668339:
                #0  0x00007fa6faaa4256 __ppoll (libc.so.6 + 0xfc256)
                #1  0x0000558beee54dfe n/a (qemu-system-x86_64 + 0x8dadfe)
                #2  0x0000558beee526ee n/a (qemu-system-x86_64 + 0x8d86ee)
                #3  0x0000558beeaafaa7 n/a (qemu-system-x86_64 + 0x535aa7)
                #4  0x0000558beecaff46 n/a (qemu-system-x86_64 + 0x735f46)
                #5  0x00007fa6fa9cf24a __libc_start_call_main (libc.so.6 + 0x2724a)
                #6  0x00007fa6fa9cf305 __libc_start_main_impl (libc.so.6 + 0x27305)
                #7  0x0000558bee8a20a1 n/a (qemu-system-x86_64 + 0x3280a1)

                Stack trace of thread 668372:
                #0  0x00007fa6faaa5c5b __GI___ioctl (libc.so.6 + 0xfdc5b)
                #1  0x0000558beeca56cf n/a (qemu-system-x86_64 + 0x72b6cf)
                #2  0x0000558beeca5ba5 n/a (qemu-system-x86_64 + 0x72bba5)
                #3  0x0000558beeca708d n/a (qemu-system-x86_64 + 0x72d08d)
                #4  0x0000558beee3eb78 n/a (qemu-system-x86_64 + 0x8c4b78)
                #5  0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #6  0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 3815673:
                #0  0x00007fa6faa2de96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
                #1  0x00007fa6faa3083c __pthread_cond_wait_common (libc.so.6 + 0x8883c)
                #2  0x0000558beee3ed01 n/a (qemu-system-x86_64 + 0x8c4d01)
                #3  0x0000558beee3f8a0 n/a (qemu-system-x86_64 + 0x8c58a0)
                #4  0x0000558beee540d4 n/a (qemu-system-x86_64 + 0x8da0d4)
                #5  0x0000558beee3eb78 n/a (qemu-system-x86_64 + 0x8c4b78)
                #6  0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #7  0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 668340:
                #0  0x00007fa6faaa9719 syscall (libc.so.6 + 0x101719)
                #1  0x0000558beee3fcfa n/a (qemu-system-x86_64 + 0x8c5cfa)
                #2  0x0000558beee49602 n/a (qemu-system-x86_64 + 0x8cf602)
                #3  0x0000558beee3eb78 n/a (qemu-system-x86_64 + 0x8c4b78)
                #4  0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #5  0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)

                Stack trace of thread 668379:
                #0  0x00007fa6faa2de96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
                #1  0x00007fa6faa30558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
                #2  0x0000558beee3f68b n/a (qemu-system-x86_64 + 0x8c568b)
                #3  0x0000558bee8cbf2b n/a (qemu-system-x86_64 + 0x351f2b)
                #4  0x0000558bee8ccbc8 n/a (qemu-system-x86_64 + 0x352bc8)
                #5  0x0000558beee3eb78 n/a (qemu-system-x86_64 + 0x8c4b78)
                #6  0x00007fa6faa31134 start_thread (libc.so.6 + 0x89134)
                #7  0x00007fa6faab17dc __clone3 (libc.so.6 + 0x1097dc)
                ELF object binary architecture: AMD x86-64

GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/qemu-system-x86_64...
Reading symbols from /usr/lib/debug/.build-id/ff/4f7aff226255e307bbad356b50687ed09ad0e6.debug...

warning: Can't open file anon_inode:kvm-vcpu:3 which was expanded to anon_inode:kvm-vcpu:3 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:2 which was expanded to anon_inode:kvm-vcpu:2 during file-backed mapping note processing

warning: Can't open file /[aio] (deleted) during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:1 which was expanded to anon_inode:kvm-vcpu:1 during file-backed mapping note processing

warning: Can't open file anon_inode:kvm-vcpu:0 which was expanded to anon_inode:kvm-vcpu:0 during file-backed mapping note processing

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 668375]
[New LWP 668370]
[New LWP 668371]
[New LWP 668376]
[New LWP 668373]
[New LWP 668339]
[New LWP 668372]
[New LWP 3815673]
[New LWP 668340]
[New LWP 668379]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/kvm -id 123 -name VM,debug-threads=on -no-shutdown -chardev'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa6faa3cfcd in unlink_chunk (p=p@entry=0x7fa4dc292330, av=0x7fa4dc000030) at ./malloc/malloc.c:1628
1628    ./malloc/malloc.c: No such file or directory.
[Current thread is 1 (Thread 0x7fa4e6bff6c0 (LWP 668375))]

 

[DeeMaas]

(gdb) thread apply all backtrace

Thread 10 (Thread 0x7fa4c97ff6c0 (LWP 668379)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x558bf1769228) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x558bf1769228, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007fa6faa2defb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558bf1769228, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fa6faa30558 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558bf1769238, cond=0x558bf1769200) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x558bf1769200, mutex=mutex@entry=0x558bf1769238) at ./nptl/pthread_cond_wait.c:618
#5  0x0000558beee3f68b in qemu_cond_wait_impl (cond=0x558bf1769200, mutex=0x558bf1769238, file=0x558beef03cf4 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:225
#6  0x0000558bee8cbf2b in vnc_worker_thread_loop (queue=queue@entry=0x558bf1769200) at ../ui/vnc-jobs.c:248
#7  0x0000558bee8ccbc8 in vnc_worker_thread (arg=arg@entry=0x558bf1769200) at ../ui/vnc-jobs.c:362
#8  0x0000558beee3eb78 in qemu_thread_start (args=0x558bf106e760) at ../util/qemu-thread-posix.c:541
#9  0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7fa6f78996c0 (LWP 668340)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000558beee3fcfa in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at ./include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x558bef7929c8 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:464
#3  0x0000558beee49602 in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:278
#4  0x0000558beee3eb78 in qemu_thread_start (args=0x558bf0b0b5b0) at ../util/qemu-thread-posix.c:541
#5  0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7fa4e57fe6c0 (LWP 3815673)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fa4e57f8fe0, op=393, expected=0, futex_word=0x558bf0dd8b44) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x558bf0dd8b44, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fa4e57f8fe0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007fa6faa2defb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x558bf0dd8b44, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fa4e57f8fe0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007fa6faa3083c in __pthread_cond_wait_common (abstime=0x7fa4e57f8fe0, clockid=0, mutex=0x558bf0dd8ab0, cond=0x558bf0dd8b18) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x558bf0dd8b18, mutex=mutex@entry=0x558bf0dd8ab0, abstime=abstime@entry=0x7fa4e57f8fe0) at ./nptl/pthread_cond_wait.c:643
#5  0x0000558beee3ed01 in qemu_cond_timedwait_ts (cond=cond@entry=0x558bf0dd8b18, mutex=mutex@entry=0x558bf0dd8ab0, ts=ts@entry=0x7fa4e57f8fe0, file=file@entry=0x558bef09afd8 "../util/thread-pool.c", line=line@entry=90) at ../util/qemu-thread-posix.c:239
#6  0x0000558beee3f8a0 in qemu_cond_timedwait_impl (cond=0x558bf0dd8b18, mutex=0x558bf0dd8ab0, ms=<optimized out>, file=0x558bef09afd8 "../util/thread-pool.c", line=90) at ../util/qemu-thread-posix.c:253
#7  0x0000558beee540d4 in worker_thread (opaque=opaque@entry=0x558bf0dd8aa0) at ../util/thread-pool.c:90
#8  0x0000558beee3eb78 in qemu_thread_start (args=0x558bf134f1b0) at ../util/qemu-thread-posix.c:541
#9  0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7fa6f4fff6c0 (LWP 668372)):
#0  __GI___ioctl (fd=30, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x0000558beeca56cf in kvm_vcpu_ioctl (cpu=cpu@entry=0x558bf0efaa10, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x0000558beeca5ba5 in kvm_cpu_exec (cpu=cpu@entry=0x558bf0efaa10) at ../accel/kvm/kvm-all.c:2991
#3  0x0000558beeca708d in kvm_vcpu_thread_fn (arg=arg@entry=0x558bf0efaa10) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000558beee3eb78 in qemu_thread_start (args=0x558bf0f03890) at ../util/qemu-thread-posix.c:541
#5  0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7fa6f7b02500 (LWP 668339)):
#0  0x00007fa6faaa4256 in __ppoll (fds=0x558bf145a6a0, nfds=151, timeout=<optimized out>, timeout@entry=0x7ffd99e35fa0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#1  0x0000558beee54dfe in ppoll (__ss=0x0, __timeout=0x7ffd99e35fa0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:64
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=983080) at ../util/qemu-timer.c:351
#3  0x0000558beee526ee in os_host_main_loop_wait (timeout=983080) at ../util/main-loop.c:308
#4  main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:592
#5  0x0000558beeaafaa7 in qemu_main_loop () at ../softmmu/runstate.c:732
#6  0x0000558beecaff46 in qemu_default_main () at ../softmmu/main.c:37
--Type <RET> for more, q to quit, c to continue without paging--
#7  0x00007fa6fa9cf24a in __libc_start_call_main (main=main@entry=0x558bee8a0480 <main>, argc=argc@entry=86, argv=argv@entry=0x7ffd99e361b8) at ../sysdeps/nptl/libc_start_call_main.h:58
#8  0x00007fa6fa9cf305 in __libc_start_main_impl (main=0x558bee8a0480 <main>, argc=86, argv=0x7ffd99e361b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd99e361a8) at ../csu/libc-start.c:360
#9  0x0000558bee8a20a1 in _start ()

Thread 5 (Thread 0x7fa4e7dff6c0 (LWP 668373)):
#0  __GI___ioctl (fd=32, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x0000558beeca56cf in kvm_vcpu_ioctl (cpu=cpu@entry=0x558bf0f04e80, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x0000558beeca5ba5 in kvm_cpu_exec (cpu=cpu@entry=0x558bf0f04e80) at ../accel/kvm/kvm-all.c:2991
#3  0x0000558beeca708d in kvm_vcpu_thread_fn (arg=arg@entry=0x558bf0f04e80) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000558beee3eb78 in qemu_thread_start (args=0x558bf0f0de20) at ../util/qemu-thread-posix.c:541
#5  0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7fa4e5fff6c0 (LWP 668376)):
#0  0x00007fa6fa950350 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#1  0x00007fa6fa94ad80 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#2  0x00007fa6fa94b267 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#3  0x00007fa6fa92ccc3 in SSL_write () from /lib/x86_64-linux-gnu/libssl.so.3
#4  0x00007fa6fc97bfb0 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#5  0x00007fa6fc97c92c in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#6  0x00007fa6fc9681d3 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#7  0x00007fa6fc968347 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#8  0x00007fa6fc9685d3 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#9  0x00007fa6fc968700 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#10 0x00007fa6fc95279d in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#11 0x00007fa6fc27e67f in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#12 0x00007fa6fc27ea38 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007fa6fc27ecef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007fa6fc97dfa9 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#15 0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#16 0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7fa6f5bff6c0 (LWP 668371)):
#0  __GI___ioctl (fd=28, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x0000558beeca56cf in kvm_vcpu_ioctl (cpu=cpu@entry=0x558bf0ef1070, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x0000558beeca5ba5 in kvm_cpu_exec (cpu=cpu@entry=0x558bf0ef1070) at ../accel/kvm/kvm-all.c:2991
#3  0x0000558beeca708d in kvm_vcpu_thread_fn (arg=arg@entry=0x558bf0ef1070) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000558beee3eb78 in qemu_thread_start (args=0x558bf0efa060) at ../util/qemu-thread-posix.c:541
#5  0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fa6f6dd56c0 (LWP 668370)):
#0  __GI___ioctl (fd=26, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x0000558beeca56cf in kvm_vcpu_ioctl (cpu=cpu@entry=0x558bf0ebdeb0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3179
#2  0x0000558beeca5ba5 in kvm_cpu_exec (cpu=cpu@entry=0x558bf0ebdeb0) at ../accel/kvm/kvm-all.c:2991
#3  0x0000558beeca708d in kvm_vcpu_thread_fn (arg=arg@entry=0x558bf0ebdeb0) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000558beee3eb78 in qemu_thread_start (args=0x558bf0ec71b0) at ../util/qemu-thread-posix.c:541
#5  0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7fa4e6bff6c0 (LWP 668375)):
#0  0x00007fa6faa3cfcd in unlink_chunk (p=p@entry=0x7fa4dc292330, av=0x7fa4dc000030) at ./malloc/malloc.c:1628
#1  0x00007fa6faa3ff4d in _int_malloc (av=av@entry=0x7fa4dc000030, bytes=bytes@entry=4112) at ./malloc/malloc.c:4201
#2  0x00007fa6faa416e2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at ./malloc/malloc.c:3674
#3  0x00007fa6fc2846d1 in g_malloc0 () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x0000558bee8abcb0 in cursor_alloc (width=<optimized out>, height=<optimized out>) at ../ui/cursor.c:103
#5  0x0000558beeb2384b in qxl_cursor (group_id=1, cursor=0x7fa4d325b7c8, qxl=0x558bf25ccda0) at ../hw/display/qxl-render.c:252
#6  qxl_render_cursor (qxl=qxl@entry=0x558bf25ccda0, ext=ext@entry=0x7fa4e6bf9f70) at ../hw/display/qxl-render.c:333
--Type <RET> for more, q to quit, c to continue without paging--
#7  0x0000558beeb225ab in interface_get_cursor_command (sin=0x558bf25cd868, ext=0x7fa4e6bf9f70) at ../hw/display/qxl.c:821
#8  0x00007fa6fc97e1bc in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#9  0x00007fa6fc97ecac in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#10 0x00007fa6fc27e7a9 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#11 0x00007fa6fc27ea38 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#12 0x00007fa6fc27ecef in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007fa6fc97dfa9 in ?? () from /lib/x86_64-linux-gnu/libspice-server.so.1
#14 0x00007fa6faa31134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x00007fa6faab17dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

 

[DeeMaas]

 

[orion]

When VMs crashed I found that system storage latancy were over 7000 ms. Also every VM had 4Gb memory.
I delete storage replica and latency down below 20 ms. Also I increased memory to 6Gb. There are no crashes after this steps. I'm continue monitoring the situation.

I don't have storage latency and there are still random VMs crashes with messages in logs like yours. I only use SPICE on Windows Server VMs of which I don't have many, but sudden stops of machines are irritating. This started happening after the last updates.