VMs hang or become unresponsive

J

jamin.collins

New Member
May 16, 2023
16
2
3
I've searched through the forum and read numerous posts from others that have virtual machines that seem to rather randomly hang. I've tried the various suggestions (turning off ballooning, etc) and find that I still have a few VMs that just become unresponsive.

Prior to switching to Proxmox, I was using this same hardware to run the same virtual machine workloads without the virtual machines hanging. My previous configuration was built on top of Arch Linux with CEPH backing libvirt/qemu virtual machines. The disk images of the virtual machines are the same (imported) and the virtual machine definitions are as similar as I could make them when recreating them in Proxmox.

I'm open to suggestions on how to correct/fix this instability.
 
  • Like
Reactions: rahman
What is your networking, especially for CEPH and corosync. CEPH is very demanding, and corosync is very critical and sensitive to network blip that CEPH can cause.
 
Dual 2.5GB dedicated networks for the cluster. Same as I was using before moving to Proxmox. Literally, the only change is moving to Proxmox (base OS and virtual machine definitions).
 
Is corosync on a bond for the dual networks shared with CEPH, or is one network dedicated for CEPH and the other dedicated for corosync?
So each node has two 2.5GB network ports?
 
There are no bonds. Just two 2.5GB networks, link 0 and link 1 for Proxmox. CEPH uses link 0 for its cluster network and link 1 for its monitor and client traffic. While corosync lists both link 0 and link 1.

> So each node has two 2.5GB network ports?

Each node has 3 network ports. The third is a external traffic.
 
Hi,
please share the output of pveversion -v and at least one configuration of an affected VM qm config <ID>. Can you correlate the hangs with certain operations like snapshot or backup? What is the output of qm status <ID> --verbose for a hanging VM? Is there anything in the host's or guest's system logs/journal?
 
I have not been able to find anything meaningful in the host or VM logs. Here is output from the requested commands.

I might still have the VM definition from when it was running under libvirt/qemu if that would help.

I do see there are some updates for the host, I'll apply those.

Code: 
# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph: 18.2.0-pve2
ceph-fuse: 18.2.0-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.3
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-2
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.5
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1

Code: 
# qm config 113
agent: 1
balloon: 0
boot: order=virtio0;ide2;net0
cores: 2
cpu: qemu64
ide2: none,media=cdrom
memory: 2048
meta: creation-qemu=8.1.2,ctime=1702392697
name: archive
net0: virtio=52:54:00:88:51:2a,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=955c7a39-2ecc-4365-ab8e-bf647f17af5a
sockets: 1
virtio0: nvme:vm-113-disk-0,iothread=1,size=20G
virtio1: hdd:vm-113-disk-0,iothread=1,size=1536G
vmgenid: e1d6f708-da71-4eb0-bbac-b00462cce650

Code: 
# qm status 113 --verbose
cpus: 2
disk: 0
diskread: 0
diskwrite: 0
maxdisk: 21474836480
maxmem: 2147483648
mem: 878279040
name: archive
netin: 41565374
netout: 4076680
nics:
    tap113i0:
        netin: 41565374
        netout: 4076680
pid: 2268577
proxmox-support:
qmpstatus: running
serial: 1
status: running
uptime: 238739
vmid: 113
 
Last edited:
I've left the VM in question in the hung state for now, as it's not super critical. Please let me know if there are any tests or investigation I can do on it.
 
jamin.collins said:
I've left the VM in question in the hung state for now, as it's not super critical. Please let me know if there are any tests or investigation I can do on it.
Click to expand...
Please install debugger and debug symbols with apt install pve-qemu-kvm-dbgsym gdb and share the output of gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/113.pid).

Additionally you can also install jq and socat and run
Code: 
echo '{"execute": "qmp_capabilities"}{"execute": "query-block", "arguments": {}}' | socat - /var/run/qemu-server/113.qmp | jq
which might be interesting too.
 
Looks like the package updates might have skewed things a bit from the running binary and the debug symbols package.

Also the socat command never completes, it just hangs.

I can reboot this host to get everything up to date and in sync and wait for the VM to hang again.

Code: 
# gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/113.pid)
[New LWP 2268578]
[New LWP 2268579]
[New LWP 2268580]
[New LWP 2268582]
[New LWP 2268583]
[New LWP 2268584]
[New LWP 2268585]
[New LWP 2268589]
[New LWP 2268590]
[New LWP 2268591]
[New LWP 2268592]
[New LWP 2268593]
[New LWP 2268594]
[New LWP 2268595]
[New LWP 2268596]
[New LWP 2268597]
[New LWP 2268598]
[New LWP 2268599]
[New LWP 2268600]
[New LWP 2268601]
[New LWP 2268602]
[New LWP 2268603]
[New LWP 2268610]
[New LWP 2268611]
[New LWP 2268612]
[New LWP 2268613]
[New LWP 2268614]
[New LWP 2268615]
[New LWP 2268616]
[New LWP 2268617]
[New LWP 2268618]
[New LWP 2268619]
[New LWP 2268620]
[New LWP 2268687]
[New LWP 2268688]
[New LWP 2268689]
[New LWP 2268691]

warning: Could not load vsyscall page because no executable was specified
0x00007f9d36ed9fbb in ?? ()

Thread 38 (LWP 2268691 "vnc_worker"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 37 (LWP 2268689 "CPU 1/KVM"):
#0  0x00007f9d36ed9fbb in ?? ()
#1  0x00007f9d36edf5a3 in ?? ()
#2  0x00007f9cecff3f10 in ?? ()
#3  0x000055bd89f98b00 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 36 (LWP 2268688 "CPU 0/KVM"):
#0  0x00007f9d36ed9fbb in ?? ()
#1  0x00007f9d36edf5a3 in ?? ()
#2  0x00007f9ced7f4f10 in ?? ()
#3  0x000055bd89f98b00 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 35 (LWP 2268687 "vhost-2268577"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 34 (LWP 2268620 "taskfin_librbd"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 33 (LWP 2268619 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 32 (LWP 2268618 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9ceeff7f90 in ?? ()
#2  0x00007f9d00000089 in ?? ()
#3  0x000055bd8bc33744 in ?? ()
#4  0x00007f9ceeff8090 in ?? ()
#5  0x000055bd8bc33700 in ?? ()
#6  0x000055bd8bc33718 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 31 (LWP 2268617 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 30 (LWP 2268616 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000546a47 in ?? ()
#2  0x000055bd00000089 in ?? ()
#3  0x000055bd8c64c1c8 in ?? ()
#4  0x00007f9cefffa090 in ?? ()
#5  0x0000000065cb9823 in ?? ()
#6  0x000055bd8c64c1a0 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 29 (LWP 2268615 "ms_local"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 28 (LWP 2268614 "ms_dispatch"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x000055bd00000189 in ?? ()
#3  0x000055bd8c6f9020 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 27 (LWP 2268613 "ceph_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 26 (LWP 2268612 "io_context_pool"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 25 (LWP 2268611 "io_context_pool"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9ce8029f70 in ?? ()
#2  0x0000000000000189 in ?? ()
#3  0x000055bd8bbd89e8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 24 (LWP 2268610 "service"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9d037f9050 in ?? ()
#2  0xffffffff00000089 in ?? ()
#3  0x00007f9cf80047b8 in ?? ()
#4  0x00007f9d037f90c0 in ?? ()
#5  0x00007f9cf8004790 in ?? ()
#6  0x00007f9cf8004790 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 23 (LWP 2268603 "msgr-worker-2"):
#0  0x00007f9d36f5cc66 in ?? ()
#1  0x000055bd8c6d2a80 in ?? ()
#2  0x00000028e864d2e0 in ?? ()
#3  0x000055bd8c6c3bf0 in ?? ()
#4  0x0000753000001388 in ?? ()
#5  0x000055bd8c6533e0 in ?? ()
#6  0x00007f9d350244d7 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 22 (LWP 2268602 "msgr-worker-1"):
#0  0x00007f9d36f5cc66 in ?? ()
#1  0x000055bd8c6a6b70 in ?? ()
#2  0x00000025e864d2e0 in ?? ()
#3  0x000055bd8c697cb0 in ?? ()
#4  0x0000753000001388 in ?? ()
#5  0x000055bd8c652260 in ?? ()
#6  0x00007f9d350244d7 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 21 (LWP 2268601 "msgr-worker-0"):
#0  0x00007f9d36f5cc66 in ?? ()
#1  0x000055bd8c67b530 in ?? ()
#2  0x00000022e864d2e0 in ?? ()
#3  0x000055bd8c66bd70 in ?? ()
#4  0x0000753000001388 in ?? ()
#5  0x000055bd8c650260 in ?? ()
#6  0x00007f9d350244d7 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 20 (LWP 2268600 "log"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 19 (LWP 2268599 "taskfin_librbd"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 18 (LWP 2268598 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 17 (LWP 2268597 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9d0f7f8f90 in ?? ()
#2  0x00007f9d00000089 in ?? ()
#3  0x000055bd8b7788a4 in ?? ()
#4  0x00007f9d0f7f9090 in ?? ()
#5  0x000055bd8b778860 in ?? ()
#6  0x000055bd8b778878 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 16 (LWP 2268596 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 15 (LWP 2268595 "safe_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000546a47 in ?? ()
#2  0x000055bd00000089 in ?? ()
#3  0x000055bd8b9e49e8 in ?? ()
#4  0x00007f9d20ff4090 in ?? ()
#5  0x0000000065cb9823 in ?? ()
#6  0x000055bd8b9e49c0 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 14 (LWP 2268594 "ms_local"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 13 (LWP 2268593 "ms_dispatch"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9d21ff5f10 in ?? ()
#2  0x000055bd00000189 in ?? ()
#3  0x000055bd8ba959c0 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 12 (LWP 2268592 "ceph_timer"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 11 (LWP 2268591 "io_context_pool"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9d08022080 in ?? ()
#2  0xbb9eae1200000189 in ?? ()
#3  0x000055bd8b66f3f8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 10 (LWP 2268590 "io_context_pool"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9d0400b960 in ?? ()
#2  0xbb9eae1200000189 in ?? ()
#3  0x000055bd8b66f3fc in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 9 (LWP 2268589 "service"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x00007f9d237f9050 in ?? ()
#2  0xffffffff00000089 in ?? ()
#3  0x000055bd8b762388 in ?? ()
#4  0x00007f9d237f90c0 in ?? ()
#5  0x000055bd8b762360 in ?? ()
#6  0x000055bd8b762360 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 8 (LWP 2268585 "msgr-worker-2"):
#0  0x00007f9d36f5cc66 in ?? ()
#1  0x000055bd8ba6b1b0 in ?? ()
#2  0x0000001ae864d2e0 in ?? ()
#3  0x000055bd8ba5c470 in ?? ()
#4  0x0000753000001388 in ?? ()
#5  0x000055bd8b9ee810 in ?? ()
#6  0x00007f9d350244d7 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 7 (LWP 2268584 "msgr-worker-1"):
#0  0x00007f9d36f5cc66 in ?? ()
#1  0x000055bd8ba3f2a0 in ?? ()
#2  0x00000017e864d2e0 in ?? ()
#3  0x000055bd8ba30530 in ?? ()
#4  0x0000753000001388 in ?? ()
#5  0x000055bd8b675440 in ?? ()
#6  0x00007f9d350244d7 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 6 (LWP 2268583 "msgr-worker-0"):
#0  0x00007f9d36f5cc66 in ?? ()
#1  0x000055bd8ba136a8 in ?? ()
#2  0x00000014e864d2e0 in ?? ()
#3  0x000055bd8ba045f0 in ?? ()
#4  0x0000753000001388 in ?? ()
#5  0x000055bd8b67cc40 in ?? ()
#6  0x00007f9d350244d7 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 5 (LWP 2268582 "log"):
#0  0x00007f9d36ed9da6 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 4 (LWP 2268580 "kvm"):
#0  0x00007f9d36ed9fbb in ?? ()
#1  0x00007f9d36ee02d2 in ?? ()
#2  0x000000000000a201 in ?? ()
#3  0x000055bd89f98b00 in ?? ()
#4  0x000055bd89f98b00 in ?? ()
#5  0x000055bd8965cee3 in ?? ()
#6  0x0000000000000000 in ?? ()

Thread 3 (LWP 2268579 "kvm"):
#0  0x00007f9d36f50156 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 2 (LWP 2268578 "call_rcu"):
#0  0x00007f9d36f55559 in ?? ()
#1  0x000055bd8965dc8a in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 1 (LWP 2268577 "kvm"):
#0  0x00007f9d36ed9fbb in ?? ()
#1  0x00007f9d36ee032a in ?? ()
#2  0x000055bd8d45e640 in ?? ()
#3  0x000055bd8b972110 in ?? ()
#4  0x000055bd8b972110 in ?? ()
#5  0x000055bd8965cee3 in ?? ()
#6  0x000055bd8bb8bb00 in ?? ()
#7  0xbb9eae124dc85700 in ?? ()
#8  0x0000000000000000 in ?? ()
[Inferior 1 (process 2268577) detached]
 
jamin.collins said:
Looks like the package updates might have skewed things a bit from the running binary and the debug symbols package.
Click to expand...
Yes, the installed version for the debug symbols needs to match the running instance to be able to get debug info.

jamin.collins said:
Also the socat command never completes, it just hangs.
Click to expand...
The command just queries the attached block device. Either the QEMU instance is deadlocked (we should see this in the debug info) or the issue is with the network connection or Ceph storage.
 
Given that all the other (19) VMs are backed by the same CEPH cluster are working fine, my guess is QEMU.

I'll restart this host and keep an eye on it.
 
It's another VM that has hung this time, I've attached the output of the above commands. The forum would not allow me to inline the output as it's too long.

As before, the second command never returns.
 

Attachments

  • vm-hang-proxmox-gdb-trace.txt
    52.9 KB · Views: 2
Well, that looks like a deadlock. What I'd guess from the output is:

Code: 
Thread 1 (Thread 0x7f71a621d500 (LWP 18709) "kvm"):
#0  futex_wait (private=0, expected=2, futex_word=0x55a41e9594e0) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x55a41e9594e0, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f71a8d0241a in lll_mutex_lock_optimized (mutex=0x55a41e9594e0) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=mutex@entry=0x55a41e9594e0) at ./nptl/pthread_mutex_lock.c:128
#4  0x000055a41c361d83 in qemu_mutex_lock_impl (mutex=0x55a41e9594e0, file=0x55a41c5bd3e7 "../util/async.c", line=728) at ../util/qemu-thread-posix.c:94
#5  0x000055a41c2467b8 in bdrv_do_drained_begin (poll=<optimized out>, parent=0x0, bs=0x55a41eb63760) at ../block/io.c:381
#6  bdrv_do_drained_begin (bs=0x55a41eb63760, parent=0x0, poll=<optimized out>) at ../block/io.c:351
#7  0x000055a41c23c4e0 in blk_drain (blk=0x55a41eb6fff0) at ../block/block-backend.c:2077
#8  0x000055a41be37cee in virtio_blk_data_plane_stop (vdev=<optimized out>) at ../hw/block/dataplane/virtio-blk.c:363
#9  0x000055a41bf96b40 in virtio_bus_stop_ioeventfd (bus=0x55a420581e10) at ../hw/virtio/virtio-bus.c:259
#10 0x000055a41c143f4d in virtio_vmstate_change (opaque=0x55a420581e90, running=<optimized out>, state=<optimized out>) at ../hw/virtio/virtio.c:3168
#11 0x000055a41bfd21e0 in vm_state_notify (running=running@entry=false, state=state@entry=RUN_STATE_PAUSED) at ../softmmu/runstate.c:341
#12 0x000055a41bfc922a in do_vm_stop (send_stop=true, state=RUN_STATE_PAUSED) at ../softmmu/cpus.c:263
#13 vm_stop (state=state@entry=RUN_STATE_PAUSED) at ../softmmu/cpus.c:676
#14 0x000055a41c01326b in qmp_stop (errp=errp@entry=0x7ffd6d1c95f8) at ../monitor/qmp-cmds.c:62
#15 0x000055a41c31cde5 in qmp_marshal_stop (args=<optimized out>, ret=<optimized out>, errp=0x7f71a5427e90) at qapi/qapi-commands-misc.c:211
#16 0x000055a41c354b70 in do_qmp_dispatch_bh (opaque=0x7f71a5427ea0) at ../qapi/qmp-dispatch.c:136
#17 0x000055a41c374016 in aio_bh_poll (ctx=ctx@entry=0x55a41e6e0650) at ../util/async.c:216
#18 0x000055a41c35e6ee in aio_dispatch (ctx=0x55a41e6e0650) at ../util/aio-posix.c:423
#19 0x000055a41c373b5e in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at ../util/async.c:358
#20 0x00007f71aa53e7a9 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#21 0x000055a41c3755c8 in glib_pollfds_poll () at ../util/main-loop.c:290
#22 os_host_main_loop_wait (timeout=0) at ../util/main-loop.c:313
#23 main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:592
#24 0x000055a41bfd2a87 in qemu_main_loop () at ../softmmu/runstate.c:732
#25 0x000055a41c1d2f36 in qemu_default_main () at ../softmmu/main.c:37
#26 0x00007f71a8c9d24a in __libc_start_call_main (main=main@entry=0x55a41bdc3460 <main>, argc=argc@entry=76, argv=argv@entry=0x7ffd6d1c9968) at ../sysdeps/nptl/libc_start_call_main.h:58
#27 0x00007f71a8c9d305 in __libc_start_main_impl (main=0x55a41bdc3460 <main>, argc=76, argv=0x7ffd6d1c9968, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd6d1c9958) at ../csu/libc-start.c:360
#28 0x000055a41bdc5081 in _start ()

Thread 1 here waits on the AioContext lock for the drive's IO thread.

Code: 
Thread 4 (Thread 0x7f71a4b256c0 (LWP 18712) "kvm"):
#0  futex_wait (private=0, expected=2, futex_word=0x55a41cc9dba0 <qemu_global_mutex>) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x55a41cc9dba0 <qemu_global_mutex>, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f71a8d023c2 in lll_mutex_lock_optimized (mutex=0x55a41cc9dba0 <qemu_global_mutex>) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=mutex@entry=0x55a41cc9dba0 <qemu_global_mutex>) at ./nptl/pthread_mutex_lock.c:93
#4  0x000055a41c361d83 in qemu_mutex_lock_impl (mutex=0x55a41cc9dba0 <qemu_global_mutex>, file=0x55a41c54028a "../softmmu/physmem.c", line=2617) at ../util/qemu-thread-posix.c:94
#5  0x000055a41bfc8aa6 in qemu_mutex_lock_iothread_impl (file=file@entry=0x55a41c54028a "../softmmu/physmem.c", line=line@entry=2617) at ../softmmu/cpus.c:504
#6  0x000055a41c1786cf in prepare_mmio_access (mr=0x55a41e8bd4a0) at ../softmmu/physmem.c:2617
#7  address_space_stl_internal (endian=DEVICE_LITTLE_ENDIAN, result=0x0, attrs=..., val=38, addr=<optimized out>, as=<optimized out>) at ./memory_ldst.c.inc:318
#8  address_space_stl_le (as=<optimized out>, addr=<optimized out>, val=38, attrs=..., result=0x0) at ./memory_ldst.c.inc:357
#9  0x000055a41c10b12e in virtio_blk_rw_complete (opaque=<optimized out>, ret=0) at ../hw/block/virtio-blk.c:133
#10 0x000055a41c23ba9d in blk_aio_complete (acb=0x7f719c0068c0) at ../block/block-backend.c:1563
#11 blk_aio_complete (acb=0x7f719c0068c0) at ../block/block-backend.c:1560
#12 blk_aio_write_entry (opaque=0x7f719c0068c0) at ../block/block-backend.c:1630
#13 0x000055a41c376cdb in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at ../util/coroutine-ucontext.c:177
#14 0x00007f71a8cc79c0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x00007f71a5320450 in ?? ()
#16 0x0000000000000000 in ?? ()

Thread 4 here waits on the "big QEMU lock".

It's likely that each thread is currently holding the lock the other thread waits for.

It also looks like the issue happened during a QMP stop command which is used when taking a snapshot or doing a Pause/qm suspend for the guest. Did you use one of those operations? The other place where it is used is for suspend mode backup which is just there for historic reasons for VMs. Should you be using that, please switch to snapshot mode.

Potential workarounds are:
- Not using iothread setting.
- Using VirtIO SCSI instead of VirtIO block.

The good news is that AioContext locks were removed in QEMU 9.0 exactly because such issues would pop up in different scenarios. The bad news is that it's still in development and will take a while until it reaches Proxmox VE.

I'll see if I can reproduce the issue and whether it can be fixed in QEMU 8.1.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back