qmp command 'query-proxmox-support' failed

[ozgurerdogan]

One node logs this error and vms are not reachable:

VM 221 qmp command failed - VM 221 qmp command 'query-proxmox-support' failed - unable to connect to VM 221 qmp socket - timeout after 31 retries

Could this be vm is eating so much resource or something related to pve? We have all updates installed.

This causes downtime. So any suggestion is welcome

proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.60-1-pve: 5.15.60-1
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.15.39-3-pve: 5.15.39-3
pve-kernel-5.15.39-2-pve: 5.15.39-2
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
pve-zsync: 2.2.3
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

 

[fiona]

Hi,
please post a big enough part of /var/log/syslog around the time the issue occurs. Are there any specific operations running (e.g. backup/replication/...) before the issue occurs?

 

[ozgurerdogan]

No specific task. But one vm is doing high disk io. (12 TB Disk read in 8 days with nvme speed.)

I set disk type to Async IO: native (as per a thread in forum). Disk type is Hard Disk (scsi0) with VirtIO SCSI (Shall I try sata?)

You can see logs in attached files. There are some following Tained message but these logs are not there on all hangs..

INFO: task z_wr_int_3:3898 blocked for more than 120 seconds.
Tainted: P O 5.15.64-1-pve #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:z_wr_int_3 state stack: 0 pid: 3898 ppid: 2 flags:0x00004000
Call Trace:
<TASK>
__schedule+0x34e/0x1740
? cpumask_next_wrap+0x33/0x90
? select_task_rq_fair+0x18b/0x1af0
schedule+0x69/0x110
schedule_preempt_disabled+0xe/0x20
__mutex_lock.constprop.0+0x255/0x480
__mutex_lock_slowpath+0x13/0x20
mutex_lock+0x38/0x50
arc_buf_destroy+0x65/0x110 [zfs]
dbuf_destroy+0x31/0x500 [zfs]
? __cond_resched+0x1a/0x50
dbuf_evict_one+0x13c/0x1a0 [zfs]
dbuf_rele_and_unlock+0x727/0x7e0 [zfs]
? dsl_dataset_block_born+0x256/0x400 [zfs]
dbuf_write_done+0xeb/0x200 [zfs]
arc_write_done+0x8f/0x420 [zfs]
zio_done+0x40b/0x1290 [zfs]
zio_execute+0x95/0x160 [zfs]
taskq_thread+0x29f/0x4d0 [spl]
? wake_up_q+0x90/0x90
? zio_gang_tree_free+0x70/0x70 [zfs]
? taskq_thread_spawn+0x60/0x60 [spl]
kthread+0x12a/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x22/0x30
</TASK>
INFO: task txg_sync:4039 blocked for more than 120 seconds.
Tainted: P O 5.15.64-1-pve #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:txg_sync state stack: 0 pid: 4039 ppid: 2 flags:0x00004000
Call Trace:
<TASK>

 

[fiona]

Yes, there are ZFS-related hung tasks in the kernel. Did it start happening after a specific kernel update? Is there enough RAM for ZFS https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_zfs_limit_memory_usage ? Maybe setting bandwidth limits for the VM's disk helps?

 

[ozgurerdogan]

Not sure about kernel update. But I can say I updated kernel about 2 weeks ago.
There are 15 GB free ram of 128 GB total. But there are some backup zfs snapshot from other nodes. They do not need ram when not in use right?
I will consider limiting disk. But I saw many similar threads in forum. So wanted to check if there is quick workaround.

Again those zfs hangs are not in logs on some other hangs...