block_resize + IO thread enabled = qmp command 'block_resize' failed - got timeout

[cosme]

Hi!

Since the last upgrade we received the following errors when attempting an online resize on a ceph disk with virtio scsi single and IO thread enabled:

pvedaemon[48698]: <root@pam> update VM 139: resize --disk scsi2 --size +16G
kernel: rbd2: detected capacity change from 34359738368 to 51539607552
pvedaemon[48698]: VM 139 qmp command failed - VM 139 qmp command 'block_resize' failed - got timeout
pvedaemon[34945]: VM 139 qmp command failed - VM 139 qmp command 'query-proxmox-support' failed - got timeout
pvestatd[4874]: VM 139 qmp command failed - VM 139 qmp command 'query-proxmox-support' failed - got timeout
pvestatd[4874]: status update time (6.291 seconds)
pvestatd[4874]: VM 139 qmp command failed - VM 139 qmp command 'query-proxmox-support' failed - got timeout

Then the VM stops responding and progressively the same happens with the rest of the machines on the same Proxmox node.

proxmox-ve: 6.3-1 (running kernel: 5.4.103-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-7
pve-kernel-helper: 6.3-7
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.101-1-pve: 5.4.101-1
ceph-fuse: 14.2.16-pve1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.3-1
proxmox-backup-client: 1.0.9-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-6
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-3
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-7
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2

 

[Stefan_R]

You commented already on this thread, but it seems your issue is seperate from the one discussed there. I can reliably reproduce it, though only on krbd. This also appears to be a QEMU bug.

I will take a look seperately, but for now, can you try disabling krbd, then restarting your VM and retrying the resize command?

 

[fkr]

This should be fixed in qemu 6.0. The RC (https://download.qemu.org/qemu-6.0.0-rc2.tar.xz) already contains the fix.
This commit is the fix:

https://github.com/qemu/qemu/commit/8089eab2bd5fb160b038e64e14cf7ffb3f37091e

Upstream Bugreport: https://bugzilla.redhat.com/show_bug.cgi?id=1903511

Am not a proxmox user myself, so if anyone can validate that this fixes the issue, please do so.

(Infos thanks to the nice folks on #qemu on irc.oftc.net )

 

[Stefan_R]

Thanks for the update! We already backported the fix you linked in pve-qemu-kvm 5.2.0-5, currently available on pve-no-subscription