Cannot run migrate VMs on node - unable to connect to VM qmp socket

[decibel83]

Hi,
I am experiencing many problems on one node of my Proxmox cluster.

Some containers are locked by backup (but no backups are running on that node), and one VM is not booting, nor from the hard disk:

and even not from a rescue boot CD:

I tried to migrate that VM to another node but the migration job starts and remains hanged in an endless point:



In the syslog of that node I see these messages:

Aug  1 12:45:02 node04 pvedaemon[8578]: VM 1151 qmp command failed - VM 1151 qmp command 'quit' failed - unable to connect to VM 1151 qmp socke
t - timeout after 31 retries
Aug  1 12:45:02 node04 pvedaemon[8578]: VM quit/powerdown failed - terminating now with SIGTERM
Aug  1 12:45:05 node04 pvedaemon[23141]: VM 1151 qmp command failed - VM 1151 qmp command 'query-proxmox-support' failed - unable to connect to VM 1151 qmp socket - timeout after 31 retries
Aug  1 12:45:08 node04 pvestatd[1616]: VM 1151 qmp command failed - VM 1151 qmp command 'query-proxmox-support' failed - unable to connect to VM 1151 qmp socket - timeout after 31 retries

Proxmox is updated to latest version:

root@node04:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.4.157-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-7
pve-kernel-helper: 7.2-7
pve-kernel-5.4: 6.4-19
pve-kernel-5.15.39-2-pve: 5.15.39-2
pve-kernel-5.4.195-1-pve: 5.4.195-1
pve-kernel-5.4.157-1-pve: 5.4.157-1
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-3-pve: 4.13.13-34
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.4-1-pve: 4.13.4-26
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1

Could you help me to understand what's going on, please?

Thank you very much!
Bye

 

[Neobin]

proxmox-ve: 7.2-1 (running kernel: 5.4.157-1-pve)
pve-kernel-5.15.39-2-pve: 5.15.39-2

Did you upgrade from PVE 6.x recently? Without an reboot?
Anyway reboot the host for the new kernel to take effect.

 

[LnxBil]

Have you a configured, installed and working qemu guest agent process in your VM?

 

[decibel83]

Did you upgrade from PVE 6.x recently? Without an reboot?
Anyway reboot the host for the new kernel to take effect.

It's true, but it's strange that other VMs start and migrate without any problems.
Anyway, I will reboot with the new kernel and check again.
Thank you!