WARNING: unable to connect to VM 100 socket - timeout after 31 retries

[ozgurerdogan]

When I want to restart or shutdown wm it stuck at the end of logging of screen. And I have to use Stop command.

Here is logs. Do you have any idea?

ov 17 22:40:08 vztl3 pvedaemon[2671]: WARNING: unable to connect to VM 100 socket - timeout after 31 retries
Nov 17 22:40:11 vztl3 pvedaemon[7543]: WARNING: VM still running - terminating now with SIGTERM
Nov 17 22:40:12 vztl3 pvestatd[2692]: WARNING: unable to connect to VM 100 socket - timeout after 31 retries
Nov 17 22:40:15 vztl3 pvedaemon[2671]: WARNING: unable to connect to VM 100 socket - timeout after 31 retries
Nov 17 22:40:17 vztl3 pvedaemon[7576]: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout
Nov  17 22:40:17 vztl3 pvedaemon[7234]: <root@pam> end task  UPID:vztl3:00001D98:0004DDD5:546A5D27:qmstop:100:root@pam: unexpected  status
Nov 17 22:40:21 vztl3 pvedaemon[7543]: WARNING: VM still running - terminating now with SIGKILL
Nov 17 22:40:21 vztl3 kernel: vmbr0: port 5(tap100i0) entering disabled state
Nov 17 22:40:21 vztl3 kernel: vmbr0: port 5(tap100i0) entering disabled state
Nov 17 22:40:21 vztl3 pvestatd[2692]: WARNING: unable to connect to VM 100 socket - Connection refused
Nov 17 22:40:21 vztl3 pvedaemon[2671]: <root@pam> end task UPID:vztl3:00001CF2:0004B697:546A5CC2:vncproxy:100:root@pam: OK
Nov 17 22:40:22 vztl3 pvedaemon[2670]: <root@pam> end task UPID:vztl3:00001D77:0004D3BB:546A5D0D:qmstop:100:root@pam: OK
Nov 17 22:40:22 vztl3 pvedaemon[7613]: starting vnc proxy UPID:vztl3:00001DBD:0004E403:546A5D36:vncproxy:100:root@pam:
Nov  17 22:40:22 vztl3 pvedaemon[7234]: <root@pam> starting task  UPID:vztl3:00001DBD:0004E403:546A5D36:vncproxy:100:root@pam:
Nov 17  22:40:23 vztl3 ntpd[2360]: Deleting interface #11 tap100i0,  fe80::e8a6:fdff:fe98:22f1#123, interface stats: received=0, sent=0,  dropped=0, active_time=637 secs
Nov 17 22:40:23 vztl3 ntpd[2360]: peers refreshed
Nov 17 22:40:23 vztl3 qm[7615]: VM 100 qmp command failed - VM 100 not running
Nov  17 22:40:23 vztl3 pvedaemon[7613]: command '/bin/nc -l -p 5900 -w 10 -c  '/usr/sbin/qm vncproxy 100 2>/dev/null'' failed: exit code 255
Nov  17 22:40:23 vztl3 pvedaemon[7234]: <root@pam> end task  UPID:vztl3:00001DBD:0004E403:546A5D36:vncproxy:100:root@pam: command  '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 100 2>/dev/null''  failed: exit code 255

 

[dietmar]

What is the output of

# pveversion -v

 

[ozgurerdogan]

Vm seems to be running fine but not sure if it safe to run so.

proxmox-ve-2.6.32: 3.3-139 (running kernel: 2.6.32-34-pve)
pve-manager: 3.3-5 (running version: 3.3-5/bfebec03)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-34-pve: 2.6.32-139
pve-kernel-2.6.32-31-pve: 2.6.32-132
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.3-3
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-25
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

 

[dietmar]

Vm seems to be running fine but not sure if it safe to run so.

Can you reproduce that bug?

 

[ozgurerdogan]

Can you reproduce that bug?

Actually I had moved that vm from another node and yesterday I could not shutdown vm as it was stucking at the end of shutdown. Other vms on same node fine. And today, this vm also shutdowns fine. I tried several time. I will update in case of any news ok?

 

[ozgurerdogan]

It is happening again. I do not know how to reproduce it. It just stuck at shutdown. And only stop stop command can help.

 

[ozgurerdogan]

Any help about this. This became a serious issue. I have to restart node everytime.
What I see in logs is:
unable to connect to VM 102 qmp socket - timeout after 31 retries

 

[rahul1985joshi]

Just reboot/shutdown base node on which this VM resides & it will automatically fix this issue.

Any help about this. This became a serious issue. I have to restart node everytime.
What I see in logs is:
unable to connect to VM 102 qmp socket - timeout after 31 retries

 

[rahul1985joshi]

Just reboot/shutdown base node on which this VM resides & it will automatically fix this issue.

Yes there is no permanent fix for this issue till now in 5 version..only reboot will fix this issue.

 

[ozgurerdogan]

it was 4 years ago but thanks.

 

[speatzle_]

I also just ran into this issue on a Cluster...

 

[cliffpercent]

Necro, here as well. 3-node cluster. All nodes are on the same versions. VM storage is from NFS mount from another server, with ZFS.

The issue occurred only on node 1. Only elevated stopping worked. When restarting, a VM hit the state of no reaction (CPU, mem usage stats load longer than normal, don't change; no response on serial console; doesn't respond to queries from external network (ping, ssh, web)) again, only an hour later — stopping and starting the VM should exclude the possibility of the VM being of older version (even though migrated from older to older, upgraded node, migrated to only newer).

Moving the first troubled VM to node 2 seemed to resolve the issue. Later, many other VMs died the same way. Decided to move everything off of node 1, and now running memtest.

The RAM on the nodes got recently doubled, that seems to be the cause here.

proxmox-ve: 6.3-1 (running kernel: 5.4.98-1-pve)
pve-manager: 6.3-4 (running version: 6.3-4/0a38c56f)
pve-kernel-5.4: 6.3-5
pve-kernel-helper: 6.3-5
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.8-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-5
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-2
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve1

 

[aljaxus]

Similar situation here; 2-node cluster; VMs are just randomly getting stuck;

I have updated, but haven't rebooted.

Spoiler: package versions

proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-4 (running version: 6.3-4/0a38c56f)
pve-kernel-5.4: 6.3-6
pve-kernel-helper: 6.3-6
pve-kernel-5.4.101-1-pve: 5.4.101-1
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 15.2.8-pve2
ceph-fuse: 15.2.8-pve2
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.8-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-5
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-2
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve2

 

[sand-max]

Someboby solved that?
I got same problem. I updated node to latest version from 6.2-10

@pvenode1:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.103-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-7
pve-kernel-helper: 6.3-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-4.15: 5.4-9
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-6
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-3
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-8
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve2

And in about ~10 hours got messages.

Mar 17 01:52:19 pvenode1 pvestatd[3563]: VM 808 qmp command failed - VM 808 qmp command 'query-proxmox-support' failed - unable to connect to VM 808 qmp socket - timeout after 31 retries
Mar 17 01:52:22 pvenode1 pvestatd[3563]: VM 304 qmp command failed - VM 304 qmp command 'query-proxmox-support' failed - unable to connect to VM 304 qmp socket - timeout after 31 retries
Mar 17 01:52:25 pvenode1 pvestatd[3563]: VM 601 qmp command failed - VM 601 qmp command 'query-proxmox-support' failed - unable to connect to VM 601 qmp socket - timeout after 31 retries
Mar 17 01:52:28 pvenode1 pvestatd[3563]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 31 retries
Mar 17 01:52:31 pvenode1 pvestatd[3563]: VM 1236 qmp command failed - VM 1236 qmp command 'query-proxmox-support' failed - unable to connect to VM 1236 qmp socket - timeout after 31 retries
Mar 17 01:52:31 pvenode1 pvestatd[3563]: status update time (30.762 seconds)

I have rebooted node for fix that .

P.S. May be it happened because of VM was running on old version? I migrated to runnnig node all VM and back after to new updated node

 

[Baader-IT]

We do have the same Problem with pve Version 6.3.3

Mar 25 16:31:19 NODENAME pvedaemon[19173]: VMNAME qmp command failed - VMNAME qmp command 'query-proxmox-support' failed - got timeout
Mar 25 16:31:19 NODENAME pvestatd[1942]: VMNAME qmp command failed - VMNAME qmp command 'query-proxmox-support' failed - unable to connect to VMNAME qmp socket - timeout after 31 retries

Unsere VM ist bereits neu auf dem Host gestartet worden.
Problem "immer mal wieder" da.
Auch innerhalb der VM können wir Beeinträchtigungen feststellen:
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s!

Gibt es bereits erste Ideen oder Fix'es für das Problem?

 

[Dominic]

Hi,

please make sure to upgrade to the latest version of Proxmox VE and then stop and start again (or migrate) the affected VMs.

The output of

pveversion -v | grep -E "proxmox-ve|kvm|backup-client"

should contain pve-qemu-kvm 5.2.0-4

For more information, please see here https://forum.proxmox.com/threads/all-vms-locking-up-after-latest-pve-update.85397/page-4

 

[swoop]

Hi,

i know this thread is half a year old, but i have the same problem.
after i updated my Proxmox 7 to

proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
proxmox-backup-client: 2.1.2-1
pve-qemu-kvm: 6.1.0-3

Now i have the same Problem on mein single Proxmox-Server. On this Server, the only Windows VM is having the problem, all Linux VMs or LXCs are running fine.

i now restarted the host again, now i am observing.

SG

 

[CAHEKs]

Hi,

i know this thread is half a year old, but i have the same problem.
after i updated my Proxmox 7 to

proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
proxmox-backup-client: 2.1.2-1
pve-qemu-kvm: 6.1.0-3

Now i have the same Problem on mein single Proxmox-Server. On this Server, the only Windows VM is having the problem, all Linux VMs or LXCs are running fine.

i now restarted the host again, now i am observing.

SG

I have same problem with one windows VM, migrate from lvm to other storage (ceph), temporary solve problem.. version 7 have much bugs

 

[swoop]

Good Morning,

my only temporary solution is to STOP the VM and restart it again.
Reset or shutdown is not wroking.
But after about 30 minutes and 90 minutes the VM stuck again. So it's not a sulution.

But on my 4Node Cluster with Ceph, where i updated and restarted half of the nodes, until now is working good.

regards

 

[fiona]

Hi,
@swoop @CAHEKs do you have a VM disk attached via SATA controller? If yes, please try to edit the disk and set Async IO to native (in Advanced). Other workarounds should be using an older/newer kernel (5.11 or 5.15) or different controller for the disk.