WARNING: unable to connect to VM 100 socket - timeout after 31 retries

ozgurerdogan

Renowned Member
May 2, 2010
604
5
83
Bursa, Turkey, Turkey
When I want to restart or shutdown wm it stuck at the end of logging of screen. And I have to use Stop command.

Here is logs. Do you have any idea?

Code:
ov 17 22:40:08 vztl3 pvedaemon[2671]: WARNING: unable to connect to VM 100 socket - timeout after 31 retries
Nov 17 22:40:11 vztl3 pvedaemon[7543]: WARNING: VM still running - terminating now with SIGTERM
Nov 17 22:40:12 vztl3 pvestatd[2692]: WARNING: unable to connect to VM 100 socket - timeout after 31 retries
Nov 17 22:40:15 vztl3 pvedaemon[2671]: WARNING: unable to connect to VM 100 socket - timeout after 31 retries
Nov 17 22:40:17 vztl3 pvedaemon[7576]: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout
Nov  17 22:40:17 vztl3 pvedaemon[7234]: <root@pam> end task  UPID:vztl3:00001D98:0004DDD5:546A5D27:qmstop:100:root@pam: unexpected  status
Nov 17 22:40:21 vztl3 pvedaemon[7543]: WARNING: VM still running - terminating now with SIGKILL
Nov 17 22:40:21 vztl3 kernel: vmbr0: port 5(tap100i0) entering disabled state
Nov 17 22:40:21 vztl3 kernel: vmbr0: port 5(tap100i0) entering disabled state
Nov 17 22:40:21 vztl3 pvestatd[2692]: WARNING: unable to connect to VM 100 socket - Connection refused
Nov 17 22:40:21 vztl3 pvedaemon[2671]: <root@pam> end task UPID:vztl3:00001CF2:0004B697:546A5CC2:vncproxy:100:root@pam: OK
Nov 17 22:40:22 vztl3 pvedaemon[2670]: <root@pam> end task UPID:vztl3:00001D77:0004D3BB:546A5D0D:qmstop:100:root@pam: OK
Nov 17 22:40:22 vztl3 pvedaemon[7613]: starting vnc proxy UPID:vztl3:00001DBD:0004E403:546A5D36:vncproxy:100:root@pam:
Nov  17 22:40:22 vztl3 pvedaemon[7234]: <root@pam> starting task  UPID:vztl3:00001DBD:0004E403:546A5D36:vncproxy:100:root@pam:
Nov 17  22:40:23 vztl3 ntpd[2360]: Deleting interface #11 tap100i0,  fe80::e8a6:fdff:fe98:22f1#123, interface stats: received=0, sent=0,  dropped=0, active_time=637 secs
Nov 17 22:40:23 vztl3 ntpd[2360]: peers refreshed
Nov 17 22:40:23 vztl3 qm[7615]: VM 100 qmp command failed - VM 100 not running
Nov  17 22:40:23 vztl3 pvedaemon[7613]: command '/bin/nc -l -p 5900 -w 10 -c  '/usr/sbin/qm vncproxy 100 2>/dev/null'' failed: exit code 255
Nov  17 22:40:23 vztl3 pvedaemon[7234]: <root@pam> end task  UPID:vztl3:00001DBD:0004E403:546A5D36:vncproxy:100:root@pam: command  '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 100 2>/dev/null''  failed: exit code 255
 
Vm seems to be running fine but not sure if it safe to run so.

Code:
proxmox-ve-2.6.32: 3.3-139 (running kernel: 2.6.32-34-pve)
pve-manager: 3.3-5 (running version: 3.3-5/bfebec03)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-34-pve: 2.6.32-139
pve-kernel-2.6.32-31-pve: 2.6.32-132
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.3-3
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-25
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
 
Just reboot/shutdown base node on which this VM resides & it will automatically fix this issue.
Any help about this. This became a serious issue. I have to restart node everytime.
What I see in logs is:
unable to connect to VM 102 qmp socket - timeout after 31 retries
 
Necro, here as well. 3-node cluster. All nodes are on the same versions. VM storage is from NFS mount from another server, with ZFS.

The issue occurred only on node 1. Only elevated stopping worked. When restarting, a VM hit the state of no reaction (CPU, mem usage stats load longer than normal, don't change; no response on serial console; doesn't respond to queries from external network (ping, ssh, web)) again, only an hour later — stopping and starting the VM should exclude the possibility of the VM being of older version (even though migrated from older to older, upgraded node, migrated to only newer).

Moving the first troubled VM to node 2 seemed to resolve the issue. Later, many other VMs died the same way. Decided to move everything off of node 1, and now running memtest.

The RAM on the nodes got recently doubled, that seems to be the cause here.

Code:
proxmox-ve: 6.3-1 (running kernel: 5.4.98-1-pve)
pve-manager: 6.3-4 (running version: 6.3-4/0a38c56f)
pve-kernel-5.4: 6.3-5
pve-kernel-helper: 6.3-5
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.8-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-5
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-2
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve1
 
Similar situation here; 2-node cluster; VMs are just randomly getting stuck;

I have updated, but haven't rebooted.

Code:
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-4 (running version: 6.3-4/0a38c56f)
pve-kernel-5.4: 6.3-6
pve-kernel-helper: 6.3-6
pve-kernel-5.4.101-1-pve: 5.4.101-1
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 15.2.8-pve2
ceph-fuse: 15.2.8-pve2
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: not correctly installed
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.8-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-5
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-2
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve2
 
Someboby solved that?
I got same problem. I updated node to latest version from 6.2-10

Code:
@pvenode1:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.103-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-7
pve-kernel-helper: 6.3-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-4.15: 5.4-9
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-6
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-3
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-8
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.3-pve2


And in about ~10 hours got messages.

Code:
Mar 17 01:52:19 pvenode1 pvestatd[3563]: VM 808 qmp command failed - VM 808 qmp command 'query-proxmox-support' failed - unable to connect to VM 808 qmp socket - timeout after 31 retries
Mar 17 01:52:22 pvenode1 pvestatd[3563]: VM 304 qmp command failed - VM 304 qmp command 'query-proxmox-support' failed - unable to connect to VM 304 qmp socket - timeout after 31 retries
Mar 17 01:52:25 pvenode1 pvestatd[3563]: VM 601 qmp command failed - VM 601 qmp command 'query-proxmox-support' failed - unable to connect to VM 601 qmp socket - timeout after 31 retries
Mar 17 01:52:28 pvenode1 pvestatd[3563]: VM 122 qmp command failed - VM 122 qmp command 'query-proxmox-support' failed - unable to connect to VM 122 qmp socket - timeout after 31 retries
Mar 17 01:52:31 pvenode1 pvestatd[3563]: VM 1236 qmp command failed - VM 1236 qmp command 'query-proxmox-support' failed - unable to connect to VM 1236 qmp socket - timeout after 31 retries
Mar 17 01:52:31 pvenode1 pvestatd[3563]: status update time (30.762 seconds)


I have rebooted node for fix that .

P.S. May be it happened because of VM was running on old version? I migrated to runnnig node all VM and back after to new updated node
 
Last edited:
We do have the same Problem with pve Version 6.3.3

Mar 25 16:31:19 NODENAME pvedaemon[19173]: VMNAME qmp command failed - VMNAME qmp command 'query-proxmox-support' failed - got timeout
Mar 25 16:31:19 NODENAME pvestatd[1942]: VMNAME qmp command failed - VMNAME qmp command 'query-proxmox-support' failed - unable to connect to VMNAME qmp socket - timeout after 31 retries

Unsere VM ist bereits neu auf dem Host gestartet worden.
Problem "immer mal wieder" da.
Auch innerhalb der VM können wir Beeinträchtigungen feststellen:
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s!

Gibt es bereits erste Ideen oder Fix'es für das Problem?
 
Last edited:
Last edited:
Hi,

i know this thread is half a year old, but i have the same problem.
after i updated my Proxmox 7 to
Code:
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
proxmox-backup-client: 2.1.2-1
pve-qemu-kvm: 6.1.0-3

Now i have the same Problem on mein single Proxmox-Server. On this Server, the only Windows VM is having the problem, all Linux VMs or LXCs are running fine.

i now restarted the host again, now i am observing.

SG
 
Hi,

i know this thread is half a year old, but i have the same problem.
after i updated my Proxmox 7 to
Code:
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
proxmox-backup-client: 2.1.2-1
pve-qemu-kvm: 6.1.0-3

Now i have the same Problem on mein single Proxmox-Server. On this Server, the only Windows VM is having the problem, all Linux VMs or LXCs are running fine.

i now restarted the host again, now i am observing.

SG

I have same problem with one windows VM, migrate from lvm to other storage (ceph), temporary solve problem.. version 7 have much bugs :(
 
Good Morning,

my only temporary solution is to STOP the VM and restart it again.
Reset or shutdown is not wroking.
But after about 30 minutes and 90 minutes the VM stuck again. So it's not a sulution.

But on my 4Node Cluster with Ceph, where i updated and restarted half of the nodes, until now is working good.

regards
 
Last edited:
Hi,
@swoop @CAHEKs do you have a VM disk attached via SATA controller? If yes, please try to edit the disk and set Async IO to native (in Advanced). Other workarounds should be using an older/newer kernel (5.11 or 5.15) or different controller for the disk.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!