Unable to connect to VM qmp socket

decibel83

Renowned Member
Oct 15, 2008
210
1
83
Hi.

I have a four node cluster and every node has the same hardware configuration and setup.

Suddenly I had two KVM virtual machines completely hanged.
The first machine (on node1) was on 100+ of load average with hundreds of processes in D state but very low CPU usage, so I was thinking about I/O problems.
The second machine (on node2) is not reachable on networks and if I try to access to its console I get the "Failed to connect to server (code: 1006)" on the NOVNC console and "Sep 11 09:17:13 node2 pvedaemon[929876]: unable to connect to VM 208 qmp socket - timeout after 31 retries" on the syslog.

Both machines are running on the local storage (2 x 960 SSD in RAID1 on "Symbios Logic MegaRAID SAS-3 3108").

This is my pveversion -v:

Code:
root@node2:~# pveversion -vproxmox-ve-2.6.32: 3.4-150 (running kernel: 2.6.32-37-pve)
pve-manager: 3.4-3 (running version: 3.4-3/2fc72fee)
pve-kernel-2.6.32-37-pve: 2.6.32-150
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.4-3
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-32
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

And this is my pveperf:

Code:
root@node2:~# pveperf
CPU BOGOMIPS:      183995.80
REGEX/SECOND:      1819374
HD SIZE:           18.21 GB (/dev/mapper/system-root)
BUFFERED READS:    431.20 MB/sec
AVERAGE SEEK TIME: 0.09 ms
FSYNCS/SECOND:     5030.94
DNS EXT:           36.27 ms
DNS INT:           21.13 ms (dc.ems.network)

Could you help me please?

Thank you very much!
Bye