[SOLVED] Cluster with 3 hosts is gone, what now?

[fireon]

Hello,

have an cluster with free nodes. Backup fails and cluster is gone. The machines where never be updated since setup and they are running 119 days (should not be a problem ). We have many vlans configured on bond with LACP Layer 2+3 on 802.3ad.

On every host we tausend of this logs

[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Sep 24 23:24:25 srv-virtu01 pmxcfs[665148]: [status] crit: cpg_send_message failed: 9[/FONT][/COLOR]

Seems to be ok:

Node  Sts   Inc   Joined               Name
   1   M   3940   2015-09-24 23:27:11  srv-virtu03
   2   M   3940   2015-09-24 23:27:11  srv-virtu02
   3   M   3940   2015-09-24 23:27:11  srv-virtu01

Version: 6.2.0
Config Version: 3
Cluster Name: iteascl01
Cluster Id: 54505
Cluster Member: Yes
Cluster Generation: 3940
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2  
Active subsystems: 5
Flags: 
Ports Bound: 0  
Node name: srv-virtu01
Node ID: 3
Multicast addresses: 239.192.212.190 
Node addresses: 10.70.99.9

omping is also fine!!!

I have restarted the services:

cman
pve-cluster

on every node. For some minutes 2 hosts were ok. Then same error.

Thanks.

 

[dietmar]

What is the output of

# pveversion -v

 

[fireon]

I had to fix the problem shortly. So i updated one host with dist-upgrade, after this (without reboot) it was working fine. So the output of the machines are:

proxmox-ve-2.6.32: not correctly installed (running kernel: 2.6.32-34-pve)
pve-manager: 3.4-11 (running version: 3.4-11/6502936f)
pve-kernel-2.6.32-34-pve: 2.6.32-140
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-19
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-11
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

View attachment updatelog.txt

so i think something was restarted turning the update has solved the issue.

Thanks