Quorum activity blocked after adding additional nodes

[verisys]

I have a cluster currently running Proxmox 4.0. It was working fine until I attempted to add 3 additional nodes. After adding the nodes, the cluster stopped running and quorum activity is now blocked. I did receive errors on the new nodes about being unable to write to /etc/pve/nodes/HOSTNAME when connecting them. Also, /etc/pve/ is read only now.

I think I might have caused this when I used Saltstack to deploy my signed SSL certificate to the new nodes prior to joining the cluster. The new nodes are have been taken down now.

What I am wondering now is if there is a way I can get the original cluster nodes to start talking again and remove any trace of the new nodes. I am including the the output of "pvecm status" and pvecm nodes."

root@LV-HV-04:/etc/pve/nodes# pvecm status
Quorum information
------------------
Date: Wed Dec 16 15:28:03 2015
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 900
Quorate: No

Votequorum information
----------------------
Expected votes: 6
Highest expected: 6
Total votes: 3
Quorum: 4 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.3.8.10 (local)
0x00000002 1 10.3.8.11
0x00000003 1 10.3.8.12

root@LV-HV-04:/etc/pve/nodes# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
1 1 LV-HV-04 (local)
2 1 LV-HV-05
3 1 LV-HV-06

root@LV-HV-04:/etc/pve# ll
total 9
drwxr-xr-x 2 root www-data 0 Jan 1 1970 .
drwxr-xr-x 107 root root 4096 Dec 15 20:47 ..
-r--r----- 1 root www-data 451 Nov 25 12:28 authkey.pub
-r--r----- 1 root www-data 9242 Jan 1 1970 .clusterlog
-r--r----- 1 root www-data 706 Dec 15 19:54 corosync.conf
-r--r----- 1 root www-data 16 Nov 25 12:23 datacenter.cfg
-rw-r----- 1 root www-data 2 Jan 1 1970 .debug
-r--r----- 1 root www-data 226 Nov 25 22:03 domains.cfg
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 local -> nodes/LV-HV-04
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 lxc -> nodes/LV-HV-04/lxc
-r--r----- 1 root www-data 426 Jan 1 1970 .members
dr-xr-xr-x 2 root www-data 0 Nov 25 12:28 nodes
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 openvz -> nodes/LV-HV-04/openvz
dr-x------ 2 root www-data 0 Nov 25 12:28 priv
-r--r----- 1 root www-data 2664 Nov 25 21:58 pve-root-ca.pem
-r--r----- 1 root www-data 1675 Nov 25 12:28 pve-www.key
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 qemu-server -> nodes/LV-HV-04/qemu-server
-r--r----- 1 root www-data 3737 Jan 1 1970 .rrd
-r--r----- 1 root www-data 201 Dec 1 19:58 storage.cfg
-r--r----- 1 root www-data 408 Nov 25 22:06 user.cfg
-r--r----- 1 root www-data 429 Jan 1 1970 .version
-r--r----- 1 root www-data 1011 Jan 1 1970 .vmlist
-r--r----- 1 root www-data 119 Nov 25 12:28 vzdump.cron

 

[verisys]

I think I got it figured and taken care of. After searching through this forum, I found that running the following command "pvecm expect 3" would the expected quorum count to 3 and the clusterfs because working again. I did have to go back through and remove references to the failed nodes in the corosync.conf, nodes folder, authorized_keys file, and known_hosts file.

Things seem to be back to normal and I was able to add the additional nodes after reinstalling them.

 

[verisys]

I must be blind, but where to do I change this to show that it is solved?

 

[promok]

I must be blind, but where to do I change this to show that it is solved?

I have recently installed node1 and enabled a new cluster on it. One local LAN, one network interface on each node. Firewall to "no". Now I try to add the brand new second node to this new cluster, and it fails, breaks the web interface on node2, and kills all the containers on node1. The containers cannot be started.
pvecm nodes shows only one local node on each node.
Quorum: 2 Activity blocked

I think, this problem is not solved yet.