Dirty Cluster!

[Nhoague]

Hello PM community!


I fear I have created quite a doozy for my cluster.


For months, cluster was operating just fine. I have high end switches meshed between dedicated interfaces for the LAN and SAN of my virtual cluster.


The other day I went to remove a node from the cluster. I followed the basic steps of pvecm delnode "node name"


The mode was removed from the cluster. Now comes the problem. I didn't shut off the node prior to removing it from the cluster and now my cluster is dead! All my guests are still running and each server individually can manage their own guests but, I have no visibility to other hosts.


First question: is this fixable? Is there some magic fix cluster command? I understand I probably have a dirty cluster since the node I removed should have been powered down first. But there has to be a simpler way to bring it back online?


Second question: if not. If I am in situation where I have to rebuild, how do I save all my storage information? I know I can save my qemu-server info and rebuild the guests pretty easily. But I have almost 50 iscsi storage points. How to recreate that?


I'm assuming if it's a dirty cluster I won't be able to reuse the cluster info and rebuild a server and join it to an existing host. Any help is appreciated!

 

[udo]

Hi,
normaly that should not be an big issue.

What is the output of following command on all cluster hosts?

pvecm nodes
pvecm status

It's always an good idea to take an backup from /etc/pve, like "tar cvf /root/etc_pve.tar /etc/pve".

Udo

 

[Nhoague]

Here is my output... no good at all!

root@sunfire:~# pvecm nodes

Node Sts Inc Joined Name

1 M 27208 2016-03-07 15:46:24 sunfire

2 X 27212 banshee

3 X 27212 magneto

4 X 27212 cyclops

5 X 27212 rogue

6 X 27212 nightcrawler

7 X 27260 pve101

root@sunfire:~# pvecm status

Version: 6.2.0

Config Version: 52

Cluster Name: dev

Cluster Id: 720

Cluster Member: Yes

Cluster Generation: 27352

Membership state: Cluster-Member

Nodes: 1

Expected votes: 6

Total votes: 1

Node votes: 1

Quorum: 4 Activity blocked

Active subsystems: 3

Flags:

Ports Bound: 0

Node name: sunfire

Node ID: 1

Multicast addresses: 239.192.2.210

Node addresses: 192.168.80.100

root@sunfire:~#


--- I do take regular backups of the /etc/pve directory, so if I have to rebuild the guests its not major a PITA, while I'd like to avoid it if possible! Thanks!

 

[udo]

Hi,
the question is, if you see on all nodes the same - this is why I asked for the output from all cluster mebmber.

Try to restart on the nodes (can be done with running VMs) pve-proxy:

service pve-cluster restart
service pvedaemon restart

Any changes?

Udo

 

[Nhoague]

Sorry for the delay, I ran this on two different hosts and still have this result.

root@sunfire:~# service pve-cluster restart

Restarting pve cluster filesystem: pve-cluster.

root@sunfire:~# service pvedaemon restart

Restarting PVE Daemon: pvedaemon.

root@sunfire:~# pvecm nodes

Node Sts Inc Joined Name

1 M 27208 2016-03-07 15:46:24 sunfire

2 X 27212 banshee

3 X 27212 magneto

4 X 27212 cyclops

5 X 27212 rogue

6 X 27212 nightcrawler

7 X 27260 pve101

root@sunfire:~# pvecm status

Version: 6.2.0

Config Version: 52

Cluster Name: dev

Cluster Id: 720

Cluster Member: Yes

Cluster Generation: 27352

Membership state: Cluster-Member

Nodes: 1

Expected votes: 6

Total votes: 1

Node votes: 1

Quorum: 4 Activity blocked

Active subsystems: 3

Flags:

Ports Bound: 0

Node name: sunfire

Node ID: 1

Multicast addresses: 239.192.2.210

Node addresses: 192.168.80.100

root@sunfire:~#

---The Quorum blocked has me concerned. What would cause that? I've never seen that before and pretty sure thats my problem.