Correctly separate nodes

[Aleksej]

Hello.

I have faced a problem.
I have migrated nodes to different places. And for now i do not need them in a cluster (as i have latency between 2.3 - 4msec it causes troubles)

After reboot, every node wants "pvecm expected 1" to start all VMs.

How can i fix it?
For now i do need any other nodes, but i don't want to break cluster (in future they will work together again)

Thanks

 

[shanreich]

There are ways of separating cluster nodes described in the documentation [1], but it is preferable to reinstall all nodes that got removed from a cluster. There are also ways to separate without reinstalling described in [1], but this can be a bit tricky, and it would also involve separating storage for the time being. Separating a node without reinstalling is therefore highly discouraged and we do not make any guarantees for this working.

Setting the expected Quorum to 1 also has its own problems, since it can lead to a split brain and is also not recommended, especially for longer durations of time. Given enough time, you will run into problems.

The best solution might be to just backup all your VMs to an external storage, then separate and reinstall the nodes. Be aware that in this case you should also make sure that the different nodes are not sharing the same storage. This can lead to problems (especially when VMIDs are reused). Then when you are ready for your nodes to join a cluster again, rebuild the cluster from scratch and restore all your VMs from a backup.

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

 

[Aleksej]

Thanks for your answer.
Actually as i said, i don't need to separate node at all.
Simply for some time it has to work separately and then return back to normal work.

Somehow one node working standalone, but other ones don't want without every time setting "pvecm expected 1".
Maybe if i manually change votes in corosync.conf it will work?

 

[shanreich]

There's not really a way to make nodes that are in a cluster behave like they are not in a cluster. The only way to achieve that would be to separate them according to [1]. Then afterwards there should be no reason why you cannot rejoin the node to the cluster again. The only thing you need to be careful is that there are no duplicate VMIDs on either node, otherwise you will run into problems.

Changing the votes in corosync.conf would not really change anything, since there is no vote configuration that would satisfy your requirements. At least I personally cannot think of one.

[1] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_separate_node_without_reinstall

 

[Aleksej]

Earlier i tried to rejoin node.
Faced with VMs loss.
Of course i can archive VM config files and restore them but i thought that there is some easier way.

And also one of nodes are down and all work correctly, but why?
Cluster now somehow waits for 3 nodes, but waited for 4...

 

[shanreich]

Earlier i tried to rejoin node.
Faced with VMs loss.
Of course i can archive VM config files and restore them but i thought that there is some easier way.

Sadly, there is not an easier way for this.

And also one of nodes are down and all work correctly, but why?
Cluster now somehow waits for 3 nodes, but waited for 4...

Depends on how many nodes are in your cluster, if you had 4 nodes then 3 nodes is sufficient for a quorum. As long as there are >50% of nodes working, the cluster will continue working without any problems. Did you delete the node from the cluster? Then it would make sense that it is only waiting for 3 nodes.

 

[Aleksej]

No, i didn't delete node. So now i understood.
I have 3 of 5 nodes left and all worked. And now when all of them are working in different places - they all want at least 3 (in my case each other).

So. On one node i changed corosync.conf and manually wrote on all other nodes "votes:0".
On other node i write script in /etc/rc3.d/pvestart with "/usr/sbin/pvecm expected 1"

In all cases it works.