proxmox cluster unstable, won't rejoin because of existing VMs

[System Error Message]

I use proxmox cluster on 8 machines for development and i constantly turn them off and on to save power, change hardware etc and lately it has gotten very unstable that i had to manually remove the nodes from the cluster just to even get a VM to start as nodes were getting stuck in things like no quorum state. I tried to rejoin them but failed because of proxmox rules that nodes with VMs cannot join a cluster. Seeing as i had not changed any VM configs yet would it be possible to force it back together?

As individual nodes i can't move the VMs to another node in order to get the cluster to form.

So far proxmox has been very unstable when clustering, even more so when you shutdown the servers you arent using at the time.

 

[itNGO]

Hi,
while in theory possible to shutdown nodes to save power, I believe it is not the way a cluster in PVE is designed for.
When you have an even node number, you need to have more then 50% of nodes up and running at all time, to keep the cluster stable.

This makes 5 nodes in your 8 node cluster have to run and reach each other. However, when all nodes are up and running after a total shutdown, the cluster also should come up and running within minutes after all nodes have booted and can see each other.

Please post your syslog and cluster config-files....

 

[System Error Message]

Unfortunately that is not what happened. I shutdown the entire cluster, booted it all back up and node 7 would not join the cluster, and could not even rejoin due to the rest of the cluster not having the join information. For a while main cluster thought it wasn't a cluster despite functioning as one. I could not even use the create cluster as it would just show an error saying that the files exist.

Where do i find the cluster config files?

Isn't there a way to create the cluster in a way that allows most of its members to be offline? I will be switching the cluster to openstack soon though, pending some clustered web infrastructure design to be finished. one of the things with using kubernetes is that you dont need over half the servers running, infact a minimum of one can be running but HA only starts when you have at least 3 whereas mysql cluster has no problems with how many data instances you have running at a time as well.

One of the issues around the world is power, and even where i work even though we have our own datacenter they too are concerned about power use, not to mention they pay more than other rates if you consider the usage though only a bit less if you consider the capacity vs combined homes. However i will definitely be having 2 proxmox servers clustered instead of all 8 pending 3 servers to be returned from being loaned to bring it to a total of 11.

Alternatively if i could backup all the VMs that would also be a solution in helping clear the nodes of VMs and reclustering them.