Shutting 2 of 3 cluster servers makes last one read only

P

Proximate

Member
Feb 13, 2022
219
13
23
65
I am testing proxmox and set up 3 servers in a cluster.
I installed a few vms on the first server.
I didn't need all three running at one point so I shut down two of the servers.
Now the first server left on seems to have shut down all of the vms and has made everything in /etc/pve/nodes/ read only.

Is this a normal behavior when using a cluster mode?
 
  • Like
Reactions: Proximate
Hmm, what I was hoping for is to have a cluster where all but one could be done and still function.
I guess that makes sense. A cluster means quorum otherwise, it's a redundant setup.

Thanks for the good lead.
 
Last edited:
Proximate said:
Hmm, what I was hoping for is to have a cluster where all but one could be done and still function.
Click to expand...
The problem is your definition of function. Everything that PVE is running will still be running and working, but you cannot change anything. (Therefore STONITH is very important in such a setup to not run into a split brain with a shared storage)
 
>(Therefore STONITH is very important in such a setup to not run into a split brain with a shared storage)

I'll have to look at this up. Very new to all this, just trying it out for the first time :).
It makes sense that everything would stay up and running and changes not allowed but not sure why the vms all went off. As soon as I fired a second server up, all went back to normal but the vms also came back on. That part has me unsure so probably missing something in the settings.
 
Were the VMs running on one of the failed nodes? If they were - did you configure them for HA ( https://pve.proxmox.com/wiki/High_Availability_Cluster )? If not, they would not be automatically recovered on failure.
Are the VMs running on shared storage where they can be accessed by all nodes? If not then there is no automatic recovery possible, only manual copy in advance of failure.

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
They were installed on the first server and on that servers storage with no HA setup just yet.
When the other two servers were shut down, the vms on the first server got shut down until I restarted one of the servers to have two running.

I could run them all off of shared storage but just trying to get some hands on time with this before getting too complex.
 
Nope, didn't happen this time. I must have missed something or some condition so I'll keep an eye on this and report back if it happens again.

Thanks for your help.
 
One way it could have happened is if first server was rebooted or rebooted itself due to loss of quorum:

On node startup, the pve-guests service is started and waits for quorum. Once quorate, it starts all guests which have the onboot flag set.
When you turn on nodes, or when power comes back after power failure, it is likely that some nodes will boot faster than others. Please keep in mind that guest startup is delayed until you reach quorum.

https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_cold_start

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Proximate
You must log in or register to reply here.
Top Bottom