HA with 3 nodes (2 on one location

[wirdo02]

Hello,

We're currently running a 3 node HA cluster where one (main) server is located in the datacenter and two (backup and heartbeat) servers are located at our office. Node 3 doesn't do anything VM wise, only acts as a heartbeat to reach the 3 node requirement.
Both are connected to a cluster using a VPN tunnel.

Everything worked great until our office went offline which contains 2 of the 3 nodes.
The VM's on the main server stopped when that happend.

Is there an option to keep the VM's active when the 2 other nodes go down? I've been reading about the active tie breaker function but they is above my head to be honest.

 

[danielb]

You'd need the qorum in the DC. In a HA setup, it's not only expected, but also a strong requirement to kill the nodes which do not have the qorum. Can't you move the heartbeat in the DC near the main server ?

 

[tim]

In general you can't loose more than one node in a 3 node cluster, besides that I wouldn't recommend you to have your cluster split to different location. This is a bad idea, because of the latency which could lead to some unexpected side effects especially when you use HA.

Anyway, keeping 2 nodes in the DC would probably increase the fault tolerance, because a power loss in a DC is less likely.

 

[wirdo02]

In general you can't loose more than one node in a 3 node cluster, besides that I wouldn't recommend you to have your cluster split to different location. This is a bad idea, because of the latency which could lead to some unexpected side effects especially when you use HA.

Anyway, keeping 2 nodes in the DC would probably increase the fault tolerance, because a power loss in a DC is less likely.

Hi Tim,

Thanks for your response. I'm aware of the risks of the cluster split but there isn't really an alternative (at least that I'm aware of)
We want to exclude the network factor in the DC location, having all nodes in the same DC would only cover hardware related incidents.

Latency is quite good over the VPN tunnel all-though we don't have any influence on the network between it of course.

I know it's not the normal way but would it be an option to run a VM on the main node with proxmox that fuctions as an cluster node so we have 2 nodes present in DC and 2 on the other location.

 

[danielb]

Having 2 - 2 nodes would be even worst. Keeping appart the fact that this is cheating because you have only one real node (and when rebooting it, you loose 2 votes for example), if you have 2 - 2 nodes, no side would be able to get the qorum, and all the nodes would self fence

 

[tim]

A even number never increases redundancy as you can easily see with your 4 node example, if you loose more than one node you will still loose quorum. Your workaround isn't gonna work as already explained, if you really need a cluster just keep the nodes at one location.

If you don't need a cluster, which at least seems like it reading your first post, you could just use one server and replicate your VMs to another site. A possible scenario would be to use the main server in the DC and replicate the vms to your office site where your backup servers reside.
In case of a disaster you could easily start your vms on the backup server.

https://pve.proxmox.com/wiki/PVE-zsync

 

[wirdo02]

A even number never increases redundancy as you can easily see with your 4 node example, if you loose more than one node you will still loose quorum. Your workaround isn't gonna work as already explained, if you really need a cluster just keep the nodes at one location.

If you don't need a cluster, which at least seems like it reading your first post, you could just use one server and replicate your VMs to another site. A possible scenario would be to use the main server in the DC and replicate the vms to your office site where your backup servers reside.
In case of a disaster you could easily start your vms on the backup server.

https://pve.proxmox.com/wiki/PVE-zsync

Thanks, didn't know this existed, this way I indeed don't need a cluster.
Been reading through the forum to see if there is away to remove the cluster as a whole.

The main node in DC is the one that is important, I can just reinstall the other one here in the office without any issues if needed.
Can I just follow the guide here? https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node including the "Separate A Node Without Reinstalling" part on the DC node? I don't have shared storage.

 

[wirdo02]

Hi Tim.

All is good. The main server is out of the cluster and operating solo.

I do have a couple questions regarding zsync tho.

Is it possible to overwrite the disk file from a container (stopped state) on the destination server?

Example: VM100 on server A is being backuped to VM100 on server B. The disk file on server B is overwritten every time by zsync.
Sync will fail when file is in use since it's locked.