Quorum between two locations

[Egert143]

Hello

I could really use some info on cluster setup. i am trying to setup 6 node cluster where 3 nodes will be on location A and other 3 in location B. They have 10G fiber connection. My plan is to have first 3 nodes run vm-s and other 3 serve as backup when location A is offline. When i tested this setup in lab i ran into problem as soon as i disconnected connection between location A and location B, half of cluster said "no quorum". i though that i needed uneven number of nodes for cluster to survive and was expecting split brain situation. Is there some kind of failsafe for such scenario? and what could be workaround to keep remaining site operational in event of connection error between sites ?

Egert

 

[dcsapak]

note that corosync is very latency sensitive, so even if you have a fast link between sites, latency can be higher and thus make the cluster more unstable.
if you really want it though, you probably want a external corosync vote (qdevice) https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

 

[Egert143]

Latency is 2-3ms is that too much for corosync ?

Also in event when connection between sites is broken, how can i force one side to still work ?

 

[Ney Barrozo]

Hi.

I have a similar problem. We will have a four node Proxmox/Ceph hyper converged cluster installed in two different locations (2 nodes each), connected by low latency 10Gbpb links. The clients will be in a third location, where I plan to have a QDEVICE for Proxmox quorum, in case of split brain. Is that enough for Ceph? Or should I have some other device to complete the quorum for Ceph in case of split brain?
Thanks for any help.

Ney

 

[sidereus]

Even number of Ceph monitors, half of it in an every of 2 locations, not looking as a very good idea, because if link on a Ceph network will be broken, the entire cluster will stop working. Better make 2+1 ceph monitors. In this case the location 1 will survive, when Ceph network between locations will be damaged. Taking this into account, I manually reduced number of my Ceph monitors from 4 to 3. It is not a pleasant task.