Routed stretched cluster

[timo.reimann]

We are currently setting up two separate PVE clusters. Both are HCI clusters running Ceph. The servers are distributed across two colocation facilities in the same city, connected via a low-latency data centre interconnect.

One cluster consists of 12 nodes in each data centre, while the second cluster consists of 3+3 nodes.

We also have access to a third data centre where we would like to deploy the tiebreaker nodes. As I understand it from the Corosync and Ceph documentation, connecting the tiebreaker nodes via routed networks should be possible.

However, the Proxmox configuration dialogs appear to require that the Corosync and Ceph IP addresses reside within the same Layer 2 network.

This raises several questions:

1. Is a stretched Layer 2 network across all three sites actually required?
2. Has anyone successfully deployed tiebreaker nodes over routed connections?
3. Are there any plans or recommendations from Proxmox development team to officially support or simplify this setup?

 

[ce3rd]

hi, in https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

Unlike corosync itself, a QDevice connects to the cluster over TCP/IP. The daemon can also run outside the LAN of the cluster and isn’t limited to the low latencies requirements of corosync.

 

[Khensu]

Also please see https://bugzilla.proxmox.com/show_bug.cgi?id=7518
I also used a Routing Setup there and it worked perfectly fine.

 

[timo.reimann]

Thank you very much! So technically it works, but the stretched cluster topology is not officially supported.

One more question on this: Is your Corosync also routed? Are any special settings required for this as well?

Does the Proxmox team have any plans to support such scenarios? I think that clusters reaching a certain size have often been, and will continue to be, designed in such topologies with a tiebreaker node that is only accessible via routing.

And as Proxmox VE increasingly becomes a serious alternative to vSphere, Nutanix and many others, such configurations will become more and more common.