HA accross different locations

[sannsio]

Hi everybody,

we are running a 7 Node Proxmox Cluster. 3 Nodes are used for Ceph only.

We are considering to rent a second rack at another data center and run the same set up for the case of a disaster in our primary data center.

Now it came to my mind that Proxmox provides High Availability out the box. Has anybody ever tried to run a cluster within the context of two different data centers? As I understand we would need Layer 2 connections between the data centers. And ss we run Ceph within a dedicated 10 Gbit network, we would even need two Layer 2 connections, one 1 Gig and one 10 Gig.

Has anyone experience with such a set up? Is it possible at all? Or is the latency maybe a show stopper?

Any tip or hint would be nice! It's hard to find any information for such a use case.Thanks!
Sandra

 

[Klaus Steinberger]

There are some show stoppers for this:

-> Latency will bog down Ceph severly

-> How do you want to handle communiction failure? you will loose too much OSD-Hosts once.
-> How would you avoid a split brain situation on communication failure?

Maybe better to replicate the vm's in some way to the second data center

 

[LnxBil]

Latency will be a problem for PVE too. Do you already have the connection and can you provide roundtrip times?

 

[sannsio]

Thanks guys,

I don't have the connection yet.

The Ceph docs say:

"If your data centers have dedicated bandwidth and low latency, you can distribute your cluster across data centers easily. If you use a WAN over the Internet, you may need to configure Ceph to ensure effective peering, heartbeat acknowledgement and writes to ensure the cluster performs well with additional WAN latency."

So, everything depends on the quality of the connection I guess, which is hard to predict.

Sandra