Ceph mirroring between datacenters

[einsibjani]

I'm setting up a new Proxmox environment, in two datacenters, with three nodes in each datacenter. The new setup
will use ceph as shared storage. Our old cluster is a mixed bag of servers spread across both datacenters with no
shared storage.

We have a redundant 10G connections between the datacenters and the latency is tolerable, so we could run it
as a single cluster, but I dont want a 6 node cluster with a very real possibility of a split brain situation.

That's why my plan A is to deploy two clusters, with RBD mirroring between the clusters, but I want to use image
mode, since I might want some images running on site B most of the time, while I want some running
on site A. If I choose to go this way, do I create the VM on both clusters, with the backup shutoff and start it after I promote
the image (and demote it on the master)?

Is anyone running Proxmox in a similar setup? Would you recommend it?

Edit: Fix typos

 

[Alwin]

If I choose this way, do I create the VM on both clusters, with the backup shutoff and start it after I promote
the image (and demote it on the master)?

Since the vmid.conf is not shared between those clusters, it needs to exist to be able to start the VM/CT.

 

[adamb]

We run a stretched ceph cluster over two datacenters. We have 40GB dark fiber with about .7 ms latency.

Overall it runs well. We avoid split brains by have 1 extra monitor at the primary site.

Rbd-mirror with journal has some serious write overhead. Personally I wouldn't even consider it on mechanical drives. I believe there is a new mirror mode coming out in Octopus that uses snapshot based replication but its pretty new.

 

[einsibjani]

Overall it runs well. We avoid split brains by have 1 extra monitor at the primary site.

Rbd-mirror with journal has some serious write overhead. Personally I wouldn't even consider it on mechanical drives. I believe there is a new mirror mode coming out in Octopus that uses snapshot based replication but its pretty new.

Interesting. What is the plan if the primary site goes down (fire, flood, nuclear meltdown etc.)? Could you start a fourth monitor on the secondary site and get it up and running?

Regarding write overhead with journaling, all the drives in the ceph cluster(s) are NVM.e SSD drives.

 

[Alwin]

Interesting. What is the plan if the primary site goes down (fire, flood, nuclear meltdown etc.)? Could you start a fourth monitor on the secondary site and get it up and running?

No. Since the cluster will not be quorate. A third location is needed for this. There are also plans to get the current Ceph MON communication for such a scenario.
https://fosdem.org/2020/schedule/event/sds_ceph_stretch_clusters/

 

[adamb]

No. Since the cluster will not be quorate. A third location is needed for this. There are also plans to get the current Ceph MON communication for such a scenario.
https://fosdem.org/2020/schedule/event/sds_ceph_stretch_clusters/

The monitor is a VM part of a small cluster which we have the ability to run at either site. In the case of a burn down at our main site, we simply bring it up at the 2nd site. It works without any issues.

 

[einsibjani]

The monitor is a VM part of a small cluster which we have the ability to run at either site. In the case of a burn down at our main site, we simply bring it up at the 2nd site. It works without any issues.

Kind of what I was thinking. Have you ever had a shutdown of the primary site, either for real or as a drill?

 

[adamb]

Kind of what I was thinking. Have you ever had a shutdown of the primary site, either for real or as a drill?

Yep we test site fail over regularly.

 

[Alwin]

The monitor is a VM part of a small cluster which we have the ability to run at either site. In the case of a burn down at our main site, we simply bring it up at the 2nd site. It works without any issues.

I didn't mean the Ceph MON. The quorum for corosync will be missing and then you won't be able to start the VM (if it resides on the same cluster).

 

[adamb]

I didn't mean the Ceph MON. The quorum for corosync will be missing and then you won't be able to start the VM (if it resides on the same cluster).

It resides on a separate cluster.