Proxmox + Ceph across 2 datacenters

[abhijithiaaxin]

Hi all,

We are planning to set up a 4-node Proxmox cluster with Ceph across 2 datacenters.

Our setup will be:

• DC1: Node 1 and Node 2
• DC2: Node 3 and Node 4
• Node 1 and 2 are using one network range.
• Node 3 and 4 are using a different network range.
• Both datacenters are connected through an inter-DC network.

We want to use Ceph for shared storage, along with VM migration and HA.

Our main confusion is about the network design.

If a VM is running on Node 1 in DC1, its VM network is from the DC1 network. If HA moves the VM to Node 3 in DC2, the VM network is different there.

So we are not sure how to design the VM network so that the VM remains accessible after migration.

We also want to understand what will happen if the inter-DC network goes down. We are concerned about Corosync quorum, HA fencing and split-brain.

Could someone please suggest the correct architecture for this type of setup?

Especially:

• How should we handle the VM network between the two datacenters?
• Should the VM network be stretched between both DCs?
• Should we use routing between the two VM networks?
• How should we configure Corosync and Ceph?
• Do we need a QDevice in a third location?
• Is HA across both datacenters recommended in this setup?
• What happens if the inter-DC link fails?

We have not started the implementation yet, so we would like to finalize the network and cluster design first.

If anyone is running a similar Proxmox + Ceph setup across two datacenters, please share your architecture and suggestions.

 

[gurubert]

Is HA across both datacenters recommended in this setup?

Basically you need three locations for automatic HA. Otherwise you will always have issues with split brain situations.

Addendum: If there is no common layer 2 network available for the VM traffic you cannot migrate a VM from one location to the other.

 

[aaron]

As @gurubert already mentioned, you will need one node in a 3rd location.

Our Stretch Cluster guide has more details on how to get it up and running: https://pve.proxmox.com/wiki/Stretch_Cluster

You will have to span up a layer2 network between the locations. Keep in mind that Proxmox's cluster integration and Ceph tooling expect that all nodes use IPs in the same subnets for the Proxmox VE clustering (Corosync) and the networks used by Ceph.

Depending on your setup, you can either span VLANs across the DCs, or encapsulate them in EVPNs for example. Depending on what your switches can handle.
Important is, that the current stretch cluster as described in that guide, doesn't handle the loss of connectivity between both sides well. So that connection needs to be as stable and redundant as possible.

 

[abhijithiaaxin]

@gurubert , @aaron Thanks for the information.

Yes, our plan is to keep all 4 nodes as a single Proxmox cluster.

Our main requirement is that our production VMs/web applications should continue running if one complete datacenter goes down.

Some details about our setup:

• DC1: 2 Proxmox nodes
• DC2: 2 Proxmox nodes
• Both DCs have different public IP ranges
• Both DCs are connected through a dedicated network
• We want to use Ceph for VM storage and Proxmox HA

I have a few important questions about the overall design:

1. If we add a third location with a QDevice/witness, will our 2+2 node setup be able to provide automatic HA when one complete DC goes down?
2. Since the two DCs have different public IP ranges, what is the recommended network design for Corosync and Ceph? Should we create separate private networks between the two DCs?
3. For the VM network, we understand that Layer 2 needs to be available between both DCs. What is the recommended way to achieve this in a production environment — VLAN stretching or EVPN/VXLAN?
4. How should Ceph be configured so that VM disks remain available if one complete DC is lost? What Ceph replication/failure-domain setup would you recommend for two DCs?
5. What is the recommended overall architecture for our requirement, considering both DC failure and inter-DC link failure?

We want to finalize the design before starting the implementation, so we would really appreciate guidance on the complete architecture.

As @gurubert already mentioned, you will need one node in a 3rd location.

Our Stretch Cluster guide has more details on how to get it up and running: https://pve.proxmox.com/wiki/Stretch_Cluster

You will have to span up a layer2 network between the locations. Keep in mind that Proxmox's cluster integration and Ceph tooling expect that all nodes use IPs in the same subnets for the Proxmox VE clustering (Corosync) and the networks used by Ceph.

Depending on your setup, you can either span VLANs across the DCs, or encapsulate them in EVPNs for example. Depending on what your switches can handle.
Important is, that the current stretch cluster as described in that guide, doesn't handle the loss of connectivity between both sides well. So that connection needs to be as stable and redundant as possible.

 

[bbgeek17]

Hey @abhijithiaaxin, welcome to the forum.

Whether it is a single-DC deployment or a split-DC deployment, you generally want an odd number of nodes in your cluster. But, as others have pointed out, a split-DC installation requires three locations.

If DC1 goes down, you still have a majority of the nodes in DC2 + DC3. The same applies if DC2 goes down. If DC3 goes down, you still have a majority of the cluster in DC1 + DC2.

Of course, this only works if you have redundant links between the sites, as you also need to protect against losing connectivity simultaneously between DC1, DC2, and DC3.

Based on the discussion here and the questions you still have, you are building a system with above-average complexity. You may benefit from engaging a Proxmox Partner to assist: Proxmox Partner Directory.

The questions you are asking are all valid and on point. However, the questions you don't know to ask could be even more critical.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Francois Baillargeon]

I agree with the others that this is a setup with many caveats and has high disk consumption (due to the requirement of having rf4)

Also this sounds like Corosync will be over shared media , so make sure your QoS is tight accordingly

 

[SteveITS]

Ceph for shared storage

What is the speed and latency between the two locations? Hopefully 10 Gbit or higher? I could see this being very slow in several cases since at least 1/3 of all I/O is copied to the other data center.

4 nodes is more than 3 but you may want to read through https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/ anyway.

As far as IP addressing that's also a good question. You could route a subnet to one or the other but not really both so IPs would likely change. Some form of BGP routing/SD-WAN to move the IPs with the VMs, I suppose.

Requirements aside it would be much easier to have two clusters and use PDM to migrate VMs between them but that doesn't accomplish your goal of no downtime or moving the IP with the VM.

 

[Francois Baillargeon]

What is the speed and latency between the two locations? Hopefully 10 Gbit or higher? I could see this being very slow in several cases since at least 1/3 of all I/O is copied to the other data center.

4 nodes is more than 3 but you may want to read through https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/ anyway.

As far as IP addressing that's also a good question. You could route a subnet to one or the other but not really both so IPs would likely change. Some form of BGP routing/SD-WAN to move the IPs with the VMs, I suppose.

Requirements aside it would be much easier to have two clusters and use PDM to migrate VMs between them but that doesn't accomplish your goal of no downtime or moving the IP with the VM.

Adding to @SteveITS bandwith is indeed important, if you can get to 25 or even 100gigs between sites..

You have to keep in mind that say, in RF4, each written block gets written to 4 places for safecopying, that's a lot