Proxmox + Ceph across 2 datacenters

A

abhijithiaaxin

New Member
Aug 9, 2026
2
0
1
Hi all,

We are planning to set up a 4-node Proxmox cluster with Ceph across 2 datacenters.

Our setup will be:

  • DC1: Node 1 and Node 2
  • DC2: Node 3 and Node 4
  • Node 1 and 2 are using one network range.
  • Node 3 and 4 are using a different network range.
  • Both datacenters are connected through an inter-DC network.
We want to use Ceph for shared storage, along with VM migration and HA.

Our main confusion is about the network design.

If a VM is running on Node 1 in DC1, its VM network is from the DC1 network. If HA moves the VM to Node 3 in DC2, the VM network is different there.

So we are not sure how to design the VM network so that the VM remains accessible after migration.

We also want to understand what will happen if the inter-DC network goes down. We are concerned about Corosync quorum, HA fencing and split-brain.

Could someone please suggest the correct architecture for this type of setup?

Especially:

  • How should we handle the VM network between the two datacenters?
  • Should the VM network be stretched between both DCs?
  • Should we use routing between the two VM networks?
  • How should we configure Corosync and Ceph?
  • Do we need a QDevice in a third location?
  • Is HA across both datacenters recommended in this setup?
  • What happens if the inter-DC link fails?
We have not started the implementation yet, so we would like to finalize the network and cluster design first.

If anyone is running a similar Proxmox + Ceph setup across two datacenters, please share your architecture and suggestions.
 
As @gurubert already mentioned, you will need one node in a 3rd location.

Our Stretch Cluster guide has more details on how to get it up and running: https://pve.proxmox.com/wiki/Stretch_Cluster

You will have to span up a layer2 network between the locations. Keep in mind that Proxmox's cluster integration and Ceph tooling expect that all nodes use IPs in the same subnets for the Proxmox VE clustering (Corosync) and the networks used by Ceph.

Depending on your setup, you can either span VLANs across the DCs, or encapsulate them in EVPNs for example. Depending on what your switches can handle.
Important is, that the current stretch cluster as described in that guide, doesn't handle the loss of connectivity between both sides well. So that connection needs to be as stable and redundant as possible.
 
  • Like
Reactions: abhijithiaaxin and gurubert
@gurubert , @aaron Thanks for the information.

Yes, our plan is to keep all 4 nodes as a single Proxmox cluster.

Our main requirement is that our production VMs/web applications should continue running if one complete datacenter goes down.

Some details about our setup:

  • DC1: 2 Proxmox nodes
  • DC2: 2 Proxmox nodes
  • Both DCs have different public IP ranges
  • Both DCs are connected through a dedicated network
  • We want to use Ceph for VM storage and Proxmox HA
I have a few important questions about the overall design:

  1. If we add a third location with a QDevice/witness, will our 2+2 node setup be able to provide automatic HA when one complete DC goes down?
  2. Since the two DCs have different public IP ranges, what is the recommended network design for Corosync and Ceph? Should we create separate private networks between the two DCs?
  3. For the VM network, we understand that Layer 2 needs to be available between both DCs. What is the recommended way to achieve this in a production environment — VLAN stretching or EVPN/VXLAN?
  4. How should Ceph be configured so that VM disks remain available if one complete DC is lost? What Ceph replication/failure-domain setup would you recommend for two DCs?
  5. What is the recommended overall architecture for our requirement, considering both DC failure and inter-DC link failure?
We want to finalize the design before starting the implementation, so we would really appreciate guidance on the complete architecture.

aaron said:
As @gurubert already mentioned, you will need one node in a 3rd location.

Our Stretch Cluster guide has more details on how to get it up and running: https://pve.proxmox.com/wiki/Stretch_Cluster

You will have to span up a layer2 network between the locations. Keep in mind that Proxmox's cluster integration and Ceph tooling expect that all nodes use IPs in the same subnets for the Proxmox VE clustering (Corosync) and the networks used by Ceph.

Depending on your setup, you can either span VLANs across the DCs, or encapsulate them in EVPNs for example. Depending on what your switches can handle.
Important is, that the current stretch cluster as described in that guide, doesn't handle the loss of connectivity between both sides well. So that connection needs to be as stable and redundant as possible.
Click to expand...
 
You must log in or register to reply here.
Top Bottom
Back