Multiple PVE Clusters - Single Ceph Cluster

[guytp]

Hello,

Is it possible to have two or more PVE clusters configured with a single Ceph cluster behind them?

Rationale behind this is partly thought exercise and partly practical. We have several old Intel hosts and a bunch of newer AMD hosts. To prevent live migration between them we can use groups and migrate within the group - but I'd also then like to add some HA for if the entire stack of Intel or AMD go down and to then (outside of PVE) have an option to manually re-create VMs in the other cluster.

Could I just make a bigger PVE cluster here? Yes - but that's got some limits in size that this approach wouldn't have.

 

[shrdlicka]

Hi,
it sounds to me a bit like you want to do RBD mirroring [1]. Where one pool is mirrored to another cluster and vice versa.

The old guide [2] only explains how to set up journaling. Journaling requires that the receiving cluster is fast enough to handle incoming data. If the receiving cluster is too slow, data will pile up at the sending cluster. And a full ceph cluster is a big problem .

A colleague of mine is working on an improved guide that is not public currently. Which also explains how to set up snapshot based mirroring.


[1] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/
[2] https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring

 

[guytp]

Thanks, I will take a look as an option and see if it can suit our needs. I realised after writing the original post that PVE doesn't allow you to upload/attach storage in a way that would necessarily work with my original thoughts so this may be another option to add the redundancy for DR.

 

[gurubert]

You can always add an RBD storage to Proxmox that accesses a Ceph cluster which does not need to be managed by the same Proxmox cluster.
You just need to provide the MON IPs and a keyring to access a specific pool.

If you want to share the same pool and use the same RBD images for VMs in different Proxmox clusters you have to manage that yourself and make sure that no parallel access is done on the same image.