optimal ceph configuration for a 4 server 2 location HA setup

[dpproxmox]

Hi All,

We have 2 datacenters, one located in each building. We are setting up a new proxmox VE HA
cluster of 4 machines. The idea being if one building goes down for an extended time, the other
2 servers will be able to keep everything up. In this setup each server has 8 ssd's. one ssd
would have the operating system on it, but the other 7 would be used for a shared ceph storage
across all 4 nodes. I am looking for some advice as well as some configuration help.

First, what configuration do we need to make to the proxmox nodes such that we can allow 2
servers to go down and still remain operational?

secondly, is there a better solution than what I have proposed here?

Thank you for any help and advice,

Ross

 

[t.lamprecht]

Hi,

First, what configuration do we need to make to the proxmox nodes such that we can allow 2
servers to go down and still remain operational?

You need a QDevice setup on an external host not located on any of the involved PVE nodes. That can be a small plain Debian host, even a RPi can work out fine.
https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_corosync_external_vote_support

secondly, is there a better solution than what I have proposed here?

The idea can work out, but as it seems that you're relatively new to that tech stack I'd suggest spending quite some time on reading docs up front and naturally some good testing, especially so that the Ceph storage behaves as intended, as working with failure domains and CRUSH rules may require a bit of time to get used to.
https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_device_classes
https://docs.ceph.com/en/latest/rados/operations/crush-map/#

An alternative would be using storage replication and ZFS, could be a bit simpler to setup, but you'd also lose some features of ceph here (better rebalancing and a "real" redudantly-shared storage).

 

[fabian]

https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ is what you want, but it requires some knowledge about Ceph to set up..

edit: and just like for "our" corosync cluster, you also need a third location for the Ceph tie-breaker monitor - a raspberry likely won't cut it for that, but according to the Ceph docs (no personal experience) it can be more lightweight and less well connected than a regular monitor (for example, run as a VM somewhere)

 

[dpproxmox]

This was all great information. It looks like stretch mode is what we need here. Thanks for all the info.