Ceph-equivalent of a qdevice in an even numbered split cluster

Z

Zerstoiber

Member
Jun 2, 2022
58
40
23
Hi,

we have an even numbered cluster stretched over 2 buildings, we're using Size 4/2 with a custom crush rule for placement of 2copies in each building.
Everything works great, one side could die due to power issues etc. and in that situation we'd still have 2 copies up and Ceph could work in the surviving site.

But losing one side still causes Ceph to stop working because the Ceph Mons don't have quorum.

Regarding Proxmox Cluster/Corosync, qdevice as external voting mechanism is the solution (https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support)


But regarding Ceph here's the problem:
- Third location only has one public network that does not correspond to neither the physically seperated Ceph Network nor the multiple Corosync networks used in the production cluster, so we can't just join a Proxmox Node from the third location as a regular Proxmox cluster member (needs to have IPs in the same corosync networks, right?)

What could we do to have a Ceph Mon offsite? (or work around the quorum-problem with the Ceph Mons locally?)

Looking forward to your ideas!
(I found no matching threads via search, but if there is something previously written, feel free to just post some links for us to read up on!)
 
Hi,

Asking myself this same question, setting up a 2 node cluster with the built-in ceph and gonna try to add a raspberry pi to the cluster to see if that works.

My fear is my performance tanking by the raspberry trying to serve client/osd requests. I'm thinking I can trick this by defining a false datacenter in crush and assigning it to the daemon so that the osds only connect to the true monitors.

The other option is setting my cluster into stretched mode which has explicit compatibility with this setup (external monitor with high latency/no clients served) but it limits you badly on pool features.

Will get you back with updates after I try the first option.
 
Hello again,

Bad news, couldn't find a way to deploy a remote monitor with the orchestration that proxmox uses. I admit this is mostly a limitation of my setup (Cost-cut QDevice).

You could probably use a slightly beefier system to install proxmox, only for the pveceph, and then add it to the (ceph only) cluster so that it adds in to the quorum. This kind of defeats the purpose of it being a QDevice but I'm kind of out of options that can mesh with proxmox.

The closest thing I've gotten to achieving it with my setup is by creating dedicated Ceph VMs per host, then deploy a Ceph Cluster on my nodes with cephadm and finally add the QDevice since they're all running the same ceph orchestration. I didn't even try to run a benchmark so I'm unsure on it's performance compared to bare metal but at least I got a working even node cluster.

Sorry if I couldn't do much else, kind of limited by hardware for testing.
 
You must log in or register to reply here.
Top Bottom
Back