Hi,
we have an even numbered cluster stretched over 2 buildings, we're using Size 4/2 with a custom crush rule for placement of 2copies in each building.
Everything works great, one side could die due to power issues etc. and in that situation we'd still have 2 copies up and Ceph could work in the surviving site.
But losing one side still causes Ceph to stop working because the Ceph Mons don't have quorum.
Regarding Proxmox Cluster/Corosync, qdevice as external voting mechanism is the solution (https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support)
But regarding Ceph here's the problem:
- Third location only has one public network that does not correspond to neither the physically seperated Ceph Network nor the multiple Corosync networks used in the production cluster, so we can't just join a Proxmox Node from the third location as a regular Proxmox cluster member (needs to have IPs in the same corosync networks, right?)
What could we do to have a Ceph Mon offsite? (or work around the quorum-problem with the Ceph Mons locally?)
Looking forward to your ideas!
(I found no matching threads via search, but if there is something previously written, feel free to just post some links for us to read up on!)
we have an even numbered cluster stretched over 2 buildings, we're using Size 4/2 with a custom crush rule for placement of 2copies in each building.
Everything works great, one side could die due to power issues etc. and in that situation we'd still have 2 copies up and Ceph could work in the surviving site.
But losing one side still causes Ceph to stop working because the Ceph Mons don't have quorum.
Regarding Proxmox Cluster/Corosync, qdevice as external voting mechanism is the solution (https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support)
But regarding Ceph here's the problem:
- Third location only has one public network that does not correspond to neither the physically seperated Ceph Network nor the multiple Corosync networks used in the production cluster, so we can't just join a Proxmox Node from the third location as a regular Proxmox cluster member (needs to have IPs in the same corosync networks, right?)
What could we do to have a Ceph Mon offsite? (or work around the quorum-problem with the Ceph Mons locally?)
Looking forward to your ideas!
(I found no matching threads via search, but if there is something previously written, feel free to just post some links for us to read up on!)
