Ceph unavailable from single node

[chell]

Hi All

I have a 3 node cluster with ceph storage. I want to configure the ceph cluster to work with only one node. (in the even of a double failure)
Currently the ceph continues working correctly with the failure of one node, but as soon as two nodes are down, ceph becomes unavailable and you cant even open the ceph configuration pages.
All of the pools are configures for 3 x replication with a minimum of 1 and host failure domain.
Thanks

 

[Moayad]

By default, a Ceph cluster requires a majority of nodes to maintain quorum!

All of the pools are configures for 3 x replication with a minimum of 1 and host failure domain.

Note, with min_size=1, there is a risk of data loss if only one replica is available during a write operation.

 

[chell]

By default, a Ceph cluster requires a majority of nodes to maintain quorum!



Note, with min_size=1, there is a risk of data loss if only one replica is available during a write operation.

The normal operation will be three nodes. Only one node will be taken offline at a time for maintenance. Running on a single node will only occur if a fault occurs when another node is already down for another fault or maintenance.
This site is rural. Some repairs can take weeks, so a second failure while waiting for the first failure to be fixed is a real possibility.

 

[Moayad]

Ceph needs a MON majority for quorum; losing this halts operations.... An option is to add a lightweight `tiebreaker` node to the cluster, which doesn't need to store data but can maintain quorum. Plus consider using the `noout` flag during maintenance to avoid triggering recovery processes.

 

[chell]

Ceph needs a MON majority for quorum; losing this halts operations.... An option is to add a lightweight `tiebreaker` node to the cluster, which doesn't need to store data but can maintain quorum. Plus consider using the `noout` flag during maintenance to avoid triggering recovery processes.

Is this the reason why ceph become unavailable to configure when there is only one node?
Adding another server is not possible, however I could create a small vm on a couple of the workstations just for ceph mon's.

 

[Moayad]

Is this the reason why ceph become unavailable to configure when there is only one node?

Yes.

Adding another server is not possible, however I could create a small vm on a couple of the workstations just for ceph mon's.

This won't help if the cluster itself loses quorum.

 

[chell]

Yes.


This won't help if the cluster itself loses quorum.

Why would this be the case? Wouldn't the additional mon's be the tie breaker in the event the two nodes loose quorum?