[SOLVED] Have I f...ed up my HA with CEPH?

[proxwolfe]

Hi,

I have a small three node homelab PVE cluster running. Each of the PVE nodes is also a CEPH node and each node houses one NVME OSD and one HDD OSD. The three NVME OSDs form a pool and so do the three HDD OSDs.

The idea was to have HA for my VMs, i.e. that if one node fails, the VMs can continue running on one of the other two nodes.

But now I am having doubts whether that will work because it seems that if one CEPH node/OSD fails, the other two CEPH nodes/OSDs are not enough to keep the lights on.

Is my impression wrong or would I be better off getting rid of CEPH again? Or is there another feasible course of action for me (at the moment I don't want to add more drives)?

Thanks!

 

[mira]

As long as you have enough space available on your other 2 nodes so that the data can be backfilled, it should be fine in a 3/2 (size/min_size) setup, since there are still 2 hosts left.
But neither a host, nor a single OSD can fail once one host is already down since then you don't have 2 replicas available for every object.

 

[proxwolfe]

As long as you have enough space available on your other 2 nodes so that the data can be backfilled, it should be fine in a 3/2 (size/min_size) setup

I do have a 3/2 setup. And I am certainly not hoping that more than one node would fail at the same time (but you never know...).

So is it going to work only after the backfilling has ended or is it going to work also while the backfilling is still in progress?

 

[mira]

It will work as long as there are at least 2 replicas available. Backfilling will increase the load though, but by default it is rather slow so that it doesn't interfere too much with normal operation.

If only 1 replica is available, it will block until at least a 2nd one has been created.

 

[proxwolfe]

This gives me peace of mind - thank you!