Optimal Proxmox Ceph cluster settings for 5 node but 2 node sometimes offline and only created for Osd replica backup

[MrBruce]

Hi, I have 5 Proxmox node + 5 osd + 5 monitor + 5 manager (all node same lineup but different HDD size and different hardware), lastly added 5., and I see this:



Ceph settings:



I have only 1 pool, for all VM:



I sometimes disable 2 Proxmox node and only 3 Proxmox node available.
You recommend to set min_size to 5 from 3? (for get 5 replica?)
I should change size from 3 to 5 or not?
If only 3 replica will be created because 2 node offline from 5, all will works fine, just Ceph health will be not OK?

If I have 5 osd but replica settings is 3, all times will be make 5 replica if I have 5 Proxmox node and 5 Ceph OSD?

What happening if I set replica 5, but later change back to replica 3? Ceph will remove 2 replica from 2 random node?

Thank you.

Update:



Seems like synced the 5. node osd too.

 

[aaron]

Hi, I have 5 Proxmox node + 5 osd + 5 monitor + 5 manager

Normally having 3 mons and mgrs should be enough and be able to cover the loss of a server.

I sometimes disable 2 Proxmox node and only 3 Proxmox node available.

Why? You simulate a situation where two nodes are down and Ceph needs to create the lost PGs on the remaining nodes to get to the set number of replicas again.

The `size` parameter of the pool determines how many replicas should be present. The `min_size` determines how many replicas need to be present for the pool to be functional. The default setting is 3/2 (size=3, min_size=2).

In a 3 node cluster, each node holds a copy of the data. Once you add more nodes and don't increase the size, Ceph will place the PGs on the available nodes according to the crush map. This is why you can increase the overall capacity of a Ceph Cluster with more nodes once you cannot add more OSDs to the already present nodes.

If you reduce the size from 5 to 3 and continue to power off 2 of the 5 nodes, you could end up with some PGs having only a replica of 1 available and thus blocking write IO until the PGs have been replicated to the existing nodes.

Should the one node containing the one replica fail in the meantime you will have data loss!