Optimal Proxmox Ceph cluster settings for 5 node but 2 node sometimes offline and only created for Osd replica backup

MrBruce

New Member
Mar 7, 2020
25
3
3
Hi, I have 5 Proxmox node + 5 osd + 5 monitor + 5 manager (all node same lineup but different HDD size and different hardware), lastly added 5., and I see this:

1591921056282.png

Ceph settings:

1591921188138.png

I have only 1 pool, for all VM:

1591921539296.png

I sometimes disable 2 Proxmox node and only 3 Proxmox node available.
You recommend to set min_size to 5 from 3? (for get 5 replica?)
I should change size from 3 to 5 or not?
If only 3 replica will be created because 2 node offline from 5, all will works fine, just Ceph health will be not OK?

If I have 5 osd but replica settings is 3, all times will be make 5 replica if I have 5 Proxmox node and 5 Ceph OSD?

What happening if I set replica 5, but later change back to replica 3? Ceph will remove 2 replica from 2 random node?

Thank you.

Update:

1591922529352.png

Seems like synced the 5. node osd too.
 
Last edited:
Hi, I have 5 Proxmox node + 5 osd + 5 monitor + 5 manager
Normally having 3 mons and mgrs should be enough and be able to cover the loss of a server.
I sometimes disable 2 Proxmox node and only 3 Proxmox node available.
Why? You simulate a situation where two nodes are down and Ceph needs to create the lost PGs on the remaining nodes to get to the set number of replicas again.

The `size` parameter of the pool determines how many replicas should be present. The `min_size` determines how many replicas need to be present for the pool to be functional. The default setting is 3/2 (size=3, min_size=2).

In a 3 node cluster, each node holds a copy of the data. Once you add more nodes and don't increase the size, Ceph will place the PGs on the available nodes according to the crush map. This is why you can increase the overall capacity of a Ceph Cluster with more nodes once you cannot add more OSDs to the already present nodes.

If you reduce the size from 5 to 3 and continue to power off 2 of the 5 nodes, you could end up with some PGs having only a replica of 1 available and thus blocking write IO until the PGs have been replicated to the existing nodes.

Should the one node containing the one replica fail in the meantime you will have data loss!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!