Ceph pools.

[shiroe]

Hey everyone, knew to proxmox and the forums,

I am trying to create a ceph cluster.
when creating a pool the default size is 3 and the default minimum size is 2.
What would be the issue with creating a size of 2 and a min size of 2?
Does this only have 1 disk fault tolerance basically?
would this effect fault tolerance/HA?

thanks,
Shiroe.

 

[kofik]

With the defaults you can lose 1 node and still have possibility to check if your data is consistent since you have 2 remaining replicas. If you lose that second replica you end up in read-only mode but you still have data.

If I'm not mistaken:

• Once you fix the missing OSDs you should be able to go back to a working state.
• However in your size = 2, and min size = 2 you cannot lose any disk without the cluster going into read-only.

So yes, the defaults do have some meaning in my oppinion. It is quite helpful for me to be able to lose one cluster node knowing that my VMs will remain up, though with reduced performance. Once the 3 node comes back up, I will also have some performance hit until the pool is back in sync across all nodes.

 

[shiroe]

With the defaults you can lose 1 node and still have possibility to check if your data is consistent since you have 2 remaining replicas. If you lose that second replica you end up in read-only mode but you still have data.

If I'm not mistaken:

• Once you fix the missing OSDs you should be able to go back to a working state.
• However in your size = 2, and min size = 2 you cannot lose any disk without the cluster going into read-only.

So yes, the defaults do have some meaning in my oppinion. It is quite helpful for me to be able to lose one cluster node knowing that my VMs will remain up, though with reduced performance. Once the 3 node comes back up, I will also have some performance hit until the pool is back in sync across all nodes.

Awesome,
Thank you for taking the time to explain this, I have been reading and seeing forums where people discussed a 2/1 but not a 2/2 scenario.
I am transitioning over from Nutanix CE to PVE with ceph.

 

[kofik]

I'm definitely not considering myself an expert in that area, which is the reason at work we did set up a (really small) test cluster with just 3 desktops besides our production cluster so that I could play (read: break things) and better understand how Ceph behaves under stress.

I've learned from past experiences that going razor thin in terms of redundancy usually hurts way more when things break in castastrophic ways and that it does help to have at least some basic understanding of how you react to at least basic issues like a broken disk. I honestly have not yet played / experienced how Ceph behaves once it goes below the minimum amount of replicas, but your post motivates me to to test this on our test cluster.

 

[shiroe]

I'm definitely not considering myself an expert in that area, which is the reason at work we did set up a (really small) test cluster with just 3 desktops besides our production cluster so that I could play (read: break things) and better understand how Ceph behaves under stress.

I've learned from past experiences that going razor thin in terms of redundancy usually hurts way more when things break in castastrophic ways and that it does help to have at least some basic understanding of how you react to at least basic issues like a broken disk. I honestly have not yet played / experienced how Ceph behaves once it goes below the minimum amount of replicas, but your post motivates me to to test this on our test cluster.

Yep, I understand that as well.
I have 2x intel dc s3710's and 1 1.2 nvme u2 sas ssd coming in per node now, I believe the hdd's I had were the bottleneck in my cluster.
I plan on using those hdd's to create a zfs pool as a network backup share with proxmox backup + storage.