Ceph Erasure Code Recommendation

[oops404]

Hello all,

i want to try if Ceph with erasure code is something for me. I like the possibility to expand without keeping the same disk size.

I have 3 nodes with 2 hard disks which every node have a 4TB SSD and 1 TB SSD. So in total 15 TB.

So the data is important for me. I want to have no data loss if I lose 2 of 3 nodes.

Is this the correct profile to get this?

• Profile Name: rs-profile-high-ft
• Failure Domain: Host
• Data Chunks: 2
• Coding Chunks: 1
• k-value: 3 (2 data chunks + 1 coding chunk)
• m-value: 1 (1 coding chunk)

Not really or? I`m a bit confused.

EDIT: I think maybe Erasure Code is not the correct thing for me. I need more server for that or? Should test replication Ceph with 3 replicas.

Best

 

[aaron]

Erasue coding gets interesting the more nodes you have. Starting with about 5.

If you have a 3-node cluster, I would go with regular replication. If you leave the default values for size/min_size (3/2) and lose 2 nodes, your data will be okay, as long as you don't lose another node or are able to get one of the previously lost nodes up again.

One thing though: a 3-node setup is the bare minimum for a Ceph cluster and needs some extra considerations. Mainly, how many OSDs/disks you have per node.

If a single disk (and its OSD service) fail, Ceph will try to recover the lost replicas on the remaining disks. With just two disks, the last one will be full very quickly! That's why you should add at least 4 disks per node. If one fails, the data can be split up to the remaining 3, leaving you more leeway in how full the OSDs can get under normal operation to have enough free space for such a recovery situation.

Running out of space is the one thing that is really painful in a Ceph cluster!

 

[oops404]

Thank you! I have understand that it is not so how I thought? So now im a bit confused about replication ceph?

https://forum.proxmox.com/threads/proxmox-cluster-setup-what-is-the-best.128816/post-563779

 

[aaron]

I have understand that it is not so how I thought? So now im a bit confused about replication ceph?

Anything in particular? That answer by @jsterr explains it quite good.