Ceph erasure coding configuration best practices

[Gayan Kodithuwakku]

I am trying to size a cluster with five nodes, utilizing a Ceph erasure coding configuration (EC 3+2).

1. How should I properly size the RAM for this setup, considering:
   • FTT = 2
   • Workload VM memory sizing in failure
2. Are there any best practices or considerations specific to the EC 3+2 configuration in a five-node cluster?

Thank you in advance for your guidance.

 

[Alwin Antreich]

How should I properly size the RAM for this setup, considering:

• FTT = 2
• Workload VM memory sizing in failure

What is FTT in this context? The memory target for OSDs should be 8 GB, which means 8GB per daemon. Anything lower usually sacrifices performance during recovery. With the memory target using EC or replica doesn't make a difference.

1. Are there any best practices or considerations specific to the EC 3+2 configuration in a five-node cluster?

Yes. Starting with the general recommendations [0] from the docs.

• You need 5x nodes to make it work, but a loss of a node will keep the cluster in a degraded state. Add a node for redundancy.
• You need 5x OSDs [1] for an object to read/write it, while only 3x OSDs with replica (only one for reading). Which means replica is faster.
• Additionally any object will be chunked in 3x equal pieces (+2 coding chunks), the object size does matter. Only use fast NVMe if you intent to put RBDs for your VM/CT workload on it, as latency is key.

Unless you need to desperately save space, I wouldn't use EC for a 5x node cluster.

[0] https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_recommendations_for_a_healthy_ceph_cluster
[1] https://docs.ceph.com/en/latest/rados/operations/erasure-code/