Ceph - EC-Pool Setup with 3 hosts

[Faryu]

Hello,

In the next few days we are going to setup a small Proxmox/Ceph cluster with 3 hosts, each having 3 hdds and 1 ssd.
We want to create a EC-Pool using the 3 hdd's on each host.

As fas as I know we have to setup the EC-Pool using the formula: n = k + m.
If we use k=3 and m=3, then the data will be splitted across 6 osd's.

Will the data be splitt in 3 groups (number of hosts) with each host receiving two parts?
In that case the cluster will still be operational in the scenario of one host disconnecting (still 4 parts of the required 3 accessible).

My question in this is, can the EC-Pool be configured to behave like this and if yes, what needs to be done to achieve this setup?

I have seen the ruleset-failure-domain parameter, which in our case could be set to osd or host. But since we only have 3 hosts and not the required 6 (k+m) I'm not sure if setting the failure domain to host will work as intended or if it will display a warning the entire time.

 

[PigLover]

No. I do not believe that you can ensure the placement for a 3+3 EC pool on 3 Hosts such that you can still read with one host failed.

Also, I'm not sure what you gain from this configuration vs a "normal" pool with "replica=1" (i.e., two copies of all data).

With a 3+3 EC pool you require 2x the raw storage capacity, which is the same as a regular pool with replica=1.

I don't think you can get the resiliency you want, always ensuring readability with one host down. Even if you could, it is exactly the same resiliency you get with a "replica=1" pool.

You pay a pretty heavy write latency penalty with the EC pool.

You have a bit better resiliency against a spindle failure, but that is all. I guess that is a benefit.

 

[Alwin]

@PigLover, with replica=1 you mean size=2 min_size=1?

You also need to way how long the recovery vs the possibility of another disk failing with the same PG. As it is likely that you might loose another OSD that has the same PG. So a setup with 2/1 is not recommended, this is also why the new default is size=3, min_size=2.

 

[PigLover]

Fair. But with respect to the OPs question: what is the advantage of EC with 3+3 on a 3-node cluster? Firstly, I do not think you get a set of placement rules that would guarantee resiliency against a single host failure, so depending on your goals for the cluster the EC pool might not even “work” (i.e., meet your requirements). And you get the same space efficiency pool with two copies.

IF you want to run EC pools in a way that actually gains meaningful benefit from them you need more nodes...

 

[Alwin]

@PigLover, I agree. Just wanted to point out, that a 2/1 setup should not be used either.
http://docs.ceph.com/docs/master/rados/operations/erasure-code/#

 

[udo]

Fair. But with respect to the OPs question: what is the advantage of EC with 3+3 on a 3-node cluster?

Hi,
advantage of an EC-Pool: much, much less performance!

ceph isn't the fastest at all, so I would avoid an EC-Pool if I can (I use an big EC-Pool years ago for archive-data...).

Udo

 

[Faryu]

Storage space is not really an issue at the moment, so I guess it's best to use the replication strategy for now.
Thanks for the quick replies!