SSD CEPH and network planning

[BelCloud]

I'm considering switching to ceph in order to add HA, but i'm not yet sure about the network requirements.

How much does ceph's performance degrades compared to raid10?

Considering the following setup:
3 servers with 8x 480GB Intel S3500 SSD (500M/s max seq. read, 400M/s max seq write)

In theory, one such system would be able to get 4GB/s read and 3.2GB/s write maximum (no performance degradation)
At maximum usage (utopia), 7.2G/s would mean 57.6Gbps (6x10ge cards)

However, in real life, how much would the disks performance degrade? What network connection would i require per server?

Kind regards

 

[Q-wulf]

How much does ceph's performance degrades compared to raid10?

Depends on the underlying replication used, the write patterns and wether or not your dedicated ceph network will be able to deal with it.


I suggest you have a look at this ceph documentation:
http://docs.ceph.com/docs/hammer/architecture/
specifically this diagram:

When you write to a ceph OSD that write is only registered as complete by the client once it has been registered on all relevant journals of OSD(1/2/3)




What this means for you I'm gonna try and dumb down a little so bear with me:

• Now in your example you you have 24 SSD's.
  • Replication of 1 (no single drive may break)
  • Replication of 2 (you can loose one drive): you essentially write 2 copies for each file. ==> 12 drives
  • Replication of 3 (you can loose 2 drives): you essentially write 3 copies for each file ==> 8 Drives
• You have 3 Nodes, so you probably wanna replicate on the bucketType of "host", rather then the BucketType of "OSD"
  • Replication 1: File gets written to next OSD in line (if in local node, then no network utilisation)
  • Replication 1: File gets written to next OSD in line (if in not on local node, then network utilisation occurs - including latency)
    
  • Replication 2: File gets written to next OSD in line (network utilisation likely), once that is done it gets written to a second OSD on a second node (network utilisation on one link + added latency)
  • Replication 2: File gets written to next OSD in Line (network utilisation likely), once that is done it gets written to a second OSD and a second and in parallel to a third OSD (network utilisation on 2 links + added latency)
• Now ceph has something called journals for every OSD.
  • When you use SSD + HDD the guidance is AFAIK to put the journals for a HDD on the SSD's - check http://www.sebastien-han.fr/blog/20...-if-your-ssd-is-suitable-as-a-journal-device/ for some god guidance on usability of ssd's as journals.
    • unless you have a large amound of HDD's, then you probably wanna work with cache Tiering http://docs.ceph.com/docs/master/rados/operations/cache-tiering/
  • When you use SSD-only OSDs, AFAIK the guidance is to leave the journal for the SSDs.
    • every time you write a object to a OSD it gets put into the journal (a partition on said drive)
      • Once on the journal it gets acknowledged
    • from the journal it gets placed on the Data drive - check http://docs.ceph.com/docs/hammer/rados/configuration/journal-ref/ for reference
      • This multiplies every write to every OSD by 2.
        • Replication 1: 2 writes
        • Replication 2: 4 writes + network latency
        • Replication 2: 6 writes + network latency
• But fear not, ceph has something called Placement Groups (PG's) - check http://docs.ceph.com/docs/master/rados/operations/placement-groups/#how-are-placement-groups-used for reference
  • It is basically a way for Ceph to "know" where an object is located
  • That number is configureable - but afaik the guidance is to aim for a total of 100 PG's per OSD.
  • With this it is statistically very likely that a large amount of parallel writes of multiple objects happens to be put into different PG's on different OSD's:
    • you use the space "evenly"
    • you get a benefit from writing objects in parallel
    • this is where your SSD's ability to accept parallel writes comes in.
      • the more parallel writes it can achieve, the more throughput and IOP/s your ceph cluster is able to handle.

TL;DR
In short, for a replication 3 pool, expect a reduction by a factor of 3 in maximum throughput on normal usage and at best a slowdown by a factor of 6 during heavy usage (depended on your SSD's ability to handle a large amount of parallel writes). And factor in at least the effect of your network latency.

Edit: i just cut a bunch of this post cause it was riddled with (early morning) math mistakes.

 

[BelCloud]

Thank you very much! Excellent post!

 

[jkirker]

Thanks Q-Wulf.

 

[likebugs]

Thank you very much! Excellent post!

Hi BelCloud,

Cloud you shared the benchmark results after deploy ceph?

Thanks,

 

[PigLover]

One small quibble with the above. The default behavior is to acknowledge the write when it has been registered in (n/2) journals (not all of them). Assuming you keep an odd number of replicas in the pool then this guarantees a "quorum" of replicas in case things need to be recovered. This means in a pool size "3" that 2 of the 3 replicas must be confirmed in order to confirm the write.

So in a pool with 3 replicas the expected write "slowdown" would be something close to 2x nominally and 4x in exceptional cases (as opposed to the 3x/6x noted above). I believe its actually a bit better than that because the primary OSD does not have to wait for the journal write before it can launch the writes to the replicas - but you have to review a complete ping-pong diagram of the transaction to get all the details right.

In order to improve write latency, with increased risk to resiliency, you can reduce the number of write replicas that must complete before the write is confirmed. The option "pool min size" can be set to determine the number of replicas written before a write is acknowledged. 'Min size 1' acknowledges the write after 1 OSD is written, etc. 'Min size 0' restores default behavior.

Probably not something to be messed with casually, however. If you move on after writing less than the quorum and something "bad" happens before the writes are propagated you risk the integrity of your data.

 

[spirit]

I'm considering switching to ceph in order to add HA, but i'm not yet sure about the network requirements.

How much does ceph's performance degrades compared to raid10?

Considering the following setup:
3 servers with 8x 480GB Intel S3500 SSD (500M/s max seq. read, 400M/s max seq write)

In theory, one such system would be able to get 4GB/s read and 3.2GB/s write maximum (no performance degradation)
At maximum usage (utopia), 7.2G/s would mean 57.6Gbps (6x10ge cards)

However, in real life, how much would the disks performance degrade? What network connection would i require per server?

Kind regards

It's depend mainly of the io pattern.
If you do big block read, no problem, you'll reach max speed.

if you do lot of small ios (4k), be carefull to take high frequency cpus (intel 3ghz).

with 3 nodes, 2x10cores 3,1ghz, replication x3, 18 osd s3610, I can reach 70000iops 4k write and 600000iops 4K read.



Note that for writes, you'll write datas twice on disk (1xjournal, 1x datas). This should change in the future (1year), with the new ceph bluestore implementation.