CEPH SSD Vs 10Gb Ethernet Upgrade

Discussion in 'Mail Gateway: Installation and configuration' started by Haider Jarral, Oct 8, 2018.

  1. Haider Jarral

    Haider Jarral New Member

    Joined:
    Aug 18, 2018
    Messages:
    12
    Likes Received:
    0
    Hello All,

    I need some recommendation on improving my cluster. I have a 4 node cluster with CEPH having 65 OSDs at 30 TB capacity. My cluster communication is currently at 1G NIC and I dont have any SSDs in my servers.

    I recently had issue with disk I/O time with some web-servers applications that use SQL/PHP intensively.

    I am looking into upgrading my cluster. What are the recommendations to improve my cluster performance. Should I add 10Gb ethernet cards or SSDs or both considering limited budget.

    Looking at my current CEPH stats they are not even at 1-2 MiB/s. I dont think I am saturating network links yet because the tests that I did yield a speed of 110 MiB/s. Would adding SSDs improve performance more or adding 10Gb NICs improve performance more ?

    Any help/insight is appreciated.
     
  2. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,062
    Likes Received:
    254
    In your setup: SSD.

    Monitor your current network throughput, e.g. with iptraf or some other interactive cli tool, but I suggest that you will not see saturation. First step would IMHO be an upgrade to SSD and then you'll maybe saturate your network throughput and can go one step further on that front.
     
  3. Haider Jarral

    Haider Jarral New Member

    Joined:
    Aug 18, 2018
    Messages:
    12
    Likes Received:
    0
    Thanks @LnxBil , my next question would be would I use SSDs just as journal disk ? If so can I use a single SSD/server for all my OSDs and how to replace existing journal disk which is HDD. I have 2 journal disks in each server. Sorry if thats a lot of questions. I am just looking for right direction to look into.

    This is probably peak performance snapshot plus/minus.

    upload_2018-10-8_12-56-31.png
     
  4. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,062
    Likes Received:
    254
    This is the first step to improve your performance, yes. Your pool will be much faster if you use SSD everywhere, but that'll most probably be too expensive.

    But be aware, there are fast and there are slow SSDs, best is to buy from this list:

    http://www.sebastien-han.fr/blog/20...-if-your-ssd-is-suitable-as-a-journal-device/

    Your screenshot stats, that you write mostly and this is very, very costly on HDDs, so writing it first to SSDs is much faster.

    It is said that 4-5 journals per SSD should be used, so you need to size them accordingly. If you use only one, it'll be slower, but still faster than using your pool without any SSDs, obviously.
     
  5. Haider Jarral

    Haider Jarral New Member

    Joined:
    Aug 18, 2018
    Messages:
    12
    Likes Received:
    0
    How can I determine the right size for SSDs ?
     
  6. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,062
    Likes Received:
    254
  7. Haider Jarral

    Haider Jarral New Member

    Joined:
    Aug 18, 2018
    Messages:
    12
    Likes Received:
    0
    Thank you for all the assistance. The link you mentioned is very confusing it mentions about WAL + DB which I have no idea about. Neither does it mention like concrete steps to determine right size for journal :(
     
  8. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    3,062
    Likes Received:
    254
    Depends on what you write to your disks and the size of your disks. Why haven't you just search the internet about CEPH journal sizing? There is a ton of information and in one minute I found this:

    http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008606.html
     
  9. Haider Jarral

    Haider Jarral New Member

    Joined:
    Aug 18, 2018
    Messages:
    12
    Likes Received:
    0
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice