CEPH SSD Vs 10Gb Ethernet Upgrade

Haider Jarral

Well-Known Member
Aug 18, 2018
121
5
58
37
Hello All,

I need some recommendation on improving my cluster. I have a 4 node cluster with CEPH having 65 OSDs at 30 TB capacity. My cluster communication is currently at 1G NIC and I dont have any SSDs in my servers.

I recently had issue with disk I/O time with some web-servers applications that use SQL/PHP intensively.

I am looking into upgrading my cluster. What are the recommendations to improve my cluster performance. Should I add 10Gb ethernet cards or SSDs or both considering limited budget.

Looking at my current CEPH stats they are not even at 1-2 MiB/s. I dont think I am saturating network links yet because the tests that I did yield a speed of 110 MiB/s. Would adding SSDs improve performance more or adding 10Gb NICs improve performance more ?

Any help/insight is appreciated.
 
Would adding SSDs improve performance more or adding 10Gb NICs improve performance more ?

In your setup: SSD.

Monitor your current network throughput, e.g. with iptraf or some other interactive cli tool, but I suggest that you will not see saturation. First step would IMHO be an upgrade to SSD and then you'll maybe saturate your network throughput and can go one step further on that front.
 
Thanks @LnxBil , my next question would be would I use SSDs just as journal disk ? If so can I use a single SSD/server for all my OSDs and how to replace existing journal disk which is HDD. I have 2 journal disks in each server. Sorry if thats a lot of questions. I am just looking for right direction to look into.

This is probably peak performance snapshot plus/minus.

upload_2018-10-8_12-56-31.png
 
Thanks @LnxBil , my next question would be would I use SSDs just as journal disk ?

This is the first step to improve your performance, yes. Your pool will be much faster if you use SSD everywhere, but that'll most probably be too expensive.

But be aware, there are fast and there are slow SSDs, best is to buy from this list:

http://www.sebastien-han.fr/blog/20...-if-your-ssd-is-suitable-as-a-journal-device/

Your screenshot stats, that you write mostly and this is very, very costly on HDDs, so writing it first to SSDs is much faster.

If so can I use a single SSD/server for all my OSDs and how to replace existing journal disk which is HDD. I have 2 journal disks in each server.

It is said that 4-5 journals per SSD should be used, so you need to size them accordingly. If you use only one, it'll be slower, but still faster than using your pool without any SSDs, obviously.
 
Thank you for all the assistance. The link you mentioned is very confusing it mentions about WAL + DB which I have no idea about. Neither does it mention like concrete steps to determine right size for journal :(
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!