Allflash Ceph

[Anderson Alves]

Hi guys,

I'm planing to build a 3 node Proxmox + Ceph with 6 Samsung 850 PRO 1TB, 2 SSDs (OSD) per node. I know that these SSDs are not good for Journaling, but in case of ALLFLASH setup I'm not gonna use Journaling. There will be 20 VMs running, 1 of these is a Graylog+ElasticSearch and 5 of these are Galera Cluster.

What do you think, is it a good plan?

Thanks in advance.

 

[tom]

No, this is NOT a good plan, never use consumer/workstation SSDs.
this 850 has only 300TB TBW, not suites for Ceph.

Better use enterprise class SSD (Intel or Samsung).

A reasonable priced Intel SSD: DC S3520 1.2TB, 2.5", SATA (SSDSC2BB012T701)

But before you make a final decision, do a detailed calculation of your total writes for the next years.

And use Ceph Luminous with Bluestore in this case, will be production ready in a few weeks. Small setups with SSDs benefits a lot from bluestore.

 

[spirit]

Hi guys,

I'm planing to build a 3 node Proxmox + Ceph with 6 Samsung 850 PRO 1TB, 2 SSDs (OSD) per node. I know that these SSDs are not good for Journaling, but in case of ALLFLASH setup I'm not gonna use Journaling. There will be 20 VMs running, 1 of these is a Graylog+ElasticSearch and 5 of these are Galera Cluster.

What do you think, is it a good plan?

Thanks in advance.

you still have journal on each osd, so it'll be slow.
Maybe a small fast ssd for journals could be a better solution, even if you have slow ssd for datas.

 

[TwiX]

Hi,

850 Pro is really shitty for Ceph => https://www.sebastien-han.fr/blog/2...-if-your-ssd-is-suitable-as-a-journal-device/

 

[lucaferr]

But before you make a final decision, do a detailed calculation of your total writes for the next years.

That's a very good advice when planning a SSD-backed ceph infrastructure. But how to estimate the Ceph overhead? Let's say that my VMs write 100 GB of effective data per day and my pool has size=2 and uses Ceph 12 with bluestore, with data, WAL and DB on the same device ...how many GBs will be written on every SSD per day?

 

[biax]

How's Samsung 960 PRO though in terms of value for money? I know it doesn't give optimal performance but if its half i suppose its good enough.

 

[Anderson Alves]

Could you please give me some more details as far as not being reliable?
Would it be due to low TBW?
We write almost 13GB per day...I could put Graylog+EL on another local storage with an HDD, so that my writes goes down to 5GB per day. If these Samsung 850 PRO last for 1 year, that'd be more than enough for me. That would give me the time I need to get better SSDs. What do you say?

Another question, I have 3 SSDs of 200GB from HP, they are rated as Class C for Endurance, with 3650 TBW. Will it be any good if I use these 200GB for journaling of 2 OSD of 1TB HDD 7200RPM?

By the way, here are the pieces that I already have played with:
3 x Dell R430 - 1x Xeon 2650v2 - 96Gb of RAm
3 x SSD 200Gb HP class C
6 x SSD 1Tb Samsung 850 PRO
6 x HDD 1Tb Dell SATA 7200RPM

Please help me to decide, I have to go in production ASAP...
Thank you in advance for any advices.

 

[biax]

Could you please give me some more details as far as not being reliable?
Would it be due to low TBW?
We write almost 13GB per day...I could put Graylog+EL on another local storage with an HDD, so that my writes goes down to 5GB per day. If these Samsung 850 PRO last for 1 year, that'd be more than enough for me. That would give me the time I need to get better SSDs. What do you say?

Another question, I have 3 SSDs of 200GB from HP, they are rated as Class C for Endurance, with 3650 TBW. Will it be any good if I use these 200GB for journaling of 2 OSD of 1TB HDD 7200RPM?

By the way, here are the pieces that I already have played with:
3 x Dell R430 - 1x Xeon 2650v2 - 96Gb of RAm
3 x SSD 200Gb HP class C
6 x SSD 1Tb Samsung 850 PRO
6 x HDD 1Tb Dell SATA 7200RPM

Please help me to decide, I have to go in production ASAP...
Thank you in advance for any advices.

The reason i believe is the 1 job speed as illustrated in the job blog above. I'd be satisfied with 250MB/sec for 1 job speeds if its more cost effective. I think Samsung 960 PRO and beyond should do slightly better although there are no tests out there so its only an assumption.