Ceph Bluestore with Consumer SSD DBs?

[jw6677]

Hello,

I am looking at reworking my ceph setup. I have 26 8TB SAS drives, and independantly I have 16 inexpensive consume SSD drives, all uniformly two models.

Before I go through the setup, I wanted to double check if it makes sense to use these ssds as bluestore db+wal devices, or if I am better off leaving them for a different purpose.

I ask, as I know enterprise SSD drives would be preffered, and I also believe the uniformity would be lost with 10 SAS drives without an SSD.

Is the consensus to avoid this setup, or use the drives since I have them?

 

[Alwin]

Well, it depends on your use case. Consumer grade SSDs usually don't perform well and putting them as DB/WAL will probably slow down the SAS disks. But as you said, since you have them, you can just try and benchmark performance.

 

[jw6677]

Interesting, while trying to get away from using SSDs on an OSD node, it seems my performance really tanked. Down to <100MB/s recovery and benchmark rates (compare to ~1000MB/s when using the combined flash storage). I am currently (slowly) rebuilding everything and will be stuck recovering for a couple of weeks.

Just planning out some updates, A question:
My understanding is that consumer SSD's are terrible journal devices because of their slow speed of direct writes at queue depth 1:
Is there any group consensus on the use of consumer SSDs for LVM caching for OSD block storage specifically, while DB+WAL is on an commercial SSD?

It seems to me that OSD blocks do not write direct and QD1 like bluestore DBs, so should avoid this problem with consumer SSDs, is that correct?

 

[Alwin]

Is there any group consensus on the use of consumer SSDs for LVM caching for OSD block storage specifically, while DB+WAL is on an commercial SSD?

I can't recommend them pre se for Ceph. But I also never tried such a setup, or had to find a use-case for it.

It seems to me that OSD blocks do not write direct and QD1 like bluestore DBs, so should avoid this problem with consumer SSDs, is that correct?

DB/WAL get a lot of small writes/reads. While the block part of the OSD gets mostly big writes/reads. On separating those two types, the performance may be increased as a whole. The important part is to have a stable performance, even if its not a good one.