Ceph: Recommended size for caching SSD in a hybrid cluster

[Nyctophilia]

Hello,

I've been researching through different places to size our upcoming proxmox cluster which will also be running ceph.

Unfortunately I didn't find any obvious overlapping information. I was looking in the official ceph documentation, red hat KBs, different Proxmox posts and so on.

Is there a quick rule of thumb to calculate the caching size for one node?

For example if I have 4 nodes with 4x 1,2TB SAS drives per node and replication with the factor 2, is a Datacenter SSD with 400GB enough?

Thank you in advance

 

[sg90]

Caching is not recommend in CEPH and is slowly being filtered out of the software over time.

What you can do with Bluestore is place the WAL & DB of the OSD onto a SSD, so the metadata is retrieved quickly via the SSD and the SAS disk is left with just the raw data I/O this can be read about here : https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/

1. This is only recommend if you have a high end DC SSD with data loss protection and high throughput at different Q depths, otherwise if you have 4 OSD's on one SSD you may expirence slower performance than just letting the single disk complete all I/O.
2. You will lose all 4 OSD's when the SSD die's, so this needs to be taken into consideration with amount of OSD's and free disk space left.
3. It is highly highly recomened to run at a replication of 3 and never ever 2, your asking for data corruption and loss with a replication of 2.

Thanks

 

[Nyctophilia]

Thank you for your detailed response.

So if I get this right: If caching is not recommended practice anymore, so then you should go all flash? Or a mixed construct of ssd and hdd pools?

 

[sg90]

Thank you for your detailed response.

So if I get this right: If caching is not recommended practice anymore, so then you should go all flash? Or a mixed construct of ssd and hdd pools?

Only you can answer this with the performance / amount of storage you require. If you need storage then SAS disk's with the DB + WAL on a good DC rated SSD or even better NVME is your best option.