Ceph: Recommended size for caching SSD in a hybrid cluster

Nyctophilia

New Member
Aug 28, 2019
9
0
1
Hello,

I've been researching through different places to size our upcoming proxmox cluster which will also be running ceph.

Unfortunately I didn't find any obvious overlapping information. I was looking in the official ceph documentation, red hat KBs, different Proxmox posts and so on.

Is there a quick rule of thumb to calculate the caching size for one node?

For example if I have 4 nodes with 4x 1,2TB SAS drives per node and replication with the factor 2, is a Datacenter SSD with 400GB enough?

Thank you in advance
 
Caching is not recommend in CEPH and is slowly being filtered out of the software over time.

What you can do with Bluestore is place the WAL & DB of the OSD onto a SSD, so the metadata is retrieved quickly via the SSD and the SAS disk is left with just the raw data I/O this can be read about here : https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/

  1. This is only recommend if you have a high end DC SSD with data loss protection and high throughput at different Q depths, otherwise if you have 4 OSD's on one SSD you may expirence slower performance than just letting the single disk complete all I/O.
  2. You will lose all 4 OSD's when the SSD die's, so this needs to be taken into consideration with amount of OSD's and free disk space left.
  3. It is highly highly recomened to run at a replication of 3 and never ever 2, your asking for data corruption and loss with a replication of 2.

Thanks
 
Thank you for your detailed response.

So if I get this right: If caching is not recommended practice anymore, so then you should go all flash? Or a mixed construct of ssd and hdd pools?
 
Thank you for your detailed response.

So if I get this right: If caching is not recommended practice anymore, so then you should go all flash? Or a mixed construct of ssd and hdd pools?

Only you can answer this with the performance / amount of storage you require. If you need storage then SAS disk's with the DB + WAL on a good DC rated SSD or even better NVME is your best option.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!