[SOLVED] Recommendation small Ceph setup

BJ78945

Well-Known Member
Apr 15, 2015
75
16
48
Hello,

at the moment I use GlusterFS in a low setup (EVO SSD and only 2GBE between the nodes). I want to increase performance and want to build a 3 node Ceph cluster for testing. I have the following devices:

6x SM883 960GB SSD
10x EVO 870 250GB SSD
12x different 300GB SAS 10k HDD

Alle three nodes will be connected with 40GBE DAC in a mesh setup.

I want to get a good performance. So I want to use two EVO SSD (RAID1) for the OS and the SM883 for OSD. I think I don't use the SAS drives. I read that 12 OSDs will be recommended. Also see in the ceph docs, that for faster drives more than one OSD could bring better performance. Is it a good idea to setup 2 OSD per SM883 drive, so I have 12 OSDs in the three node setup? Or perform it better to use the 12 SAS and for 2 SAS one SSD for journal? Or could it also be better to use the cheap EVO SSD as OSD and the SM883 as journal?

If all of them are not really a good choice, which drives you could recommend? 1,5TB useable storage would be enough for the beginning. Per node I have space for 8 2,5" devices.

Best regards,

Alex
 
Better use 2 of the SAS HDDs for OS instead of the consumer SSDs as those would wearout pretty quick.

I'd use 2 ceph pools, one using the SM883's as those are proper datacenter drives and another pool using the EVOs.
Just be aware that the EVOs may not live very long, depending on your write load, also the SM883's will probably have a way more stable performance.
 
Thank you very much for your answer. One question left. Is it a good idea to put 2 OSD on one SM883? Or should I buy more SM883 to become 4 OSD per node? I saw that at least 12 OSD should be used.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!