Proxmox/Ceph Cluster HW Recommendation

[faridz88]

Hi all,
We are planning a cluster which consists of 3 BMs to use as storage nodes running CEPH and other 16 as compute nodes each running ~30 VMs on proxmox.
What is your opinion on this setup and configuration?

Storage Nodes:
MB/Chasis: SuperMicro 2028R-ACR24H /L (2U, 24 2.5" bays)
Memory: 64 GB
CPU: 2660
SSD: 24 * 1 TB Samsung Evo 860 [Pro] / 960 GB PM1643 (Enterprise)
HBA: 3 * onboard Broadcom 3108

Compute Nodes:
MB/Chasis: SuperMicro SYS-6018R-TD (1U)
CPU: E5-2680 V3
Memory: 96 GB
NIC: Intel 10G SFP+Dual Port (x540 DA2)

Network:
Storage: Nexus 3064-X (48 * 10G SFP+, 4 * 40G SFP+) / Nexus 3172 switches (48 * 10G SFP+, 6 * 40G SFP+)
Public Access: Same as above


Regards,
Farid

 

[RokaKen]

Your hardware looks fine. Note that some members have been forced to disable the HPET in BIOS on SuperMicro X11 boards, but I believe the ones you list come with X10 boards. YMMV, but accurate and precision time is a requisite.

 

[wahmed]

If you are putting those Ceph nodes on 40g SFP+ for Sync cluster then you will be fine. Otherwise you are going to saturate your 10g network with 24 SSDs per node during rebalancing. This chassis comes with dual 10g so I am assuming you are going to have 2 separate networks for Ceph Public and Sync cluster.

I am sure you already know this, Ceph performs better with higher count nodes. Even though you are on SSDs, ideally you would add 1 or 2 extra nodes and distribute the SSDs to achieve as much performance as possible. If you end up using just 3 nodes with 24 SSDs in each on 10g SFP, I would highly recommend to up the memory on the ceph nodes and tweak your ceph recovery to really limit network bandwidth usage.

Are you going to be using 2 or 3 replicas?