Proxmox/Ceph Cluster HW Recommendation


Jun 21, 2016
Hi all,
We are planning a cluster which consists of 3 BMs to use as storage nodes running CEPH and other 16 as compute nodes each running ~30 VMs on proxmox.
What is your opinion on this setup and configuration?

Storage Nodes:
MB/Chasis: SuperMicro 2028R-ACR24H /L (2U, 24 2.5" bays)
Memory: 64 GB
CPU: 2660
SSD: 24 * 1 TB Samsung Evo 860 [Pro] / 960 GB PM1643 (Enterprise)
HBA: 3 * onboard Broadcom 3108

Compute Nodes:
MB/Chasis: SuperMicro SYS-6018R-TD (1U)
CPU: E5-2680 V3
Memory: 96 GB
NIC: Intel 10G SFP+Dual Port (x540 DA2)

Storage: Nexus 3064-X (48 * 10G SFP+, 4 * 40G SFP+) / Nexus 3172 switches (48 * 10G SFP+, 6 * 40G SFP+)
Public Access: Same as above

Your hardware looks fine. Note that some members have been forced to disable the HPET in BIOS on SuperMicro X11 boards, but I believe the ones you list come with X10 boards. YMMV, but accurate and precision time is a requisite.
If you are putting those Ceph nodes on 40g SFP+ for Sync cluster then you will be fine. Otherwise you are going to saturate your 10g network with 24 SSDs per node during rebalancing. This chassis comes with dual 10g so I am assuming you are going to have 2 separate networks for Ceph Public and Sync cluster.

I am sure you already know this, Ceph performs better with higher count nodes. Even though you are on SSDs, ideally you would add 1 or 2 extra nodes and distribute the SSDs to achieve as much performance as possible. If you end up using just 3 nodes with 24 SSDs in each on 10g SFP, I would highly recommend to up the memory on the ceph nodes and tweak your ceph recovery to really limit network bandwidth usage.

Are you going to be using 2 or 3 replicas?


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!