AUG/2024 - New Proxmox Ceph Node Hardware Config

[kellogs]

Hello Guys,

Any comment on the following hardware for brand new Proxmox ceph cluster (storage only)

Chassis: Supermicro 1114CS-TNR-EU

CPU: Single AMD EPYC Milan 7313P 16C/32T 3.70GHz/3.00GHz 155W

RAM: 8x Samsung 16GB DDR4-3200 ECC RDIMM 2Rx8 1.2V M393A2K43DB3-CWE M393A2K43DB3-CW

OS Disk: 2x Micron 7450 PRO M.2 480GB PCIe 4.0 x4 NVMe 0.9DWPD 22x80mm MTFDKBA480TFR-1BC1ZABYY

Data Disk: 8x Samsung PM9A3 2.5in 3.8TB U.2 PCIe 4.0 x4 NVMe 1DWPD 7mm MZQL23T8HCLS-00A07

Network: Supermicro AIOM Dual Port 100GbE QSFP28 based on Broadcom BCM57508

 

[Blueloop]

• How many nodes?
• What sort of switches?
• How much storage do you need and will it need to flex?

You probably don't need 128GB of RAM on each node, however it depends if you have three nodes or 100 and the RAM is a relatively small part of the unit cost.

I'd get a spreadsheet out and try to model this thing if there are more than three nodes involved. Also I think there are some sizing tools on the Ceph website.

 

[kellogs]

I am thinking to start with 8 nodes and a pair 100G stacked switches, the storage for now i only need 50TB

 

[alexskysilk]

rule of thumb on sizing:

1 core per daemon (mon/mgr/osd)
4GB ram per daemon (strictly speaking, monitors will need ram proportionate to the size of the cluster, anywhere from 2-16GB but 4 is probably safe)
Save 4 cores and ~4GB of ram for the underlying OS. it will probably be idle, but better have and not need then the other way around.