Which configuration is better for a 3-node Proxmox/Ceph cluster?

[Neo42TheOne]

I need to set up a 3-node cluster to support about 20 VM's, most are databases for a mid-size company.
I plan on ordering 3 identical servers from Supermicro.

Model: AS -1115HS-TNR
CPU: AMD EPYC™ 9535 Processor 64-Core 2.40GHz 256MB Cache (300W)
RAM: (12) 64GB DDR5 6400MHz ECC RDIMM Server Memory
HD: (12) 7.68TB 2.5" CD8-P NVMe PCIe 5.0 Solid State Drive (1 x DWPD)

Networks:
(2) 10GBe switches for the corosync/management/VM networks uplinked via 2 fiber optic cables to our (2) core switches (one link per switch)
(1) 100GBe switch for the Ceph network

Is this a sane setup?
Should I go with the 128-core CPU and 384GB RAM? It would be about the same cost each.
Do I need another redundant switch for the 100GBe network?

 

[lucavornheder]

Hi,
regarding the CPU an RAM sizing it depends on your databases.

The Network seems fine, i would size the switch redundant.

 

[MarkusKo]

if you want your databases to be high performance don't put the db vm's on ceph or zfs storage
if you need HA for the databases do that on database level, not on vm level (for performance reasons)
if the databases are only moderately used it could be ok, but that really depends on your needs.

you should add two smaller ssd's dedicated just for proxmox ve installation and the other ssd's for vm storage exclusively

if you want to use ceph then you should use 5++ nodes, 3 is the absolute minimum and should only be considered for home labs / testing / non-ciritcal workloads, not for mid sized business workloads

and yes, network has to be redundant, especially for ceph or HA

*edit*
some more context: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

 

[Neo42TheOne]

if you want your databases to be high performance don't put the db vm's on ceph or zfs storage
if you need HA for the databases do that on database level, not on vm level (for performance reasons)
if the databases are only moderately used it could be ok, but that really depends on your needs.

you should add two smaller ssd's dedicated just for proxmox ve installation and the other ssd's for vm storage exclusively

if you want to use ceph then you should use 5++ nodes, 3 is the absolute minimum and should only be considered for home labs / testing / non-ciritcal workloads, not for mid sized business workloads

and yes, network has to be redundant, especially for ceph or HA

*edit*
some more context: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

The databases should be ok as they currently are located on VMware VM's and the systems can withstand a short outage.
I also agree with the 5 nodes.. my original plan was for 5, but management nixed it.

 

[Neo42TheOne]

This is starting to turn into a project! SuperMicro called and said they don't have the 7.6TB drives I ordered and they want to replace them with (6) 15.3TB instead.
Will the drive performance take a big hit or should I look elsewhere for the smaller drives?

 

[MarkusKo]

I'm not an expert on ceph or enterprise setups but i think with 6x 15TB PCIe5 NVMe SSD's you will be fine.

The main bottleneck will be ceph networking, 100Gbit is in theory 12,5GB/s, even if you have 2x100GBit LACP you get at best 25GB/s.
Each drive can handle about 12GB/s so in theory even if only 2 drives are used at the same time the network will already be the bottleneck.

Just make sure you plan your ceph setup, there are many things to consider - amount of nodes, osd's, monitors, redundancy level, ceph replication network, ceph public network and more ...

maybe check out the offical benchmarks

 

[Neo42TheOne]

I'm not an expert on ceph or enterprise setups but i think with 6x 15TB PCIe5 NVMe SSD's you will be fine.

The main bottleneck will be ceph networking, 100Gbit is in theory 12,5GB/s, even if you have 2x100GBit LACP you get at best 25GB/s.
Each drive can handle about 12GB/s so in theory even if only 2 drives are used at the same time the network will already be the bottleneck.

Just make sure you plan your ceph setup, there are many things to consider - amount of nodes, osd's, monitors, redundancy level, ceph replication network, ceph public network and more ...

maybe check out the offical benchmarks

Thanks for that link... it was very helpful! I decided to reconfigure for 5 nodes instead. Everything I have read seems to indicate that should be the minimum. Networking is going to be the issue now as the switches I planned on using for the Ceph network only have 6 ports each. This keeps getting more convoluted as I go along!