Ceph with mixed hardware and disks

[AllYourSmurf]

I'm rebuilding my cluster and I'd like to take a stab at Ceph. Here's what I Have to work with:

6 servers, all dual-xeons with at least 6 cores (24 threads). min 128GB ram.
Disks are where things get weird:

1. 7 x 3TB platter (21tb)
2. 5 x 2TB platter (10tb)
3. 5 x 2TB platter (10tb)
4. 1 500GB platter + 8 x 1.5 TB nvme (12tb)
5. 7 x 3TB nvme (21tb)
6. 4 x 750GB nvme (3 TB) with room to grow.

My plan is to link everything up with 25gb dedicated NICs for the ceph network.
Is this unreasonable?

 

[bl1mp]

Hi AllYourSmurf
if you are able to do so, you should distribute your nvmes more equally across, the nvme nodes or even better across all nodes.

With the current distribution only 3TB of your NVME-Storages will be usable (with the default 3-2 fault tolerance), because a copy of every PG needs to be on a different node, and node 6 can only provide 3TB.

To calculate the speed for the network, that might depend on the I/O expectations, maybe now and in future. Just be aware that every write operation will be send to two other nodes (and acknowledged) before it is considered as successful. So either your storage devices or your network will be a "bottleneck". In a mixed setup with different types of HDDs and NVMEs that can also impact the performance.

25G sounds reasonable to start with and for the reason you have 6 nodes you will most likely utelize a switch based setup, so maybe you can just add a LACP-Bond later, if your network is to slow.

BG, Lucas

 

[alexskysilk]

It depends on your performance expectations, but ceph is quite capable to deal with this kind of mixed hardware- as long as you make sure that you have EQUIVALENT CAPACITY PER OSD TYPE on each osd server.

In your case, assuming you want all 6 nodes to be OSD bearing, you'd want to divide your 41TB of HDD to roughly 6-7TB per node, and your 35TB of NVME into roughly 6TB per node. and remember to set a device class to your pools.

 

[AllYourSmurf]

Thank you both. This helps a lot. I'll be looking at hardware capabilities next to see if that kind of re-balancing is possible. I don't have the switch yet, so I have time to work out the storage layout.