Proxmox/Ceph cluster: Best practice to increase storage size?

[Tudeli]

Hi guys,

we are currently running a 6 node Proxmox/Ceph cluster (by the way: works like a charm, thanks a lot for this wonderful piece of software). Each node has 12 OSDs, the OSDs are 960 GB SSDs. What would be the best practice if you need to increase the storage size?

1.) Adding two identical nodes (each with 12 OSDs/960 GB)?
2.) Adding one node with 24 OSDs/960 GB?
3.) Adding one node with 12 OSDs/1.92 TB?
4.) Replacing the SSDs of two existing nodes with 1.92 TB models?

Thanks a lot in advance for your thoughts on this!

 

[aaron]

All options are viable.

From an operational and performance point of view would be number 1. Keeping the nodes as similar as possible in the long run will make mgmt a lot easier and.

Once you have nodes or OSDs with more capacity, they will get a higher weight and thus more data will be stored on them. From a performance POV this could become a bottleneck if they cannot deal with the additional load as they will be accessed more frequently than other nodes/OSDs.

If such an OSD or full node fails, it means that the rest of the cluster has to take up the lost replicas. If there is a huge imbalance, this could mean that a lost node or OSD can cause the remaining cluster to fill up.

Instead of just replacing the OSDs on one or two nodes, why not replace the same number of OSDs in all nodes? Then you still have an unbalance on the OSD level, but the nodes themselves should be weighted the same.

I hope this helps you to decide what you want to do, given the budget you have.

 

[Tudeli]

Instead of just replacing the OSDs on one or two nodes, why not replace the same number of OSDs in all nodes? Then you still have an unbalance on the OSD level, but the nodes themselves should be weighted the same.

Thanks a lot for you reply! Replacing 4 SSDs on each node with larger ones. I hadn't even considered this option and it perfectly fits our requirements.