Combining two separate clusters

[troycarpenter]

Hi all.

In our lab, we maintain two separate but identical Proxmox clusters with Ceph. Each cluster has 5 compute nodes and 3 storage nodes, so 8 total members per cluster. The storage nodes are cluster members but do not host any VMs. Each storage node has 10 5TB drives (spinners...I'll wait for the groaning to stop).

The requirement to have two separate clusters has gone away, and now I want to combine the two. Better said, I want to eliminate one cluster and join the nodes to the other. Just using the raw hardware, that would be 6 Ceph storage nodes and 10 compute nodes for a total of 16 members. I've been reading recommendations for both Ceph nodes and total number of nodes where it relates to quorum and split brain and all (I only realized while writing this that each cluster already can be split brain...I've always thought of them in terms of 3 storage and 5 compute nodes). What is the recommendation/best practices for this type of operation? 5 or 6 Ceph nodes for performance? Dropping a Ceph node from 6 to 5 will also make the entire cluster odd again (15 total nodes).

(EDIT) One other consideration...In combing the storage nodes, I would like to replace the original 30 drives in the 3 storage servers with SSD and have a pool of fast storage and a pool of slower storage. Does that change anything?

I'm expecting that for the decommissioned hardware of the first cluster, I will simply reinstall Proxmox as if they were new hardware and join them to the other.

Thanks in advance.

 

[alexskysilk]

I've been reading recommendations for both Ceph nodes and total number of nodes where it relates to quorum and split brain and all (I only realized while writing this that each cluster already can be split brain...I've always thought of them in terms of 3 storage and 5 compute nodes). What is the recommendation/best practices for this type of operation? 5 or 6 Ceph nodes for performance? Dropping a Ceph node from 6 to 5 will also make the entire cluster odd again (15 total nodes).

Where did you read that? What keeps quorum in a ceph cluster are your monitors. you have 3 monitors, you have quorum. it doesnt matter how many nodes are in the cluster.

I'm expecting that for the decommissioned hardware of the first cluster, I will simply reinstall Proxmox as if they were new hardware and join them to the other.

yep. that is the way.

(EDIT) One other consideration...In combing the storage nodes, I would like to replace the original 30 drives in the 3 storage servers with SSD and have a pool of fast storage and a pool of slower storage. Does that change anything?

No, why would it? you can create new pools whenever you want, and you can specify per pool crush rules without disrupting other pools.