cluster network setup

[kyriazis]

Hello...

I have a Proxmox cluster with about 20 nodes. I am running Ceph on it, but I only have OSDs on 4 nodes. I have, however, installed Ceph on all nodes. Currently (sadly) I am not using a cluster network, so all the Ceph balancing traffic goes to the public network.

I'd like to add a cluster network, but here is my question: Do I have to add all nodes to the cluster network, or just the ones that have OSDs on them? Our future plans is to always have more compute/VM nodes than storage (and by that I mean that the storage nodes also host VMs).

Thank you!

George

 

[aaron]

https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/ has a nice diagram showing which Ceph parts use which network.

The Ceph cluster network is used for the replication between the OSDs, therefore your compute nodes do not need access to it as they communicate via the Ceph public network with the nodes holding the OSDs, Monitors, etc.

 

[kyriazis]

Thanks Aaron, I've already read the document that you pointed me at, but the text was vague as to which nodes actually need to be on the cluster network. Diagrams don't always show the whole picture.

Thanks for the clarification.