I'm in need of some help from some of the seasoned professionals out there. I'm setting up a 5-node cluster with Ceph that will run around 30-40 VMs. Each node has two 4-port 10G NICs. I'm going to use LACP to bond one port from each NIC to create four connections on each server. However, this cluster will also be running VMs for a different department that has its own separate network infrastructure and will need two connections to be used for VM network traffic.
I've got two main ideas for configuring the networks, but I'm not sure which will be the best.
Option #1
Basically, both setups are easy to implement and don't make much difference. I prefer the first setup, as I think it is more cleanly separated and you only need a separate 20G for your Ceph Private network if you've really high ceph client traffic.
The only important thing here is that you make QoS settings on your switch so that the corosync traffic always has the highest priority and there are no latency peaks. Normally we recommend using a separate 1G switch for corosync which is often easier than configuring QoS, if possible.
I would also recommend configuring one of the other networks as a corosync backup network [0], it doesn't matter which one you use.
