I have been experimenting with ceph RBD mirror, and had some initial difficulty understanding the network requirements and how to meet them in my specific environment. Since first posting I have made some progress so I am updating this post to reflect what I have learned in the hope that it may help anyone else looking at ceph RBD for the first time.
Point 1. The key thing I learned is that the two clusters must be able to reach each other "over the network(s) defined in ceph.conf".
In my case I have two hyperconverged clusters running ceph. Each cluster has 3 nodes, separate full mesh (triangular ) ceph public and cluster networks, and redundant corosync. The clusters are physically identical in terms of hardware spec.
Important to note is that the ceph public, cluster (nor corosync) networks are not physically interconnected between the two clusters, although they do use the same subnets (and nodes do have unique IP address in those subnets).
Due to the standalone full mesh networks in each cluster I needed to find a way to satisfy point 1.
Both clusters are connected on the same management interface network, and nodes can reach each other across this network, so for the purposes of an experiment, in each cluster I added static routes for traffic targeting the specific ceph public/cluster node addresses of the peer cluster, in order to route route this traffic via the management LAN..
This worked perfectly.
Obviously I don't plan to use my management network for rbd mirroring, so my plan is to setup a point to point LAN connection between a node in each cluster with its own subnet, and set the static routes to use that.
Point 1. The key thing I learned is that the two clusters must be able to reach each other "over the network(s) defined in ceph.conf".
In my case I have two hyperconverged clusters running ceph. Each cluster has 3 nodes, separate full mesh (triangular ) ceph public and cluster networks, and redundant corosync. The clusters are physically identical in terms of hardware spec.
Important to note is that the ceph public, cluster (nor corosync) networks are not physically interconnected between the two clusters, although they do use the same subnets (and nodes do have unique IP address in those subnets).
Due to the standalone full mesh networks in each cluster I needed to find a way to satisfy point 1.
Both clusters are connected on the same management interface network, and nodes can reach each other across this network, so for the purposes of an experiment, in each cluster I added static routes for traffic targeting the specific ceph public/cluster node addresses of the peer cluster, in order to route route this traffic via the management LAN..
This worked perfectly.
Obviously I don't plan to use my management network for rbd mirroring, so my plan is to setup a point to point LAN connection between a node in each cluster with its own subnet, and set the static routes to use that.
Last edited: