Ceph mirror network requirements


Mar 11, 2020
I have been experimenting with ceph RBD mirror, and had some initial difficulty understanding the network requirements and how to meet them in my specific environment. Since first posting I have made some progress so I am updating this post to reflect what I have learned in the hope that it may help anyone else looking at ceph RBD for the first time.

Point 1. The key thing I learned is that the two clusters must be able to reach each other "over the network(s) defined in ceph.conf".

In my case I have two hyperconverged clusters running ceph. Each cluster has 3 nodes, separate full mesh (triangular ) ceph public and cluster networks, and redundant corosync. The clusters are physically identical in terms of hardware spec.

Important to note is that the ceph public, cluster (nor corosync) networks are not physically interconnected between the two clusters, although they do use the same subnets (and nodes do have unique IP address in those subnets).

Due to the standalone full mesh networks in each cluster I needed to find a way to satisfy point 1.

Both clusters are connected on the same management interface network, and nodes can reach each other across this network, so for the purposes of an experiment, in each cluster I added static routes for traffic targeting the specific ceph public/cluster node addresses of the peer cluster, in order to route route this traffic via the management LAN..
This worked perfectly.

Obviously I don't plan to use my management network for rbd mirroring, so my plan is to setup a point to point LAN connection between a node in each cluster with its own subnet, and set the static routes to use that.
Last edited:

Did you get it working?
Also question about point 1. Should it not be enough for Ceph rbd mirror to reach both clusters?

Like VLAN1 (network1,cluster1) <-> RBD-mirror daemon <-> VLAN2 (network2,cluster2).


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!