Struggling to create Ceph OSD on Node that is not apart of same Gateway/LAN (4 Nodes, Different LAN)

[CheerfullyFalse]

Hi everyone,

I have 4 nodes in my Proxmox Datacenter. Two nodes are in North America and two are in Asia, thus they are both in different geographical locations and networks.

• Node 1 (10.10.100.1)
• Node 2 (10.10.100.2)
• Node 3 (10.100.200.1)
• Node 4 (10.100.200.2)

I have successfully setup Ceph and the required OSD's for the nodes in Asia but I would like to obviously add the nodes in North America to the cluster. Currently I get an error saying they are not on the same cluster network:

No address from ceph cluster network (10.10.100.0/24) found on node 'pve-3'. Check your network config. (500)

How should I go about solving this issue, I know that multi-site configuration with Ceph is possible but I'm not sure how it integrates with Proxmox.

 

[Maximiliano]

Hello,

Do note that when using Corosync or Ceph we recommend setups where between any pair of nodes you have fewer than 5ms of ping. Optimally, you want the ping to be lower than 1ms. At the current moment and in some scenarios, around ~10ms you can already see nodes dropping out of the cluster and HA guest fencing themselves.

As a reference, Just by taking the distance between California and Japan, even if you had a direct optic fiber connection you would have a ping of ~85ms in perfect conditions, without even taking into account any other kind of overhead. Out of curiosity whats the latency you see between each location, how stable is the connection?

This setup won't work. What do you want to use this storage for? If you want it for VM storage note that each IO operation will do *multiple* round trips on the cluster and latency will quickly add up.

I would advice to look into other alternative for storage replication between the two locations. See [1] or [2] where in the last case you would have to sync VM configs manually as of today.

On another note, clusters with a even number of nodes are discouraged as you can have a situation of a vote tie, both partitions of the cluster will be out of quorum. In this case write operations to /etc/pve, starting/stoping VM, and other operations will be blocked. You have to setup a QDevice for these scenarios [3] to have an odd number of votes without the need of adding another node to the cluster.

[1] https://pve.proxmox.com/wiki/Storage_Replication
[2] https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring
[3] https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support