3 nodes Ceph config with public ip

[faisa7847]

Hey Everyone i have a 3 node cluster with proxmox 8.2.2 they have public IP assigned three servers in three different locations
my problem is I have installed ceph but the public IP on each of the server is different and has different network how do I resolve this and create a the ceph cluster ..and if not how do I create a HA storage accrose these three nodes ..i have tried NFS but it takes lot of time for migration

 

[tcabernoch]

OMG are they gonna rip you up for a 3 node ceph cluster. That's not gonna get a lot of approval from the cranky old guys. You need more nodes.

There is absolutely no way you'll get ceph to work over the WAN. Forget it.

I don't think this cluster-over-the-WAN is going to work either. The latency will be too high. Your cluster will immediately lose quorum.

You need to completely reconsider this plan. Start with this wiki page.
Really, go here and start reading. Its gonna answer a lot of questions, and leave you primed for more questions.
https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

Network Requirements

The Proxmox VE cluster stack requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably. While on setups with a small node count a network with higher latencies may work, this is not guaranteed and gets rather unlikely with more than three nodes and latencies above around 10 ms.

The network should not be used heavily by other members, as while corosync does not uses much bandwidth it is sensitive to latency jitters; ideally corosync runs on its own physically separated network. Especially do not use a shared network for corosync and storage (except as a potential low-priority fallback in a redundant configuration).

Before setting up a cluster, it is good practice to check if the network is fit for that purpose. To ensure that the nodes can connect to each other on the cluster network, you can test the connectivity between them with the ping tool.

 

[kellogs]

Hey Everyone i have a 3 node cluster with proxmox 8.2.2 they have public IP assigned three servers in three different locations
my problem is I have installed ceph but the public IP on each of the server is different and has different network how do I resolve this and create a the ceph cluster ..and if not how do I create a HA storage accrose these three nodes ..i have tried NFS but it takes lot of time for migration

This is a bad design. Unless your wan has super good latency and capacity otherwise don’t bury yourself into this grave, it is not worth it.

Setup ceph local and use pbs to backup and another pbs in remote location

 

[faisa7847]

The WAN ha s not much latency as its isolated the problem I get here is only while adding the monitors and managers
The setup looks like this
node1 ip pubip-1/30
node 2 ip pubip-2//19
node 3 pubip-3//30

i have installed ceph and configured on node 1
now when I try to add other monitors and for node 2 and 3 I get this error

No active IP found for the requested ceph public network 'pubip-3/30' on node 'node3' (500)

 

[faisa7847]

OMG are they gonna rip you up for a 3 node ceph cluster. That's not gonna get a lot of approval from the cranky old guys. You need more nodes.

There is absolutely no way you'll get ceph to work over the WAN. Forget it.

I don't think this cluster-over-the-WAN is going to work either. The latency will be too high. Your cluster will immediately lose quorum.

You need to completely reconsider this plan. Start with this wiki page.
Really, go here and start reading. Its gonna answer a lot of questions, and leave you primed for more questions.
https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

The WAN used for CEPH is Just for CEPH other parts are not using it as we have setup internal firewalls for oubound traffic and inbound traffic for the services