Adding external server to Proxmox CEPH Cluster

[alvaroag]

Hi. We are having some expension problems related to our Proxmox Cluster. First, the current situation:

We currently have 3 servers in North America, and 3 more in Europe. All of those are in a good datacenters from the same ISP, with good bandwidth(1Gbps or more), with private network between all of them, and all of them conform a single Proxmox Cluster. We also have another server in our home country, in Aouth America, with a more limited internet connection. This server also runs Proxmox, but is not part of the Cluster. Probably, in a few months, it will be part of a cluster with other local servers, but never part of the same cluster as the other servers.

In the PVE cluster we have deployed CEPH. As well as storing some HA VMs with CEPH, we have setup CEPH FS to have some replicated, multimaster filesystem across all of those servers. At that point, everything works fine.

Now to the problem We need our local server to be able to access those replicated CEPH FS. For a long time, we have managed that just by makign a full copy of those files from time to time. But now we need a better integration, preferably real time. Adding our local server (or servers, in the future) to the cluster may not be an option, because our local link to the cluster network is based on OpenVPN, which may not be adequate for PVE Cluster. Using oru local server as CEPH client is also not a possibility, because of the high latency between our location and any of the datacenters.

So, is there any way to add a server to the PVE CEPH Cluster, without it being part of the PVE Cluster?

Any help is really appreciated.

 

[Alwin]

So, is there any way to add a server to the PVE CEPH Cluster, without it being part of the PVE Cluster?

Hm... no. Being part but not being part. Sounds more for a case of rsync (or similar replication).

Please consider the below, food for thought, as it is not directly related to your question.

and all of them conform a single Proxmox Cluster.

With Ceph and Corosync, the 1 GbE will be a bottleneck, irregardless of the latency. This is no setup for HA.

Corosync needs low and stable latency, in the latency range of a LAN. The below links show latency to their services or ping round trips.
https://enterprise.verizon.com/terms/latency/
http://www.azurespeed.com/
https://wondernetwork.com/pings

In general, spanning a cluster over different locations, especially where you don't have the infrastructure under control, is not a good idea. In your case of two locations, while it may work now, any service disruption may render both locations unusable (eg. split brain). Better run two clusters, that run independently (separation of concerns). This has the benefits that a issue on one location will not directly affect the other, maintenance can be done separately and split brain scenarios are less likely.

To replicate RBD images between those clusters, you can run rbd-mirror. This should reduce the overall bandwidth requirement, as the replication traffic of the OSDs is not going through the WAN link anymore. rbd-mirror only syncs diffs of a rbd image.
https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring

 

[dcsapak]

why not simply mount the cephfs as a client, this way it does not participate in either the ceph nor pve cluster, but you have still access to the data

 

[alvaroag]

Hm... no. Being part but not being part. Sounds more for a case of rsync (or similar replication).

The problem with rsync is that it is not real time. It is more similar to our current tar method, just more automatic; but we are currently needing something real time, or near to that.

Please consider the below, food for thought, as it is not directly related to your question.

With Ceph and Corosync, the 1 GbE will be a bottleneck, irregardless of the latency. This is no setup for HA.

Corosync needs low and stable latency, in the latency range of a LAN. The below links show latency to their services or ping round trips.
https://enterprise.verizon.com/terms/latency/
http://www.azurespeed.com/
https://wondernetwork.com/pings

In general, spanning a cluster over different locations, especially where you don't have the infrastructure under control, is not a good idea. In your case of two locations, while it may work now, any service disruption may render both locations unusable (eg. split brain). Better run two clusters, that run independently (separation of concerns). This has the benefits that a issue on one location will not directly affect the other, maintenance can be done separately and split brain scenarios are less likely.

To replicate RBD images between those clusters, you can run rbd-mirror. This should reduce the overall bandwidth requirement, as the replication traffic of the OSDs is not going through the WAN link anymore. rbd-mirror only syncs diffs of a rbd image.
https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring

Latency is pretty good between our current datacenters, about 80ms ping. Under normal circumstances, the HA VMs would really be a problem, as the bandwidth is not enough. That's why our HA VMs relay only on database operations, and write nothing to it's own disk, so rootfs is mounted read only. However, those may sometimes write to the shared filesystem. Not the best solution, but suits our needs fine.

BTW, I forgot to say that the servers have their on dedicated NIC to communicate between each one, and it is separated from the internet NIC. So, even with high internet traffic, the servers have their internal traffic bandwidth guaranteed.

why not simply mount the cephfs as a client, this way it does not participate in either the ceph nor pve cluster, but you have still access to the data

That would fit our needs, if it wasn't for the latency between our home location and the datacenters... 120ms ping in best cases against Northamerican DC, and even more for the European one.


Can't I just try to add the external server to CEPH cluster manually, without the Proxmox management interface? Is this even possible?

 

[alexskysilk]

That would fit our needs, if it wasn't for the latency between our home location and the datacenters... 120ms ping in best cases against Northamerican DC, and even more for the European one.

I suppose you should describe what your needs are in this context. since cephfs presents a normal posix file system (which you could access with a client as @dcsapak recommended, or even simpler via nfs or sshfs; how would latency interfere with your remote access?

 

[DANILO MONTAGNA]

Proxmox CEPH is just a CEPH storage like any other environment that uses CEPH as base storage system.. so, basically, you just need to connect your external ceph client to proxmox ceph storage... just pay attention to the requirements, like allow the correct access on the keyring file in order to mount the ceph storage remotely, etc... like any other CEPH install..

 

[dcsapak]

@alvaroag it does not make sense to add the node to the ceph cluster if the latency is bad, this will only make the performance for all clients worse

ceph does not simply replicate the contents synchronously, but distributes them around the nodes, which means that even if you would add the node to the cluster, it does not mean
that the data is all there and each access has to be on the network anyway

also in that case all writes which would land on that node would have very bad latency -> worse performance overall

as the other have said, mounting it via cephfs/nfs/etc is probably your best bet to achieve what you want

 

[alvaroag]

Hi. THanks for the comments.

I hadn't noticed the local infrastructure may not be optimal for CEPH. Also, I underestimated my local connection, and NFS access seems to be pretty fast. I'm now trying to check how to deal with availability. I'll probably deal with it using multiple endpoints, or something like that.

I also realized that my CEPH versions were different in my remote servers(all with hammer) and my local servers(all with nautilus), so that was likely to cause issues anyway. And upgrading the remote servers can be a real pain.

Thanks everyone for your comments