[SOLVED] VM replication to unclustered node or separate cluster

[phip]

Hi everyone,

we're currently running a small cluster (4 nodes + qdevice) in a single server room and would now like to physically move two nodes to a separate location for some georedundancy. Those two servers are mostly hot spares, replicating VMs and other data from the primary ones. Because the link between the two locations will naturally be slower and less stable than a direct switch connection, the cluster might no longer work reliably due to latency issues (>5ms), so we're thinking about either breaking up the cluster into two separate clusters or even removing the clustering entirely and running all nodes standalone.

With the new datacenter manager, I think there would be a nice and convenient way to migrate VMs between clusters/nodes, but what about replication? So far I've not found any resources about inter-cluster replication, even though this appears to be the simpler case than remote migration, especially on a ZFS backend. Of course we could implement the replication on our own using zfs send/recv, but given that Proxmox already includes the replication feature, it would certainly be preferable to use this.

Is there a way to do Proxmox VM replication across clusters or standalone hosts?

Thanks and best regards,
Philipp

 

[Johannes S]

You could use pve-zsync but it doesn't allow auto-failover aka HA:
https://pve.proxmox.com/wiki/PVE-zsync
So you would need to launch the vms on your offsite cluster manually in case of a failover event.
The Datacenter-Manager doesn't have it yet integrated , but it's on the roadmap: https://pdm.proxmox.com/docs/roadmap.html#proxmox-datacenter-manager-1-0
Regarding splitting up the cluster: You could do this, but I'm not sure whether the qdevice can be used for both clusters over that link. Although a qdevice has lower latency needs than the actual cluster network I'm not sure whether this will still work over a WAN:

Unlike corosync itself, a QDevice connects to the cluster over TCP/IP. The daemon can also run outside the LAN of the cluster and isn’t limited to the low latencies requirements of corosync.

https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

Do you happen to have a ProxmoxBackupServer on one or both of your locations? One PBS on each site would be the primary backup target of the onsite nodes and could also serve as local qdevice and sync-target for the other PBS. So each PBS would pull the backups from the other.
Alternatively an old PC or raspberry could serve as qdevice but I wouldn't trust backups with such hardware
One thing to be aware of: The qdevice will be configured in a way, that each node of the cluster can reach the qdevice via ssh without authentification.
To ensure that no bad actor can take over your backups from the qdevice (in case you want to host the qdevice and pbs on the same machine) a setup like the one described by Proxmox staff member Aaron here ensures strict seperation to protect against such a scenario:

Post in thread '2 Node HA Cluster'

Jan 10, 2022

What I maintain personally, are 2 powerful AMD EPYC nodes that run the infrastructure. The 3rd server is far less powerful in terms of CPU and memory and is using cheaper but quite a bit larger SSDs to be used as PBS for that 2-node cluster and also to remote sync another small PVE / PBS combo on the other side of the country.

On that 3rd machine I installed PVE first and PBS on top. PBS is the main functionality for that machine. On the PVE side, for now, the only guest is the qdevice container. That PBS server is there anyway, and using PVE on that one as well helps to keep using the...

• aaron

• 

 

[phip]

You could use pve-zsync but it doesn't allow auto-failover aka HA:
https://pve.proxmox.com/wiki/PVE-zsync
So you would need to launch the vms on your offsite cluster manually in case of a failover event.
The Datacenter-Manager doesn't have it yet integrated , but it's on the roadmap: https://pdm.proxmox.com/docs/roadmap.html#proxmox-datacenter-manager-1-0

Cool, thanks! I guess that's exactly what we'll need. And when it gets integrated into PDM, even better. HA is not something that we're after, so that's not a limitation. The only thing that we'll need to keep in mind is to update the replication jobs manually when we migrate a VM.

Regarding splitting up the cluster: You could do this, but I'm not sure whether the qdevice can be used for both clusters over that link. Although a qdevice has lower latency needs than the actual cluster network I'm not sure whether this will still work over a WAN:



Do you happen to have a ProxmoxBackupServer on one or both of your locations? One PBS on each site would be the primary backup target of the onsite nodes and could also serve as local qdevice and sync-target for the other PBS. So each PBS would pull the backups from the other.
Alternatively an old PC or raspberry could serve as qdevice but I wouldn't trust backups with such hardware
One thing to be aware of: The qdevice will be configured in a way, that each node of the cluster can reach the qdevice via ssh without authentification.
To ensure that no bad actor can take over your backups from the qdevice (in case you want to host the qdevice and pbs on the same machine) a setup like the one described by Proxmox staff member Aaron here ensures strict seperation to protect against such a scenario:
...

According to the qdevice repo README, a single qdevice can support multiple clusters:

[...]
It's designed to support multiple clusters and be almost configuration and state free.
[...]

https://github.com/corosync/corosync-qdevice

We already have a qdevice on an external and very much locked-down server in place, with WireGuard tunnels to protect the traffic. So I think nothing else will be required at this end. In case we decide to fully drop the cluster and operate all nodes standalone, this won't even be necessary at all anymore.

 

[Johannes S]

The only thing that we'll need to keep in mind is to update the replication jobs manually when we migrate a VM.

Exactly, I never did work with pve-zsync myself but following part of the wiki page gives a hint:

PVE Storage Replication and PVE-zsync work completely independent and do not harm each other, as long the following requirements are followed:

• The destination pool / subset are different.
• You do not migrate a guest to an other node.

So it's propably a good idea to build some automation/scripting/ansible/puppet/whatever solution around it to ensure that the zsync-job will be removed before the migration and recreated afterward. I would also use different storage pools for the regular cluster and the zsync-target, if possible to avoid that some rogue migration/replication wreak havoc on critical data. As said: No personal experience with it, so take this with a grain of salt

According to the qdevice repo README, a single qdevice can support multiple clusters:

Yes and that's not what I'm worrying about but whether it will still work if it also needs to serve an offsite location. How will your clusters+qdevice connected to each other? The qdevice doesn't have the strict latency requirements of the regular cluster but I would still expect problems if (just as extreme corner case example ) you are running it over a slow dial line.