Simplest redundancy setup for two servers

[hendr1x]

Hello everyone,
I'm a relatively new system admin.

I have two dedicated servers at two different data centers. I was hoping to find a simple setup where my second/backup server was kept "hot" and I was able to swap to it if things went south on my primary server. Could someone tell me the methods available to keep the files up to date on the backup server? Could someone also help me understand how I would update the IP addresses so the VMs would work despite being moved.

Thank you for your help - Justin

 

[bbgeek17]

I have two dedicated servers at two different data centers.

given these specific configuration and nothing else, your best and probably only option is to directly replicate data on backend storage. Replication can be synchronous or asynchronous. On top of that you need to design your own PVE configuration replication mechanism, which can be as simple as scp'ing appropriate files. You will need to then modify all the configuration to match your 2nd site.

In summary - there is no technology built into PVE that allows you to have a reliable two node split-DC cluster. A proper cluster requires a minimum of 3 nodes. If the nodes are not located in the same room/DC, the 3rd node should be in 3rd location for quorum preservation.
If you manage to create a stable split-cluster (latency is critical for corosync functionality), then you may be able to use ZFS-synchronization of VMs.

If you get the cluster and ZFS replication functioning, the IP management is left as exercise to the administrator and completely depends on your infrastructure. If IPs are static - you need to go and change them manually, if its DHCP - you need to take care of predictable allocation/recovery and accommodate DNS. If you set the IP via cloud-init portion - you need to update the config manually.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[hendr1x]

Thank you for the detailed response. Based on some reading it seems one option I have is purchase a cheaply hosted 3rd member for the cluster that does just voting. Does this seem like a good option in your opinion? In regards to IP management. I've set up PF sense before but that was all in the same Data center. Are there options available that work when the machines are all in separate data centers? I understand this is a super complicated but maybe you could provide some high level approaches that I can research? Thanks so much regardless.

 

[Dunuin]

A cluster needs low latency. Might be problematic with latency of >10ms. Keep that in mind when your servers are located in different datacenters.

 

[bbgeek17]

As @Dunuin mentioned, low latency is critical for cluster operations. By using a cloud(?) hosted quorum you are adding several layers where failure can occur (network, cloud, etc), rendering the cluster read-only or inoperable.

The PVE cluster, in general, allows for seamless VM failover in case of any failure on active node. Is your goal to move production to remote site on a reboot of the primary node? Or short 1-2min network outage? Are all other services able to function across sites transparently? Many times its more advantageous to recover from failure locally than take a whole site failover.

What happens when the sites start ping-ponging (especially with a hosted quorum)? Will the replication technology be able to catch up between the failures, and if not - would it be better not to failover to a site with stale data?

In my experience, enterprises that employ multi-sites also have playbooks of when and how site failovers happen. Very rarely these are automated events. In most cases they employ both local HA and remote DR.

Good luck in your research.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[louie1961]

I would use cloud storage, something like Amazon S3 or a cheaper alternative. Depending on your RPO, you could sync your data to the cloud with Rsync or Rclone like every hour or so. You can use storage tiering to reduce your overall cost.

 

[louie1961]

You could also potentially set up your own self hosted object storage in one of the data centers you are already in. Same principles would apply, but your data wouldn't be as safe as keeping it in the cloud where your data is automatically stored in multiple availability zones. But self hosting could be a LOT cheaper. Depends on your needs