[SOLVED] VM continuous mirroring and network switching

[cshill]

Hi Proxmox Community,
I was able to discover replication and the HA manager for quick transitions of VMs and I am happy with that for most things. I just have a couple critical services that require attention.

I'm wondering if it's possible to have two VMs on different servers have a continuous replication sync over the internet. The current replication I am seeing feels like a scan every so often and then updates that on the zpool for other servers. I need it so that the moment a file is put on VM1 it goes to VM2. Currently this is being achieved with robocopy.

The purpose of this request is that there will be a channel of information flowing into VM1 that is very important. We want this information that is in a main directory on a linux OS to be directly replicated to the mirrored VM. I have the Replication feature in Proxmox and using HA but the time it takes is about 2 minutes before it is setup on another server. I'd like it if I can have a 2nd VM that is on standby and is a perfect mirror of the 1st VM but is not actively processing information until the 1st VM goes down.

This feels more like something that needs to be configured with bash scripting but would like to know if the community knows of anything like this.

As always this is a great community thank you.

 

[bbgeek17]

I'm wondering if it's possible to have two VMs on different servers have a continuous replication sync over the internet. The current replication I am seeing feels like a scan every so often and then updates that on the zpool for other servers.

The current replication is indeed asynchronous, snapshot based. Its done on repeating time interval and asynchronously sends changes at the specified interval.

I need it so that the moment a file is put on VM1 it goes to VM2.

Generally, this is called "synchronous" replication. Sync replication requires more resources than async. Additionally, Sync replication requires the client write to be committed and acknowledged on both sides before the client is informed. As you can imagine, this requires very low latency.
Enterprises pay big money for both the software (SRDF, SM-S, etc) and for the infrastructure required (Dark Fiber).

I believe ZFS does not implement Sync replication in the proper sense, but I could be mistaken.
Within the Proxmox framework, you are limited to the options exposed by Proxmox.

You can achieve some semblance of what you are asking, at your own risk, by directly manipulating the disks: https://serverfault.com/questions/846241/zfs-on-linux-zol-and-realtime-replication

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[UdoB]

Just use a network filesystem for this! Possibly a simple NFS Server does what you want.

On the other hand this opens another can of worms regarding redundancy/reliability/SPOF/bandwidth...

Good luck

 

[cshill]

I was about to start looking into glusterfs volume but I looked into what @bbgeek17 said about SRDF and noticed the price tag and said no way. I tried to find another solution but I don't think this one is exactly what I want. It's called Linstor by Linbit and it's a cluster volume manager for persistent Linux block storage for Kubernetes. I need to look further into this but a question I'm having is wouldn't this be simpler if we have a serial attached storage device that is recognized by the two VMs, pass through, that are sharing off the same volume? I may not have that particularly correct there, still learning, but regardless instead of replicating between two storages just have the volume accessible by the different servers?

And if we want different servers to access a volume and it's not physically possible then use a network attached device?

 

[bbgeek17]

It seemed, initially, that you implied a Disaster Recovery need: if my data center in NY gets flooded, can I recover in Phoenix?
You now pivoted, or perhaps I misunderstood you, to a local High Availability requirement.

The latter can be satisfied in many ways, and the key is "shared storage".
Shared Storage comes in many varieties: CIFS, NFS, iSCSI, NVMe/TCP, SAS, FC, etc.

Two+ VMs can share a disk, the same way as it is possible for two+ physical machines to do so. There are many nuances to how the overall solution can be implemented to avoid data corruption.

You can also have HA and DR. If you have the right skills or unlimited free time - you can do it on your own "for free". Or, you can reach out to the right company that provides the required functionality with the necessary applications (PVE).

The easiest approach for a quick and "dirty" HA is with an NFS storage.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[cshill]

I'm sorry if there was some confusion with the statement. In general it's both a locally and remote solution. Also you can bare with me on this one as I'm pretty sure I know that it's not possible without solid infrastructure investment and or expensive software implementation.

I'm using PVE's replication and HA system and I'm loving it and people are impressed that's an option. The problem has to do with our critical services and it's currently setup with an HAproxy for routing traffic to other servers that are live and can process the information. It's that there are servers that are dedicated and setup just waiting for the pipe to turn on and off. We are routing information to different locations if a server goes down with those services. The problem is that the proxy and what is required for diverting traffic seems not very efficient. That's why if there could be a method to turn on synchronous replication of data between two servers and if our VM goes down it automatically moves to another server or, through PVE, a VM on another server instantly takes over the failed process from the other and starts the data process from where the last left off. This is the challenge, the miniscule delays and checksum of events.

Now I'm still on the newer side but a network file shared location makes sense to me as the VM on the new server will reconnect and continue processing data but it's a matter of speed and redundancy. If I have VM in California and a VM in Oregon can they both reasonably access the same information and in what time.

 

[bbgeek17]

It's that there are servers that are dedicated and setup just waiting for the pipe to turn on and off. We are routing information to different locations if a server goes down with those services. The problem is that the proxy and what is required for diverting traffic seems not very efficient

I think this describes Application Level HA, which, in many cases is the best approach.

That's why if there could be a method to turn on synchronous replication of data between two servers and if our VM goes down it automatically moves to another server or, through PVE, a VM on another server instantly takes over the failed process from the other and starts the data process from where the last left off.

PVE can only play a role in VM failover if the servers in question belong to the same cluster. For DR situation you dont want them in the same cluster. Otherwise you are significantly increasing blast radius of the failure.

Synchronous replication costs money, both in software and infrastructure. If you have a good use case for it, like so many large corporations, then you likely have a budget for it.

This is the challenge, the miniscule delays and checksum of events.

If you are talking about ms/usec type delay here, then even synchronous replication is not for you. The VM on target side still needs to become aware of filesystem/journal/block changes, and that is not an easy problem to solve.

You should be looking at application-level protection/failover here.

Good luck in your search and design.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[cshill]

As always thank you @bbgeek17