Implement backup PVE server without shared storage using PVE-zsync?

[hefferbub]

I'm planning a setup with PVE 4.0 with one fast server which will normally serve my (KVM) VMs. I would like to have a slower backup server that can be ready to take over the VMs in case of a problem with the fast server. I don't want to have to use shared storage due to speed concerns, and I want to use ZFS to hold the VMs.

I don't need an HA setup--I am OK with a little downtime to manually restart the VMs on the backup server. I mostly just want to avoid a lengthy data transfer when using the backup server. I assume I could use PVE-zsync to keep the VMs synced up between servers.

Is it possible to use PVE-zsync to keep the VMs synced up between servers in a way that I can start them directly on the backup server when needed? That is, can they be synced right into place, rather than needing to follow the "Recovering an VM" procedure listed on the PVE-Sync wiki page for each VM?

Is this made any easier if the two servers are part of a cluster or would it be better if they were standalone?

Any advice for how to approach this would be appreciated. Thanks.

 

[kbrault]

I have the same questions. Did you find any answers?

 

[hefferbub]

Funny, I was just working on this this morning...

I have two servers, and on each of them I have a zpool called "rpool", and I make that visible to PVE (by creating an entry under Datecenter/Storage) as "vmpool".

No Cluster mode:
The servers initially were not part of a cluster--just independent servers. In this mode, I was able to create pve-zsync jobs on the backup server that pulled the data from VMs on the main server and placed the data in the exact same location (vmpool:vm-nnn-disk-1) on the backup server (which is actually "rpool/vm-nnn-disk-1"). It would keep this data in sync every 15 minutes. Here is an example, entered on the backup server, for VM 149:

pve-zsync create --source 192.168.123.456:149 --dest rpool --verbose --maxsnap 1 --name vm149

To make this backup VM live, I needed only a few quick steps:
1. Stop the sync job with the pve-zsync disable command
2. Copy the qemu config from the directory where pve-zsync stores it (/var/lib/pve-vsync) to the real location for qemu (/etc/pve/qemu-server) and rename it to just nnn.conf, where "nnn" is the VM ID number. For example:

cp /var/lib/pve-zsync/149.conf.qemu.rep_vm149_2016-01-06_11\:54\:38 /etc/pve/qemu-server/149.conf

3. Adjust any settings in that conf file (or in the Web GUI--it shows up automatically when you copy in the config). All I had to adjust was some CPU and memory settings, because my backup server is more resource-constrained.
4. Start the VM in the web interface (having stopped the VM on the main server first, of course).

It took like one minute to bring it up.

Cluster mode:
I later decided to go ahead and make the two servers a cluster. Once I did that, I think the technique above won't work so well, because PVE may be grumpy about VMID numbers being the same on two servers. Or maybe not. In any case, I didn't test it.

I did test the Migration functionality of the web interface, and it worked fine for moving VMs between servers (in offline mode, since there is no shared storage). It took care of creating the new ZFS zvol and deleting the old one and all that sort of thing. The only problem is that (1) it takes a while for the physical copy and (2) It won't help you if the main server has failed.

So what I did was this:
1. Create a second zpool on the backup server, using a couple of extra hard drives I had available
2. Set up pve-zsync jobs to pull VMs into this backup zpool every 15 minutes.

If I need to recover from a failed main server, I would then follow the (slightly cumbersome) procedure on the pve-zsync wiki page. I will have to zfs send/receive the data from the backup pool to the production pool, but at least that should be fast because it is happening disk-to-disk on the backup server rather than over the network.

I have not yet tested this, but I'll give it a try over the next day or so and will re-post if anything interesting comes up.

Finally, somewhere I saw a post that the PVE developers are hoping to eventually release a feature to automate the process of restoring a VM from a pve-zsync backup, so hopefully this will get simpler and more transparent in the near future. Those guys rock!