Off-site backups of Windows server VMs from various PVEs on the internet

[cosmos]

I'm considering using proxmox on 2-3 totally different sites. Details:

• These sites do not belong to the same organization, but management (including backup) has been delegated to a small (two person) firm.
• PVE boxes on each site are connected to the net using either ADSL or VDSL links, meaning upload is around 1 to 5 Mbps.
• A single PVE box will be carrying one to two VMs. The VMs in all cases are Windows 2008 (and later) servers.
  
• We have one 50/5 Mbps VDSL line to manage the entirety of the PVEs.

The problem here is to have off-site backups of all Windows Server VMs. Now, if this was a LAN I would most likely consider using PVE's built-in backup functionality to create compressed images of the Windows disks to another NAS via NFS perhaps. If the VMs were Linux/FreeBSD boxes, I'd consider using rsync to a box inside our own lan. Unfortunately, due to this bandwidth/operating system combionation I'm investigating alternative means of performing the same task. That is, having each day a backup of each VM box somehow sent to our own server.

I'm not certain whether this is a sound idea, considering that I do not have much experience on servers. I'd love to hear better alternatives for that matter. Assuming now that our idea is not that bad, what could one use to have incremental/differential full system backups taken on a schedule?

I'm considering Macrium site manager, this means that I will prolly have to have a server (from our own lan) SMB share open to the internet (\\OURCENTRALNAS or something). Are there other alternatives?

 

[NewDude]

I've been using R1Soft Servernbackup solution for a few years. It works well, but I'm not sure what the current pricing looks like - they were bought, now I think they've been spun off again.

https://www.r1soft.com/product-features

Basically, their backup agent runs inside your VM, and it will trigger a snapshot of the filesystem in Linux or Windows, and back that up while the VM continues to run. If you give it credentials to MySQL/MSSQL it will force the database to flush to disk, then trigger the snapshot and allow things to continue. Figure < 2 seconds when this happens.

Backups themselves are basically changed data, so only new changes get replicated. It works pretty well, and is worth consideration.

 

[fireon]

For this things we using PVE-Zsync.

 

[cosmos]

Thank you for the information provided.

@NewDude I presume the solution you're describing is running inside a VM, like Macrium. I'll check into that. Do you recall which methods are available for sending data off-system? Windows SMB/NFS/FTP?

@fireon This works on the hypervisor level if I understand correctly. My problem is that I am not fluent with ZFS. I did have some PVE experience on a basic level, long before ZFS was included in it. I understand that part of pve strength is the multitude of formats/protocols/methods (LVM, iSCSI, NFS, ZFS) to implement things on, but I find this variety intimidating... To make a long story short, PVE-Zsync seems like a thing added like a hack. Do you believe it will stay? I'd hate to install something, only to have to replace it later on with something else due to compatibility issues with new PVE versions etc. Lack of knowledge on my part means I might have to go extra conservative on the approach used.

Furthermore, and speaking of Windows VMs, is this method bandwidth efficient during sync'ing? That is, if I have a 200Gb Windows VM with 50Gb of data on ZFS and this has to be sync'ed, what's the quantity of data transferred? That is, is it only the changes, or 50-200Gb of data each time?

 

[mbaldini]

Create VPN between the Proxmox sites and the backup site, then use Veeam Endpoint Backup (or similar, I just love Veeam because I use it even in VMWare environments) in Windows VM, that can run incremental backups.
Do not open SMB to the internet, tunnel it through a VPN

 

[cosmos]

@mbaldini thanks I had something along those lines in mind. I've seen references to using perhaps OpenVPN on the windows server VMs, if you believe something else is better please feel free to elaborate.

 

[totalimpact]

"a thing added like a hack".... definitely not, it is very robust, zfs send|receive is a standard function of any zfs implementation, PVE has just built a wrapper around it to schedule and track backups, and there is commitment from the PVE team to expand it in to the GUI someday. And it is actually pretty simple to setup in less than 5 minutes.

https://pve.proxmox.com/wiki/PVE-zsync
https://www.servethehome.com/automating-proxmox-backups-with-pve-zsync/

Now disclaimer - dont let me push you in to using something you dont want to put time in to. By all means go with the path of least resistance.

If these systems are already in production, it is discouraging that ZFS would take some work to implement.... but I have done it with zero downtime on several occasions. If you have a massive 20tb data pool, that may not work well, but a couple tb has been easy. You just need a temp storage medium, issue storage move command in gui, re-format your primary storage as zfs, and issue gui move command back to zfs - the VM never has to shut down, obviously you would be best suited to do it after hours.

If you have a proper enterprise storage right now, ie well performing hardware RAID10, you could sit the zfs on top of that and reads should remain similar to LVM, writes will take a ~30 performance hit, if the use case does not notice the hit, it is well worth it. I have benchmarked this exact scenario of LVM vs ZFS with no Zraid configuration on a proper hardware card (i will get scoffed at forever).

pve-zsync is fully incremental rsync style, so it will be your most efficient choice, and it can be run every few minutes, this really drops the load on the WAN, and gives you some damn strong restore ability with very low impact.

The remote site does not need to be PVE, it could be any system that supports zfs-receive like freenas. Obviously a remote pve would allow you to quickly spin up a vm in a disaster.

It is also DB friendly, no need to flush or any of that-
https://dev.mysql.com/doc/refman/5.6/en/ha-zfs-replication.html

 

[cosmos]

If these systems are already in production, it is discouraging that ZFS would take some work to implement.... but I have done it with zero downtime on several occasions. If you have a massive 20tb data pool, that may not work well, but a couple tb has been easy. You just need a temp storage medium, issue storage move command in gui, re-format your primary storage as zfs, and issue gui move command back to zfs - the VM never has to shut down, obviously you would be best suited to do it after hours.

The PVE systems will be fresh installations, on low-grade/cheap server hardware (ie no RAID 10 and such). Also the VM on each PVE box will be rather small, less than 250Gb per virtual disk (and even less for data). Does pve now ask how it should set up base storage on installation? BCPs for setting up disks on ZFS?

If you have a proper enterprise storage right now, ie well performing hardware RAID10, you could sit the zfs on top of that and reads should remain similar to LVM, writes will take a ~30 performance hit, if the use case does not notice the hit, it is well worth it. I have benchmarked this exact scenario of LVM vs ZFS with no Zraid configuration on a proper hardware card (i will get scoffed at forever).

So one would expect a serious performance hit for low-end raid using ZFS (no SSD etc)?

pve-zsync is fully incremental rsync style, so it will be your most efficient choice, and it can be run every few minutes, this really drops the load on the WAN, and gives you some damn strong restore ability with very low impact.

Is this some sort of block-level differential syncing?

 

[NewDude]

@NewDude I presume the solution you're describing is running inside a VM, like Macrium. I'll check into that. Do you recall which methods are available for sending data off-system? Windows SMB/NFS/FTP?

I'm not familiar with Macrium. The system I'm referring to runs a on the VM itself and requires a dedicated backup computer to initiate the backups according to the schedule you set, then store them. In my case I've got one backup server in the data center, and another here (actually, this backup server is a VM running on a QNAP device) that pulls from the primary backup server so I've got off-site backups. That's a lot of resources, but it means fast local restores, and I maintain data < 24 hours old if my datacenter suffers a site failure.

 

[totalimpact]

That does not sound very promising.

If you need to run ZFS on a raid card, it had better be good, otherwise ZFS is optimized for a non-raid HBA card, where it can constantly monitor the bare drive state. If you are doing RAID5 of 7.2k sata on some junk raid card - ouch, all bets are off. I hope its RAID1...SAS?

ZFS shines on:
64GB RAM
HBA card like LSI 9300-8i IT Mode, 6+ spinning disks in raid10, or Dell H310 flashed to LSI 9211 if you are tight.
Possibly a SSD cache if your spinners are slow

If you have not sourced servers yet, Dell R720 are cheap on ebay, and they come with legit raid cards+lots of bays, I then buy new drives of my choice. If you have to compromise, do it on the server, not the storage, storage can make or break the experience. $400 T30, or $600 R720?? https://www.ebay.com/itm/Dell-R720-2X-E5-2650-32GB-RAM-DUAL-POWER-PERC-H710/162921789612

Yes, PVE lets you choose at install, if you are on slow drives, dont customize the storage, just leave it default, and stick with LVM.

Yes, differential block-level snapshotting/syncing.

 

[cosmos]

Wealth of info from all of you guys I'll have to go study then, thanks a lot.

 

[totalimpact]

If you are looking at Macrium, which I feel is a great product, you might also look at Easus Todo Backup, which is nearly identical to Macrium, but a couple more features, some of them are cloud based, but last I looked they had limited cloud options.

 

[guletz]

Hi,
pve-zsync is a incremental block device backup. And it is very fast. Any other file-system backup like rsync (who can be used even for win ) is slow IF you will have many files to sync.
Your problem is Internet link (like adsl) witch is very bad.... Maybe you need to relocate your servers in my country, where you will have at least tens of Mb/sec . At my home I get at least 60-80 Mbit/sec ... lucky me -

 

[readyspace]

May I know if anyone here successfully recovered a backup from R1Soft into Proxmox KVM?