Backup in Large VM Environment

[wahmed]

Just want to get an idea from others what sort of backup strategy do you use in your large environment.

Of course a large environment could be any number but i think anything over 50+ VMs in a cluster can be considered somewhat large.
Backing up all these servers can be tedious task even with scheduling. One node can commit one backup at a time with default settings. So even the VMs are on across multiple nodes and backing up to more than one backup storage with different networks, still each node will become backup bottleneck. Not to mention the Read pressure the main storage where the VM is stored will go through.
Same issue even if we opt for weekly backups for all VMs. So some of the questions are:

-How do you backup the VMs on regular basis?
-How do you address main storage Read bottleneck?
-Multiple backup nodes per Proxmox node viable option?
-Should we stick with Snapshots on daily basis and do full back minimum once a month?

 

[spirit]

Hi,

At work, we are doing:

-backup in guest os datas, with bareos (www.bareos.org), incremental (1by day) + full(1 by month).
(full backups are done and balanced each day on differents guests vms)
-os and software are deployed with puppet (we save only user datas)
-for disaster recovery, san storage (netapp) is mirrored to another netapp storage on remote site.


We don't use proxmox backup currently, because of missing incremental backup.
(which has been commited this month in qemu git master, I'm hope to have it for promox 4.0)

 

[mo_]

I would also recommend sticking to in-VM targeted backups of required data only. This has the advantage that you can schedule scripts like "stop psql, rsync backup, start psql" (unless you have a proper transaction log setup that is...) which isn't as easy from the outside. As for backups "tools": I always found sticking to the basics (cron+rsync) works best, even when managing the crontabs might be a little more complicated than some fancy colorful webinterface backupsolution (but then again, crontabs can be managed by puppet).

this should be complemented by full-image backups at longer intervals... say monthly (you can do 1-5 VMs per night, spread out across the month to have negligible storage impact). Please be aware that a full image backup will currently effectively flush the Ceph cache storage pool, which has a minor to major storage performance impact on VMs (depending on use case).

 

[liska_]

Hi,
we use bacula for incremental daily backups inside guests (it has a client which can run scripts for stoping/starting db and so on) and then once a month a full backup.
As we had formerly problems with failing vms during complete backup at once, now we use proxmox backup of each node on different night or at least different part of the night.
And for disaster recovery we copy all vms data once a week to a different location.

 

[wahmed]

Good points!

We also do granular incremental file backup every 24 hrs for file serves only. We have been using BackupPC for this purpose for last 3 years. Has been working well. The nature of clientlessness of BackupPC was attractive. For maximum data safety though i have been trying to stick with regular full VM backup which obviously have negative effect on performance on different levels. May be i doing exe

I occationally hear about Bareos, but never looked in to it. I think it is time. From the screenshot it looks like. Last time i heard about it was during their lawsuit with Bacula. Was not sure if Bareos was going to survive.

I know doing longer period full VM backup makes sense. Just makes me little worry. But i guess it is better than the alternative of not being able to save the real data without baggage of OS data.

Looking forward to the Incremental backup implementation in proxmox. That will make things little easier.

 

[dietmar]

I occationally hear about Bareos, but never looked in to it. I think it is time.

There is also "amanda", which is a bit simpler than bacula or bareos.

 

[wahmed]

There is also "amanda", which is a bit simpler than bacula or bareos.

Is there are no GUI for Amanda free? I am not finding anything.

 

[dietmar]

Is there are no GUI for Amanda free? I am not finding anything.

No, but it is very easy to configure using 'vi' ;-)

 

[m.ardito]

I cannot count for a large environment (15-20 vm), though I do pve full backups daily, plus some VM with backuppc (testing) and some VM with a commercial solution (which also copies to lto tape, kept forever - hp openview data protector), and also automysqlbackup - for you know what - on linux os. I would like to test bareos, but as said is not the simplest. Though I could also cite here BURP, http://burp.grke.org/features.html which could be useful, although is not much known.

Marco

 

[cesarpk]

We don't use proxmox backup currently, because of missing incremental backup.
(which has been commited this month in qemu git master, I'm hope to have it for promox 4.0)

Hi Spirit and developers team.

If is possible, i would like to know if you have interest in add some features to PVE when the incremental backup lies ready for be used.

To the option of incremental backups:
- Selection of the virtual disks that will be are included for the Backup.
- Selection of the max quantity of incremental backups to retain in destination.
- Selection of days of week or dates of the month for do full backups periodically, this is with the purpose of that we can choose the option that will be more useful.

And to really get benefits, i believe that is very important for the restoration that we can to have these new features:

- Choose a specific storage for each image of disk that will be restored, instead of restore all the images of the virtual disks in a single storage as only destination.
- Also i think that would a good option of that we can choose for the restoration, the disk images that should be restored (many times is better to have the os, data and logs of a VM in different virtual disks for do a quick restoration of only the disk damaged).

Notes:
I believe that all these features will be very useful for gain time in a restoration of backup when the VM have several images of disks enabled in several physical hard disks different, and/or when the volumes of the images of disks has a large size.

Furthermore, these very good features for the restoration have as purpose avoid a set of steps that manually we must execute by CLI when the VM has virtual disks in different physical disks, so takes much longer in conclude a restoration successful. Also, with such features, we will avoid the need of have much more space available in disk for execute these intermediates procedures until complete all process of restauration.

With all these features, i think that PVE will have a complete solution of backups and restorations very useful for all kind of cases.

The best of success for you all
Cesar

 

[rkl]

There is also "amanda", which is a bit simpler than bacula or bareos.

We use Amanda 3 ourselves and it's great for backing up Linux guest VMs. However, the Windows Amanda client is frankly so bad (full of bugs, ludicrously installs and uses MySQL on the *client*, times out, produces a ZIP64 image that *only* PKWare's Zip Reader can read, no client fixes for over 3 years now), I've started to look at alternatives.

Bacula is far too complex, IMHO, and I don't like the idea of paying for the Windows client binaries, especially when the Linux ones are free. There's very few other free/cheap solutions that handle Windows+Linux clients, client/server setup and barcoded autoloaders (that may need to span multiple tapes for a single backup session). I'm open to suggestions

Oh, we have a backup network that is a second network interface on each guest VM and we add routes on that interface specifically to our 2 tape backup servers. This way none of the backup traffic mingles with the main network interface on each VM.

Note that much of VM guest filestore is from iSCSI SANs, so we have to limit how many clients we can backup in parallel. Too many and you can hammer the SANs (yes, we bonded a couple of Gigabit SAN interfaces together, which helped a bit). We ended up being conservative and stuck with 2 in parallel, which still finishes an hour or two before the VMs start to get busy with non-backup stuff.

 

[wahmed]

Can BareOS do backup of mobile laptops moving around on daily basis? I have setup a test BareOS VM and trying to see how it works. But it seems like the laptop clients needs to have static IP so BareOS director can talk to it. Is it correct?
Its not a problem for stationary backup clients such as Proxmox nodes or VMs in it. But problematic for moving laptops which goes from network to network.

Based on the suggestion in this thread, i decided to try both BareOS and Amanda and see which one works for my environment.

 

[spirit]

Can BareOS do backup of mobile laptops moving around on daily basis? I have setup a test BareOS VM and trying to see how it works. But it seems like the laptop clients needs to have static IP so BareOS director can talk to it. Is it correct?
Its not a problem for stationary backup clients such as Proxmox nodes or VMs in it. But problematic for moving laptops which goes from network to network.

Based on the suggestion in this thread, i decided to try both BareOS and Amanda and see which one works for my environment.

you need dynamic dns registration for your laptap in this case.

But indeed, workflow is:

bareos-director(server-scheduler)-------tell to client what to backup and where to send it------->bareos-client(laptop)------------->bareos-storage (server-storage disk).

and that mean that laptop must be up before backup.


I only use it for server, I can't comment for laptops. (at work we have WIFI && 3G everywhere for laptop, so no local data on laptop)

 

[wahmed]

you need dynamic dns registration for your laptap in this case.[/code]
In a way we already have this issue with BackupPC. But we have setup Openvpn which provides static IP address for laptops and BackupPC pulls data without issue. The BackupPC got us through close to a dozen file level disaster in last 3 years.
Proxmox does full VM backup while BackupPC does daily file level incremental backup. Through BackupPC GUI we can browse all past backups, including files/folders and only extract what is needed even down to a single file deleted 6 months ago. If i remember right, one of the main reason i went with BackupPC it is clientless.
In what way can we compare Amanda/BareOS with this setup? Which prominent feature i am gaining by using Amanda/BareOS?

 

[Defender007]

1st
Question Reply

To back up the virtual machine (which includes the operating system, application files, settings, and user data), you need to make a copy of the folder in which the virtual machine is stored. Ensure that the virtual machine is not running or suspended while you are backing up the virtual machine.

To back up the virtual machine:

1. Ensure your virtual machine is in a powered off state.
2. Locate the virtual machine folder. For more information, see Locating a hosted virtual machine's files (1003880)
3. Right-click the virtual machine folder and click Copy.
4. Navigate to the folder in which you want to store the backup, right-click anywhere within the folder, and click Paste.

2nd
Question Reply

The most common locations for data storage performance bottlenecks and problems are the front-end ports into a storage array, the controllers and the disk drives. The challenge lies in figuring out which congestion points are causing an application to perform poorly, or if it's even the storage at all.

 

[velocity08]

Have you looked at vProtect ?
vprotect.io

Have been interested to try but no time yet.

It has connectivity into ProxMox and works like Veeam.

 

[Awangku]

Have you looked at vProtect ?
vprotect.io

Have been interested to try but no time yet.

It has connectivity into ProxMox and works like Veeam.

Have you tried? I am stuck on configuring this backup solution. It's still pretty new in the market, so I am not able to find any proper installation guide of Proxmox with vProtect backup and restore. The one provided by vProtect storware itself are pretty darn confusing too