Differential backups

mbaldini

Active Member
Nov 7, 2015
170
20
38
Having mixed servers with Proxmox and VMWare, I can say that VMWare + Veeam Backup&Replication is the best backup solution that I can think of. Incremental backups of VMs from which I can recover the full VM, only some virtual disks or even only some files from the guest OS. I can replicate backup to secondary storage, with periodic full backups (weekly, monthly, yearly). Deduplication friendly backups. Single console with great management and full backup reports.
Proxmox with vzdump is nowhere near to a solution like that, I'm aware. Obviously there are different costs.
In my Proxmox servers I decided to go for a double backup solution. I use vzdump for nightly (or weekly for big VMs) full backup. I use Veeam Agent inside the guest OS to backup the content of the VM (it can create differential backup) so I can recover single files at guest OS level.
 
  • Like
Reactions: DerDanilo and np86
Jan 21, 2017
350
53
33
Berlin
Since Veeam is out and kinda working great in Linux for a while now this is a possible solution, even if it's a bit pain to install on all instances.
But it makes a lot of sense for big instances.

Veeam does not yet support more storage backends. CIFS is a possible solution but traffic is not encrypted. If Veeam Backups are encrypted I don't see any issue using a public CIFS share (with privat AUTH) since Veeam encrypts locally already.
 
Jan 29, 2017
168
10
23
44
Hi

How many people who used Proxmox has VM's with disks more than 500 GB? IMHO, I have 5 VM (DC,BDC,RODC etc) with just 40-80 GB disks.

And you can don't make differential backups for big VM's. Make full backups. With the patch you can do it. What's the problem ?

I have an Samba and Netatalk VM with 50TB replicated to another ZFS Proxmox
 
Jan 21, 2017
350
53
33
Berlin
I have an Samba and Netatalk VM with 50TB replicated to another ZFS Proxmox
For such setups one should use software defined network storage.
And usually the data inside those volumes is saved differently as there is just to much data.
E.g. backup tool on file level that saves differently.
Check out Quobyte as commercial alternative to Ceph of you require support and properly implemented Windows clients.
 
Jan 21, 2017
350
53
33
Berlin
Hi

How many people who used Proxmox has VM's with disks more than 500 GB? IMHO, I have 5 VM (DC,BDC,RODC etc) with just 40-80 GB disks.

And you can don't make differential backups for big VM's. Make full backups. With the patch you can do it. What's the problem ?

Split into 500GB disk for backup and restore reasons we have up to 3TB instances.
 

Sasha Sandow

New Member
Mar 15, 2017
10
0
1
52
I can only speak for myself, but I have been following this discussion for a while now. I am very disappointed that it has been several months and Proxmox thinks that it is acceptable to offer a hypervisor/vitalization solution that does not support either differential or incremental backups still to this day and they really don't think this should not be their priority.

I see that they have released yet another version and still will not even openly discuss solutions for this issue. An endless amount of additional storage systems are added continually but still not a single mention of of a direly needed system to do differential/incremental backups. I would love to continue to support this company but I am afraid I might need to finally fold to pressure and recommendations from just about every backup provider I have spoken to and seriously consider beginning to migrate all servers to a less desirable but more supported platform like VMWare or Hyper-V. Can anybody at this point suggest a differential/incremental backup system that works with Proxmox with out needing to switch everything to ZFS for VMs that are over 500GB but less then 1TB?
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
14,615
633
133
I can only speak for myself, but I have been following this discussion for a while now. I am very disappointed that it has been several months and Proxmox thinks that it is acceptable to offer a hypervisor/vitalization solution that does not support either differential or incremental backups still to this day and they really don't think this should not be their priority.

Your summary of the situation is a bit wrong, at least I see it differently. I do differential and incremental backups of my virtualized server load with several third party tools day by day.

The integrated vzdump is currently doing full backups only, but this not the end of the story.
 

virtRoo

New Member
Jan 27, 2019
27
4
3
Given a lot of people have been looking forward to achieving live differential/incremental backup for KVM, I'd just like to share some of my fancy ideas here as we're all in the same boat. File-level backup is simply not scalable for us.

Due to the fact that there are so many different implementations/standards of KVM overall, I can totally understand why not many vendors have been able to come up with a proper backup solution for "KVM" to this date.

Since Veeam for Linux has been brought up in this thread, I was thinking maybe Veeam could still be used to achieve incremental image backup for some use cases, granted it wouldn't be a perfectly integrated solution for every scenario.

For those who run Proxmox on a standalone host without shared storage and use LVM as an individual virtual disk for each VM, maybe we could simply install Veeam Agent for Linux on the host and let it back up each LVM with incremental backup at the block level given the fact Veeam Linux supports Debian 9 and LVM, and each virtual disk is just a local LVM on the host. VM data is what I care most, I don't need VM memory dump, and configs can be backed up easily.

I've managed to get this working in my test environment with Proxmox 5.3-11, and I was able to achieve exactly what I've been after, which is live incremental backup at the block level. Again, this is just for testing so try it at your own risk. The main caveat is file-level extraction doesn't work, but I can simply use Veeam to restore the image onto a new LVM on a different host and then mount the LVM to extract the files myself, not a big deal.

Veeam also supports running scripts for backup process so maybe one could try doing some kind of fsfreeze via qemu guest agent.

However, it really depends how one's storage for VM is set up, and I'm not sure how this kind of workaround can come into play with shared storage, or storing virtual disks as qcow2/raw.

I've got no hands-on experience with this workaround in a production environment, so I guess I'd have to do more testing first.
 
Last edited:

LnxBil

Famous Member
Feb 21, 2015
5,468
596
133
Germany
I've managed to get this working in my test environment with Proxmox 5.3-11
Nicely done.

Another way which wasn't brought up here already is to use a ZFS-based backup-server, to which all the vzdumps are copied, extracted and stored, so that you can use the CoW benefits of ZFS to create differential backups. We backup our datacenters with this method for over 3 years now and it just works.

This is a backup of a redmine VM with a vzdump archive size of 14 GB (according to the logfile) and we have here 64 backups which have combined almost 900 GB or vzdump files, we have used 23,4 GB in the ZFS - that's almost a factor 40 in savings:

Code:
root@backup ~ > grep size /proxmox/dump/vzdump-qemu-1005-2019_03_12-18_30_02.log
Mar 12 18:38:23 INFO: archive file size: 14.26GB

root@backup ~ > zfs list -r -t all rpool/proxmox/1005
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
rpool/proxmox/1005                                    23,4G  4,75T  14,6G  /rpool/proxmox/1005
rpool/proxmox/1005@2017_09_15-18_33_08                1008M      -  13,7G  -
rpool/proxmox/1005@2018_01_15-18_30_02                 767M      -  13,7G  -
rpool/proxmox/1005@2018_07_19-18_30_02                 127M      -  14,1G  -
rpool/proxmox/1005@2018_08_20-18_30_02                 245M      -  14,2G  -
rpool/proxmox/1005@2018_09_17-18_30_02                 242M      -  14,2G  -
rpool/proxmox/1005@2018_10_17-18_30_01                 218M      -  14,3G  -
rpool/proxmox/1005@2018_11_16-18_32_59                 196M      -  14,4G  -
rpool/proxmox/1005@2018_12_06-18_30_02                 121M      -  14,4G  -
rpool/proxmox/1005@2018_12_13-18_30_02                99,5M      -  14,4G  -
rpool/proxmox/1005@2018_12_17-18_30_01                61,8M      -  14,4G  -
rpool/proxmox/1005@2018_12_18-18_30_02                64,1M      -  14,4G  -
rpool/proxmox/1005@2018_12_25-18_30_02                 103M      -  14,5G  -
rpool/proxmox/1005@2018_12_27-18_30_02                 106M      -  14,5G  -
rpool/proxmox/1005@2019_01_01-18_30_01                67,2M      -  14,5G  -
[.... many more .... ]
rpool/proxmox/1005@2019_03_11-18_30_01                49,6M      -  14,6G  -
rpool/proxmox/1005@2019_03_12-18_30_02                   0B      -  14,6G  -
 
  • Like
Reactions: guletz
Jul 18, 2016
21
0
21
France
gplexpert.com
Hello,

Nice. but you have to do a vzdump every days. The differential is just to optimize space on zfs ?

So it should take a very long time to do vzdump for all datacenter right ?

Another question : can we use some technical case for backup using ceph method ?
I tried eve4pve-barc but it doen't work with ceph on a different cluster server
 

LnxBil

Famous Member
Feb 21, 2015
5,468
596
133
Germany
So it should take a very long time to do vzdump for all datacenter right ?

Depending on your data and your throughput to your backup storage, yes.

Most of the server do not need a backup every day, this depends on the stored data. For a fileserver, we use ZFS internally, so that we can do snaphots every 15 minutes and only vzdump the OS, not the data on ZFS. We use replication for backup there. Our full backup is over 1 TB on vzdump data alone. To optimize the dump size, we also run periodically zero sweeps of the virtual disks (thick lvm, so no trimming). For the ZFS space optimization, we also do not compress logfiles after rotation, so that the files stay as they are for the time beeing without being written again. Depending on the logfiles, this also improves compressibility and uniqueness of data inside of vzdump backup that is stored on ZFS itself.
 

Power2All

New Member
Sep 13, 2017
5
0
1
38
I have been using the modded backup solution for quiet some time.
I wonder why you even WANT a 500GB or bigger VM, that's just plain stupid.
It's more then enough to host a VM with a OS and some little storage for temp work and such, but for data, you would not want to use Proxmox's storage system anyhow.
For purely VM OS backups, the differential backup method is perfect, and it doesn't interrupt my systems.
Also, so far it has been rock solid and reliable, compared to all other backup methods there are around.
I wonder why the implementation that has been given, could not be changed, so that it would give a warning when you want to make a backup of a system, that has a bigger storage then 500GB... At the least you could do that.
Lower then 500GB backups are stable and works fine as far as I've been using it.

Honestly, there is always a solution for a problem, including the "limitation" which can be bypassed through a warning screen.
If you need to backup a OS disk that is 500GB, then I don't know what is wrong with you..
 
Last edited:

LnxBil

Famous Member
Feb 21, 2015
5,468
596
133
Germany
It's more then enough to host a VM with a OS and some little storage for temp work and such, but for data, you would not want to use Proxmox's storage system anyhow.

What is your suggestion there?

If you need to backup a OS disk that is 500GB, then I don't know what is wrong with you..

Is your comment still true if you omit the "OS"? It's very easy to get huge systems...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!