Incremental Backup solution

[Pourya Mehdinejad]

Hi
We are already using Proxmox VE 5.1 Clustering and we have many VMs.
We've been using native proxmox LZO Backup, which is taking backups of all of our VMs every night, but we are kinda of worry since some of our VMs are growing to terabytes.
I wanted to get some advice from you guys, is there any other solution to take Incremental backups just like VEEAM ?

Our current backup storage is running on SAS 15K and is connected with 10Gbps, but still it should put some load on VMs at Backup times.

 

[napalu]

Hi

I've been working on a block-based backup solution to suit my needs which a) supports data deduplication and b) compresses the deduplicated blocks and c) uses public key cryptography to encrypt the blocks so that the backups can be replicated to external storage (I use a cloud storage provider) using rsync.

This is still a work in progress and only been tested under Proxmox 5.1 and it's not an incremental solution but the benefits of the approach are paying off in our environment - we currently have mixed lxc and kvm VMs using roughly 2.3 TB of storage which translates to roughly 275 GB of backup storage.

Perhaps deduplication rather than an incremental solution would also suit your needs?

 

[Pourya Mehdinejad]

Hi

I've been working on a block-based backup solution to suit my needs which a) supports data deduplication and b) compresses the deduplicated blocks and c) uses public key cryptography to encrypt the blocks so that the backups can be replicated to external storage (I use a cloud storage provider) using rsync.

This is still a work in progress and only been tested under Proxmox 5.1 and it's not an incremental solution but the benefits of the approach are paying off in our environment - we currently have mixed lxc and kvm VMs using roughly 2.3 TB of storage which translates to roughly 275 GB of backup storage.

Perhaps deduplication rather than an incremental solution would also suit your needs?

It's interesting though you've mentioned it's not still stable with Proxmox 5.1, but it worth a try.
can you send a link to How to ?

 

[napalu]

Hi
No I don't have a how-to right now - currently I'm automating the solution which so far is made up of several parts:
- an sqlite database to store information about the backups
- a driver program which invokes the Proxmox backup and pipes it to the deduplication process (so the backup is never stored but the blocks are deduplicated "on the fly" and then compressed and encrypted
- a command line interface to interact with the backups (restore, remove, clear blocks, list backups and so on).

As I said, this is still a work in progress but the basic approach is simple - it involves piping the Proxmox backup (important: uncompressed) to a deduplication process - this can be easily done by invoking vzdump with options --compress 0 and --stdout.

The bulk of the work is done by the deduplication process. Because I specifically need public key encryption of the blocks to safely store them in the cloud I'm using a custom solution but you could have a look at solutions like http://www.quicklz.com/exdupe/ or pcompress https://github.com/moinakg/pcompress (disclaimer: haven't used it but both look promising) to handle the deduplication parts.

 

[guletz]

Hi,

If you have zfs on your on both source and destination, one of the best solution is zfs send/recive or the PMX tool pve-zsync(who is doing the same). In short words, it is a incremental backup at the disk block level. And if your NEW block on disk on source are not so many, for sure it is the best solution. When you have TB of data to backup and/or milions of files/folders, any file-level backup is less efficient.
One simple example: I have a VM with around 1 TB of data, and is growing with small values, like 1-2 GB / 24 h. With pve-zsync it takes under 30 min. With rsync, the same backup in takes at least 3-4 hours.

 

[Frank_Pit]

Use Google and search for "proxmox ve differential backups" then take a look at ayufan.eu blog (at the time of writing supports only Proxmox VE versions up to 5.0) but it can do only differential and NOT incremental.