Reducing backup storage requirements?

[tomstephens89]

Hi there,

I am a fan of the out the box Proxmox backup solution. Its simple yet effective! However on larger clusters with many VM's, running daily full backups obviously means huge disk space requirements.

I am a long time Proxmox user and have built several large clusters with it, but in this case I am replacing a HyperV cluster which currently uses Veeam for VM level backups.

I am not expecting Proxmox to go as fully featured as Veeam for backup services, however a simple incremental/full rotation would be great out of the box.

In this instance I was thinking of using FreeNAS on a 12 bay Dell R510 with external MD1200 arrays also attached. I was planning on presenting an NFS store over 10Gbe to the Proxmox cluster.

So...

Are there any plans to implement an incremental/full backup system into proxmox ve?

Is there a way to use a filesystem with De-duplication within Freenas/Similar that will keep the pile of full VM backups down in size?

Thanks

 

[dietmar]

We have some experimental code using content addressable storage technology. But nothing you can use now...

 

[tomstephens89]

We have some experimental code using content addressable storage technology. But nothing you can use now...

Thanks for the reply. So essentially this means the best way to control the storage requirements of Proxmox's built in backup system is to use a destination file system that supports de-duplication and compression? ZFS on freenas for example.

Any examples of a good backup storage system would be appreciated.

 

[dietmar]

this means the best way to control the storage requirements of Proxmox's built in backup system is to use a destination file system that supports de-duplication and compression?

No, those features wont help much. I recommend to move backup files to a tape to store backups reliable (off site).

 

[LnxBil]

Is there a way to use a filesystem with De-duplication within Freenas/Similar that will keep the pile of full VM backups down in size?

You can use ZFS (but not directly and dedup will also not work as expected):
- uncompress your backup to the ZFS (one ZFS dataset per VM) with vma
- rsync or blockwise diff and copy the files onto it (this is crucial for the copy-on-write per block, so that only new blocks are written)
- one snapshot for each backup one you can reduce the space requirements dramatically.
I gave a talk in german about this on the Proxtalks 17. I still did not have time to convert everything to english to publish it properly.

Here some stats for space saving directly from my slides:

$ zfs list -t all -r -o name,use...ratio rpool/proxmox/1000
NAME                                     USED  REFER  RATIO
rpool/proxmox/1000                      4,57G  2,82G  2.57x
rpool/proxmox/1000@2017_03_03-18_30_01  81,7M  1,88G  1.95x
rpool/proxmox/1000@2017_04_28-18_30_02  72,6M  1,99G  1.92x
rpool/proxmox/1000@2017_06_30-18_30_02   123M  2,39G  1.92x
rpool/proxmox/1000@2017_07_14-18_30_01   118M  2,43G  1.93x
rpool/proxmox/1000@2017_08_04-18_30_02  47,2M  2,37G  1.86x
rpool/proxmox/1000@2017_08_18-18_30_02   108M  2,62G  1.87x
rpool/proxmox/1000@2017_09_01-18_30_02   110M  2,66G  1.88x
rpool/proxmox/1000@2017_09_08-18_30_02   113M  2,67G  1.89x
rpool/proxmox/1000@2017_09_15-18_30_02   120M  2,72G  1.89x
rpool/proxmox/1000@2017_09_22-18_30_02      0  2,74G  1.89x

Datum      vDisk       VMA.lzop    dZFS   Einsparung
----------------------------------------------------
17-09-15   1.009.282   322.195     59.773     82 %
17-09-22   1.065.113   329.495     48.574     86 %
17-09-29   1.056.528   339.904     85.338     75 %
17-10-06   1.365.757   429.057    106.438     76 %
17-10-13   1.640.633   468.588    135.476     72 %
17-10-20   1.501.048   468.283     49.856     90 %

vDisk is the accumulated disk requirement of all disks inside PVE
VMA.lzop is the lzop compressed backup size
dZFS is the "real" difference in Data that ZFS has detected
"Einsparung" is german for savings

 

[tomstephens89]

You can use ZFS (but not directly and dedup will also not work as expected):
- uncompress your backup to the ZFS (one ZFS dataset per VM) with vma
- rsync or blockwise diff and copy the files onto it (this is crucial for the copy-on-write per block, so that only new blocks are written)
- one snapshot for each backup one you can reduce the space requirements dramatically.
I gave a talk in german about this on the Proxtalks 17. I still did not have time to convert everything to english to publish it properly.

Here some stats for space saving directly from my slides:

$ zfs list -t all -r -o name,use...ratio rpool/proxmox/1000
NAME                                     USED  REFER  RATIO
rpool/proxmox/1000                      4,57G  2,82G  2.57x
rpool/proxmox/1000@2017_03_03-18_30_01  81,7M  1,88G  1.95x
rpool/proxmox/1000@2017_04_28-18_30_02  72,6M  1,99G  1.92x
rpool/proxmox/1000@2017_06_30-18_30_02   123M  2,39G  1.92x
rpool/proxmox/1000@2017_07_14-18_30_01   118M  2,43G  1.93x
rpool/proxmox/1000@2017_08_04-18_30_02  47,2M  2,37G  1.86x
rpool/proxmox/1000@2017_08_18-18_30_02   108M  2,62G  1.87x
rpool/proxmox/1000@2017_09_01-18_30_02   110M  2,66G  1.88x
rpool/proxmox/1000@2017_09_08-18_30_02   113M  2,67G  1.89x
rpool/proxmox/1000@2017_09_15-18_30_02   120M  2,72G  1.89x
rpool/proxmox/1000@2017_09_22-18_30_02      0  2,74G  1.89x

Datum      vDisk       VMA.lzop    dZFS   Einsparung
----------------------------------------------------
17-09-15   1.009.282   322.195     59.773     82 %
17-09-22   1.065.113   329.495     48.574     86 %
17-09-29   1.056.528   339.904     85.338     75 %
17-10-06   1.365.757   429.057    106.438     76 %
17-10-13   1.640.633   468.588    135.476     72 %
17-10-20   1.501.048   468.283     49.856     90 %

vDisk is the accumulated disk requirement of all disks inside PVE
VMA.lzop is the lzop compressed backup size
dZFS is the "real" difference in Data that ZFS has detected
"Einsparung" is german for savings

Thanks for the reply. This does however appear to add a fair amount of additional steps into the backup process.

Guess the easiest backup solution will just be to make sure I have a ton of storage available and to rotate as needed with a periodic archival of a full backup set to tape or elsewhere.

 

[LnxBil]

This does however appear to add a fair amount of additional steps into the backup process.

Perfectly scriptable on the backup side if you backup to a NFS storage. We use systemd timers to check for new files and start automatic import. Works like a charm for years. No manual steps required.