How to do differential backups?

[monster]

I found out through this page here:

http://manpages.ubuntu.com/manpages/karmic/man1/vzdump.1.html

that vzdump used to support an xdelta option to do differential backups, and that it eventually got removed, but I could not find the reason why. I have never used xdelta, therefore I have no opinion about it, but I need differential backups, one way or another.

I can't afford to pay for the bandwidth needed to download a full backup every day, but I can't afford to loose one week of data either. Atm, probably over 90% of the backup file size is just unchanging OS & Java files, making it ideal for differential backup.

I could just make uncompressed full backups, and then build a diff myself, but that is the kind of functionality that one normally expect in any backup software.

Is there any plans to add again differential backup functionality in vzdump?

If not, is there a recommended way of doing differential backups? For example, if there xdelta causes some sort of problems, it would be good to know about it.

 

[tom]

I found out through this page here:

http://manpages.ubuntu.com/manpages/karmic/man1/vzdump.1.html

this an old copy and outdated. just type 'man vzdump' on a current Proxmox VE system to get the right man pages.

that vzdump used to support an xdelta option to do differential backups, and that it eventually got removed, but I could not find the reason why. I have never used xdelta, therefore I have no opinion about it, but I need differential backups, one way or another.

I can't afford to pay for the bandwidth needed to download a full backup every day, but I can't afford to loose one week of data either. Atm, probably over 90% of the backup file size is just unchanging OS & Java files, making it ideal for differential backup.

I could just make uncompressed full backups, and then build a diff myself, but that is the kind of functionality that one normally expect in any backup software.

Is there any plans to add again differential backup functionality in vzdump?

not now. we do not get the needed stability on the xdelta and we skipped it.

If not, is there a recommended way of doing differential backups? For example, if there xdelta causes some sort of problems, it would be good to know about it.

use vzdump for full image backup for fast desaster recovery and for small containers and VM´s. if you have big VM´s and/or you need differential backups, go for a traditional full featured backup tool of your choice.

 

[dimi]

I have the same problem! And i think about use www.lessfs.com (http://www.lessfs.com/wordpress/?page_id=50) deduplication with asynchronous replication, but i still not compile for Proxmox (debian). I will try today.
I think this way is the best!
Sincerelly
Dimitri

 

[edwards]

this an old copy and outdated. just type 'man vzdump' on a current Proxmox VE system to get the right man pages.



not now. we do not get the needed stability on the xdelta and we skipped it.



use vzdump for full image backup for fast disaster recovery and for small containers and VM´s. if you have big VM´s and/or you need differential backups, go for a traditional full featured backup tool of your choice.

tom, what traditional full featured backup tools do you suggest? xdelta was a fantastic/killer features and I think everybody is thinking to have differential backups this days. I`m looking for something like that because I need to run hourly backups for all VMs. To bad you guys don't offer it anymore...

 

[dietmar]

and I think everybody is thinking to have differential backups this days. I`m looking for something like that because I need to run hourly backups for all VMs. To bad you guys don't offer it anymore...

You already tested lessfs?

 

[udo]

tom, what traditional full featured backup tools do you suggest? xdelta was a fantastic/killer features and I think everybody is thinking to have differential backups this days. I`m looking for something like that because I need to run hourly backups for all VMs. To bad you guys don't offer it anymore...

Hi,
I don't know xdelta, but i have good experience with bacula as backup solution (we switched from netwoker to bacula since over one year).

Udo

 

[dimi]

You already tested lessfs?

Yes sure!! And works great, in this days i have try the replication feature and works well, the only problem is related to the way how replication works at the moment is synchronous copy so if you copy something in the lessfs filesystem it try to sync replication to slave server in this way if you slave have a small Bandwidth your local copy still wait the transfer to remote site.

I hope this limitation will solved in new release.
Thanks so much to all work on ProxMox

 

[monster]

Have a look at this other thread http://forum.proxmox.com/threads/5386-Incremental-backup where I posted a link to a script that uses xdelta3.

 

[dimi]

Have a look at this other thread http://forum.proxmox.com/threads/5386-Incremental-backup where I posted a link to a script that uses xdelta3.

Thanks for your suggest and i think is the simple and cost effective solution to make a backup solution, but deduplication is much more interesting from the point of Space waste and bandwidth save.
Many thanks and Happy New Year!!

 

[monster]

Thanks for your suggest and i think is the simple and cost effective solution to make a backup solution, but deduplication is much more interesting from the point of Space waste and bandwidth save.
Many thanks and Happy New Year!!

I must be missing something ... I understand that deduplication save space, but how does it save bandwidth? If I download a deduplicated full backup from my Internet server into my home test server, with sftp or rsync, then a full backup goes over the net. How else could it be?

Happy new year to you as well!

 

[dimi]

Save Bandwidth? Simple because Dedup works on Block/chunk Similarity so for example on my system if i have 10 KVM installation of redhat linux probably every bin file is equal to all virtual machine, so i save for example 1 GiB of data for every 9 VM.
Example:
- Centos 5.5 with server package circa 2GiB of filesystem space
if i have 10 of this machine i need 20GiB of Diskspace to backup it!! Using Dedup probably i can only have 2/3GiB of allocated space.
So probably for every full/Inc backup made every day i need a lot of space but on Dedup world only the Block/Chunk variation will be saved, in this way after the first Big bandwitdth Sync with my replication server the others probably will very small byte.
What do you think about!!
Many thanks

PS: Try LessFS with Master/slave configuration and check it very nice solution

 

[rrrlasse]

Instead of xdelta and xdelta3 you could use http://www.exdupe.com/ which is multithreaded. Reaches 900 megabyte/sec on eight cores.

And, instead of doing delta diff, it uses sliding window deduplication which gives much greater "compression ratio" because it both finds repeated data between the current and previous backup and also repeated data within the current set itself. This is useful when backing up multiple VMs up in one go.

 

[dimi]

Instead of xdelta and xdelta3 you could use http://www.exdupe.com/ which is multithreaded. Reaches 900 megabyte/sec on eight cores.

And, instead of doing delta diff, it uses sliding window deduplication which gives much greater "compression ratio" because it both finds repeated data between the current and previous backup and also repeated data within the current set itself. This is useful when backing up multiple VMs up in one go.

Wuauuuu!! Fabulous tools i will try it!!! the only bad thing is not opensource..... so we cant not have on proxmox as default backup tool :-(

 

[monster]

Instead of xdelta and xdelta3 you could use http://www.exdupe.com/ which is multithreaded. Reaches 900 megabyte/sec on eight cores.

I'm afraid I don't fit in "personal use", so it's nothing for me, but thanks anyway. I'm not dead against commercial products; it's just that they usually only actually save you money on a large scale, not on a small-to-medium one. 15 USD per month is fine, but 500 USD isn't, and most products fall in the later category.

 

[monster]

PS: Try LessFS with Master/slave configuration and check it very nice solution

Your reply doesn't really answer my question in a clear way. Are you saying that LessFS comes with some tools that not only manage a local file system, but also provide efficient synchronization between computers over a network? So you then use those tools instead of say, rsync, to get your files? If that is the case, do they still work well when the "copy" (slave?) is connected to the Internet with a "lowly" DSL, instead of being part of the LAN of the Master? By "well" I mean not much worst then rsync under the same condition.

 

[dimi]

Your reply doesn't really answer my question in a clear way. Are you saying that LessFS comes with some tools that not only manage a local file system, but also provide efficient synchronization between computers over a network? So you then use those tools instead of say, rsync, to get your files? If that is the case, do they still work well when the "copy" (slave?) is connected to the Internet with a "lowly" DSL, instead of being part of the LAN of the Master? By "well" I mean not much worst then rsync under the same condition.

Hi Monster! LessFS provide two configuration mode, one stand alone and another Replication. On replication mode we need two site one as a Master and a remote one as Slave (for example in my backup case XDSL connection).
If i put on LessFS Master a VM backup the data is sync to Slave side, at the begging there is no much bandwidth save because no dedup data but as soon there are dedup data only this value is sent via internet to SLAVE.
I hope to explain much better what LessFS is able to do.

 

[tom]

lessfs looks quite interesting but I cannot find any good documentation so far. who is using lessfs? where is the best place for support and howto´s?

 

[dimi]

lessfs looks quite interesting but I cannot find any good documentation so far. who is using lessfs? where is the best place for support and howto´s?

I'm sorry but there is only two site for lessfs:
- http://groups.google.com/group/lessfs
- http://www.lessfs.com/

I think is good point to start.
Sincerely