Proxmox 2.3 new backup methode vma not rdiff friendly

P

pva00

New Member
Apr 20, 2009
21
1
1
Hi,

First of all you are doing a great job with proxmox....

I recently upgraded to proxmox v2.3 with the new upgraded backup method. It is working OK but I noticed an increase of storage use for my backups.....

I created a backup script which locally on the proxmox host backups (every day) each individual VM (I only use KVM qcow2) and rename the backup file to a standard name (no date/time information included). On this file I'll do an rdiff-backup to an offsite location on which I have backups stored with a retention on 16 days. Rdiff-backup has the functionality to only copy the increment and store only increments of files, so I use as less as possible network bandwidth and backup storage.

With proxmox v2.2 (tar.lzo) my 16 days retention data use is:
Wed Mar 6 22:28:25 2013 20.6 GB 20.6 GB (current mirror)
Tue Mar 5 22:28:14 2013 1.35 GB 22.0 GB
Mon Mar 4 22:31:38 2013 1.21 GB 23.2 GB
Sun Mar 3 22:30:41 2013 1.38 GB 24.6 GB
Sat Mar 2 22:28:32 2013 881 MB 25.4 GB
Fri Mar 1 22:32:01 2013 1.10 GB 26.5 GB
Thu Feb 28 22:33:52 2013 934 MB 27.5 GB
Wed Feb 27 22:40:23 2013 1.29 GB 28.7 GB
Tue Feb 26 22:34:02 2013 1.27 GB 30.0 GB
Mon Feb 25 22:32:09 2013 1.23 GB 31.2 GB
Sun Feb 24 22:29:41 2013 1.07 GB 32.3 GB
Sat Feb 23 22:28:10 2013 1004 MB 33.3 GB
Fri Feb 22 22:41:06 2013 490 MB 33.8 GB
Thu Feb 21 22:27:54 2013 1.18 GB 35.0 GB

But with proxmox v2.3 (vma.lzo) my 16 days retention data use is:
Wed Apr 10 21:59:32 2013 20.3 GB 20.3 GB (current mirror)
Tue Apr 9 22:28:18 2013 16.3 GB 36.5 GB
Mon Apr 8 22:09:54 2013 15.9 GB 52.5 GB
Sun Apr 7 22:12:32 2013 15.2 GB 67.7 GB
Sat Apr 6 22:10:37 2013 15.9 GB 83.6 GB
Fri Apr 5 22:10:11 2013 15.8 GB 99.4 GB
Thu Apr 4 22:22:51 2013 15.4 GB 115 GB
Wed Apr 3 22:15:59 2013 15.8 GB 131 GB
Tue Apr 2 22:15:15 2013 14.9 GB 145 GB
Mon Apr 1 22:13:20 2013 15.5 GB 161 GB
Sun Mar 31 22:15:31 2013 16.0 GB 177 GB
Sat Mar 30 22:14:02 2013 16.0 GB 193 GB



I investigated it (tested with gzip (no increments, option --rsyncable should be used) and no compression (same result as lzo but without compression)) And my conclusion is the use of the new vma container is not rdiff friendly.

I request you to make vma more rdiff friendly or add an additional option to vzdump command to choose between the tar or vma container (knowing the advantages and disadvantages between tar old backup method (local LVM required for online backup, but rdiff friendly) or new method (online backup with non local LVM storage, but not rdiff friendly)

I'll hope you'll consider the possibilities...
Thanks in advance
 
yes, the new backup is different by design and cannot be made more "rdiff friendly", sorry.

re-organize your backup strategy.
 
as a suggestion: if you use qcow2 images anyhow, why not let KVM/qcow2 deal with differential writes by using snapshots / base images?
 
i understand that supporting 2 different backup modes is too much ;) but i can't find any word of warning in pve wiki http://pve.proxmox.com/wiki/Backup_and_Restore about such much larger disk usage :confused: (a user upgrading should at least be warned, imho) and other drawbacks after switching from tar to vma format... where can i find info about this? http://pve.proxmox.com/wiki/Special:Search?search=vma returns practically nothing, its just the "vma" letters in the above page, without any kind of explanation...
do you expect users to easily find this https://git.proxmox.com/?p=qemu.git;a=blob;f=docs/backup.txt;h=927d787a351221820ed66d149762ac48b340184f;hb=2f0dcd89a0de8b656d33ce6997c09879bd287af7 and live with that? :(

i can't spread enough enough the word about proxmox (using and loving since v1.5) but guys, info and documentation is not you strong point... :)
does one get better documentation with a support subscription active or else how can one be authoritatively warned about those "big" changes?

Thanks, Marco
 
Last edited:
There is no larger disk usage and therefore no reason for a "big warning". As this use case is not a Proxmox VE functionality its not documented - we try to document features and not "not existing features".
(Renaming and rsyncing backup files is not a Proxmox VE functionality).

Instead of complaining about the really great docs (e.g. compared with similar projects) you should start contribution - just create a wiki account and start writing. Documentation could always be better but we are not bad here. And you as a expert got always the chance to read the source code.

Backup is a huge topic and the integrated vzdump based is very usable for image snapshot but never feature a complete backup and archive solution like Bacula, Sep, Symantec, Arceia, etc.

There are already some ideas to create the perfect backup storage (including replication to remote sites, also dedup) - but as the world is already full with great backup and archiving solutions no one like to do the work (or finance) as the topic is already covered by a lot of other projects.
 
Tom i'm sorry if my post seemed a bit harsh, and I apologize. I thought i put smileys enough :-)

i do contribute, though, both here on forums (see profile info) and on pve wiki (http://pve.proxmox.com/wiki/Special:Contributions/M_ardito), as much as my skill and knowledge, and free time allow me ;-) but i'm just a user...

i'm still conviced that this vma change is not enough documented, at least atm, am i wrong?
i understand that vma:
- is now the only bacukp method in the current stable release
- can not restore previous version's tar backups
- is not a common standard but a proxmox newly defined format
- has many more capabilities than previous tar method, but perhaps some drawbacks, or a t least a different approach that shold be well known by users, an the thread starter posted a huge increase in backup space in his real usage scenario, isn't it? is he doing something wrong? what?

i just don't understand where is vma documentation? where it was stated that vma replaced tar? release logs? wiki? sticky threads? there is no vma reference in wiki and i only found this doc in git (thanks to another forum's post, this http://forum.proxmox.com/threads/13093-new-backup-format?p=72415#post72415)
the git doc that's suggested, this https://git.proxmox.com/?p=qemu.git...f;hb=2f0dcd89a0de8b656d33ce6997c09879bd287af7

which cites this <<We have defined a very simply format with those properties, see:docs/specs/vma_spec.txt>>

but can't find docs/specs/vma_spec.txt, maybe it's me...?

i just think that vma is not documented well, if at all. Maybe some other feature also. If you think pve is great and there is plenty of info i agree, but that is not good docs, imho. There are forums, wiki, git, etc and not a single place with versioned features and info. You think that's hard to do, and for free? yes, i agree, still i think what pve has is not really good docs. You could charge for docs, crowfound them, apply for google SoC or whatever, i would appreciate any way if i could have better docs.

my 2c, peace,
Marco
 
Understanding the technical difference of tar and vma is not that easy and by far too complicated for a end users documentation (for developers, the best doc is the source code). but the link to the git explains the reasons why tar is not suited.

Back to this thread topic: the user is doing off-site backups but each vma backup file is unique with no similarity to the vma from the old backup, so always the full file needs to be moved to the remote location.

- - - Updated - - -

Just to add, a restore of old tar.gz backup files (from 2.2 and before) is still possible!
 
Just a comment, I also tried to find the vma spec and followed the dead link. Maybe it's simply missing.
 
So, just to make sure I understand, the new format is not incremental-backup friendly and there is no alternative method to do incremental backups? I have to say this is a huge downside in the current way of operating, are there any plans to fix this? Are we expected to copy our entire VM images over slow network links every night?
 
Hi,

I've got the same problem using vma and rsync.
I solved it by extracting the vma file on the fly using my own backup script.

The only bad thing is, that it seems not possible to compress the extracted vma on the fly.

Code: 
rm -rf $PATH/$VMID
vzdump $VMID --mode snapshot --compress 0 --script /opt/vmbackup/fsfreeze.pl -stdout | vma extract - $PATH/$VMID
pigz $PATH/$VMID/*
 
dietmar said:
The whole project is open source. So feel free to improve the code if you have ideas.
Click to expand...

Where can I find the source code for the VMA tool? I'd like to compile it on a different platform so I don't have to install an entire pve to get the vma extraction functionality. I've been browsing trough the qemu repository, but I haven't found it yet.
 
Can you be a little bit more specific? The only link to git is to a spec file, I need the source code for the vma binary itself so i can compile it. i've been searching trough the pve-qemu-kvm repository, but I can't find it. Is there a vma.c somewhere? In what repository can I find it and in which directory?

Thanks
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom