VMA archive restore outside of Proxmox

T

tarax

Member
Apr 2, 2010
43
6
8
Hi,

After reading the (sparse) documentation in the wiki and in repository as well as a couple of threads here in there forum, a couple of thrilling questions raised in my mind:
  • Is it possible and how do you restore VMA archives outside, I mean without, a recent PVE install ?
  • Is there a GUI/CLI option to fallback to old XXth century tar backup method ?

Thank you in advance for your enlightenments
Bests
 
thank you for your quick answer.

I guess the 'no' answer to the second question is implied !

As VMA is in a branch of your Qemu tree... are there any plans to submit it upstream ? What's upstream attitude about this ? Will the major issue raised by your answer be later solved by some qemu tool (qemu-img ?) allowing to extract VMAs to any VM disk image format ? Or do you plan to package a VMA manipulation tool separatly ?

Thanks in advance
Bests
 
AND YOU CALL THIS BACKUP ??!!!!

Please believe it's hard to say so because I'm so much grateful for the work done over the years providing us this great platform. But this switch to a new backup archives format couldn't have been done more badly :(
- For first and at the very least, this change should have been announced far far before it had happened and widely spreaded about,
- second, we are talking about BACKUP there, and your answer is a format only usable on on PVE install ? Man, I'm sure you have already known one of those nightmarish time when you have to restore... and you know add the dependency on a PVE install ?! And this is without the fact that it's a spit in the face of standards and portability on such a matter :(
- third, you change a format allowing WAN transfer and archives storage optimizations for another one that don't... WTF ?!
- fourth... well last, I'm so disappointed... you make this change in one go, dropping support of the BACKUP system all your users are relying on, eventually breaking setups without a blink, not even giving you the benefit of one or two releases of feedback and the time to write the tools you should have coded from the start.

What a waste of time, energy and money. You had a working, portable system built on rock solid foundations and instead of further polishing the existing codebase, completing partial features, improving project documentation and integration capabilities... this is what you do !

And now, it's our turn to loose time, energy and money to develop an alternate backup system or explain our clients that we can't upgrade the "so reliable" environment we sold them even a few month ago.

Have a nice day, mine is ruined
 
Last edited:
  • Like
Reactions: Lonnie and abkrim
tarax said:
AND YOU CALL THIS BACKUP ??!!!!
Click to expand...

...yes, I could call this bacukp, too, as long as it works. ;)

interoperable, standard-based, backwards compatible backup? no, probably it isn't.

(I also feel the "vma" change was not so cleanly done: I wrote the VMA wiki page myself, after some googling, just to help other people to know more about vma format.)

pve devs did this change for good reasons (distributed storage efficiency and snapshots, if I got it well), of course, and they could let people use the old tar format for vm not using ditributes storage, only this would have increased their maintenance burden... I feel that they could be asked to make a choice of that kind, or to make a vma > tar... conversion tool of some kind, at least if sponsored by someone really needing it. Or, anyone else could do that, since the vma specifications shoud be available, I think.

see this old thread also http://forum.proxmox.com/threads/12419-CLARIFIED-Get-rid-of-tar-backup-Why

Marco
 
the new backup works great, solved a lot of problems for us and our customers.

instead of crying like a baby make sure that your backup strategy is working, do testing and make sure you can do restores.

the format is open, so if you want to add other tools feel free to develop it. we and our community developers do not need this, so we do not developed it so far. as you are not on our customer list, you probably missed some info and direct communication with our team. feel free to join.

I recommend you calm down and respecting the forum rules, be friendly and thankful for help.
 
tarax said:
second, we are talking about BACKUP there, and your answer is a format only usable on on PVE install ?
Click to expand...

vzdump is a backup tool for VMs running on PVE. The requirement that you need PVE to restore is quite natural.
 
dietmar said:
vzdump is a backup tool for VMs running on PVE. The requirement that you need PVE to restore is quite natural.
Click to expand...

While your argument seems fair, i still do understand tarax arguments on this matter.
Maybe it would be helpful if you could release the vma binary as deb, so anybody who wants to extract a *.vma.lzo file (lets say on a fileserver or directly on a storage server) could install this package and extract a PVE backup archive.

Anyway Proxmox is still a great!
Keep up your good work!
 
  • Like
Reactions: elek02 and Lonnie
dietmar said:
Sure, we just need to find someone to do that work (or who pay for that work).
Click to expand...

Seems fair :)

While i sadly don't have the skill to package that, i found a way to extract images from a vma.lzo archive on a standard debian wheezy installation.

My guess was that i could take the vma binary to another machine (in my case debian 7.0 minimal) and install all dependencies that vma might need.
Here's what i've done:

- Add the pve repo (i think it's needed for a custom libiscsi1 package and for librbd1):
echo deb "http://download.proxmox.com/debian wheezy pve" >> /etc/apt/sources.list
wget -O- "http://download.proxmox.com/debian/key.asc" | apt-key add -

- Copy the vma binary from a pve host (its located in /usr/bin/vma)
scp root@pvehost:/usr/bin/vma /usr/local/bin

- Install some dependencies:
aptitude install libglib2.0-0 libiscsi1 librbd1 libaio1 lzop

- Some dependencies maybe not needed but i installed them, because i thought that would fix a problem i stumbled upon as i at this stage executed the copied vma binary
So if you encounter some problems by invoking the binary, try with the following packages installed:
aptitude install open-iscsi qemu gcc-4.7

- Then i copied a vma.lzo backup archive to my test machine

- I extracted first the lzo archive:
lzop -d vzdump-qemu-111-2013_06_12-11_43_15.vma.lzo

- This gives you the vma archive vzdump-qemu-111-2013_06_12-11_43_15.vma

- extract the vma by invoking:
vma extract vzdump-qemu-111-2013_06_12-11_43_15.vma -v /test/lala

This creates the folder lala (in the preexisting /test folder) with the config file in it and i my case the backed up raw file

To check the outcome i mounted the first partition of the raw file and so far it looks good ;)
All files seem to be where they should be and on my PVE testhost it booted fine :cool:
Of course you could pipe the outcome of lzop -d to vma, but i haven't tried that.
 
Last edited:
  • Like
Reactions: chrone
should something like this reasonably work, for someone needing what the OP was looking for?

/*add pve repository/key*/
echo deb "http://download.proxmox.com/debian wheezy pve" >> /etc/apt/sources.list
wget -O- "http://download.proxmox.com/debian/key.asc" | apt-key add -

/*find which pve package has vma*/
#dpkg -S /usr/bin/vma
pve-qemu-kvm: /usr/bin/vma

/*just download the deb package (in the current folder)*/
#apt-get download pve-qemu-kvm

/*verify the name of the deb package*/
#ls ./pve-qemu-kvm*
./pve-qemu-kvm_1.4-13_amd64.deb

/*extract the vma file from package*/
#dpkg --fsys-tarfile ./pve-qemu-kvm_1.4-13_amd64.deb | tar xOf - ./usr/bin/vma > ./vma

/*make vma executable*/
#chmod u+x ./vma

/*(solve dependencies)*/
#aptitude install libglib2.0-0 libiscsi1 librbd1 libaio1 lzop

/*( if you encounter some problems by invoking the binary,)*/
#aptitude install open-iscsi qemu gcc-4.7

/*copy lzo backup archive locally*/
cp /path/to/backup/file.vma.lzo .

/*extract lzo archive (gives you vma archive)*/
#lzop -d ./file.vma.lzo

/*extract vma archive*/
#vma extract ./file.vma -v ./vmaextract

/*list extracted content*/
#ls ./vmaextract
./config_file
./raw_backup_file

I'm going to try...

Marco
 
  • Like
Reactions: 1 person
Hi,

I told my gratefulness right from the start. I advocated PVE for years to everyone interested I've talked to and on all social networks I'm active on.

I wish I was a Proxmox customer but this is not my decision not to pay and, in any case, calling your users "baby" is surely not the good way to convert them...

I only expressed my deep disappointment and (sadly) radically different point of view on this matter, insulted no one and don't see in which way I may have infringed the rules of this community.

I'm not the kind of admin who never wants to change anything. No, I'm thankful for the initiative and work and excited by the potential of a new modern, specialy carved VM archive format... as long as there are open, portable, standalone libs and tools to handle it ! Who cares about backup, hmm ? No one does, but the Admin really cares about RESTORE ! And because restore always happen in the worst situations, the best backup system is not the one with the fancier features, but the one that will save you lights out, because you can manipulate your data the way you need it in these hard moments, on any computer available. No further than 10 days ago, having a simple raw disk image in a tar archive allowed me to gain access to a client's datas on another server (the only available with enough space on it, and not "proxmoxable" of course), all remotely and in "no" time... this is what _I_ call backup ! This is what you had, this is what you've thrown away until one can apt-get/yum/brew/whatever install vma on any OS or distro.

I don't mean that you should do it yourself but at least this should be cleanly well separated in an independent repository available for others to make the port/packaging/distribution. Instead of this, your innovation is buried in a code base of which upstream doesn't want and the tool is not even easily installable (like add PVE repo && apt-get update && apt-get install vma) on your mother Debian.

Truly

PS: @m.ardito, @flow thank you for the documentation and support
 
Last edited:
  • Like
Reactions: gpshead, Lonnie and alc
The new backup is far more better than the old one, especially because the old backup is not working for a lot of setups (Ceph, NFS, sheepdog, etc.).
BTW, there are thousands of satisfied vma backup users. If you go through the forum posts you will find hundreds of posts and issues, caused by the too complex LVM snapshot based backups (full snapshot, wrong custom partitions, not suited storage, etc.).

If it does not work for you, you probably found a bug or your configuration is wrong - as always in the software world.
Your point of view that we are doing it completely wrong is probably only a valid view point from your side.

As the new backup system is open source, everybody can develop a missing feature. So if you miss something, there are several ways to achieve improvements, I do not see that we limit other people doing the development.

Writing again and again that we are doing everything wrong in the backup development is for sure the wrong way here and is not a suitable way to make software better.
 
tom said:
The new backup is far more better than the old one, especially because the old backup is not working for a lot of setups (Ceph, NFS, sheepdog, etc.). BTW, there are thousands of satisfied vma backup users.
Click to expand...

Excellent !

If you go through the forum posts you will find hundreds of posts and issues, caused by the too complex LVM snapshot based backups (full snapshot, wrong custom partitions, not suited storage, etc.).
Click to expand...

LVM snapshots (over DRBD devices) and tar.lzo backups are doing exactly what I need and want, and they're doing it well... AFAIC !

If it does not work for you, you probably found a bug or your configuration is wrong - as always in the software world.
Click to expand...

I'm afraid my english is even worth that I've ever thought... I _never_ said it was not working, buggy or made any quality statement about the code.
I'm saying that it puts too heavy requirements/dependencies on a restore system to my taste. Again, with it, 10 days ago, instead of solving my users' problem in a couple of hours straight from my desk, I'd have had to find them a server, drive to their place 10's of Km away, install a PVE...
I'm saying that I'm frustrated because I have to choose between upgrading and enjoying/benefiting all the great work you've done during the last two releases on one side, and the peace of mind and security of a secular, portable, versatile _working_ backup system.

Your point of view that we are doing it completely wrong is probably only a valid view point from your side.
Click to expand...

This, I neither ever said ! Again, I'm very welcoming the development of a modern specialized archive format to free us from tools designed in the stone age of computing when "big" meant a bunch of Ko ! Again, I'm grateful to Proxmox clever brains to have spotted this issue and put their time and energy to find an answer. Again, I'm deeply thankful for sharing your solution as FLOSS code with Humanity !

As the new backup system is open source, everybody can develop a missing feature. So if you miss something, there are several ways to achieve improvements, I do not see that we limit other people doing the development.

Writing again and again that we are doing everything wrong in the backup development is for sure the wrong way here and is not a suitable way to make software better.
Click to expand...

Again, that is absolutely not what I'm saying. For first because I have too much respect for work I'm not able to do myself and made available to me for free.

I'm only saying that I'd have wished:
- both systems had been maintained for a handful of releases, at least while the following point are not OK and to give us a few month to migrate production systems
- your innovation had been developed in a dedicated, independent repo to ease community ports to other OSes/distros, and contributions
- vma had been packaged (in PVE's repository) as an individual package (pve-qemu-vma ?) with the least dependencies on Proxmox specific packages/libs so that it can be easily "apt-getable" on a vanilla Debian

Anyway, please accept my apologizes if my words hurted any of you in any way.
Thank you again for making our environment of choice
Bests
 
  • Like
Reactions: alc
Dietmar is trying to push live backup in qemu upstream code since some months, but this take some time, because qemu dev don't want new format, want that it's works like they want,etc...

So, maybe for qemu 1.6 or qemu 1.7, it'll work on any distribution using last qemu.
 
Ok, this better explain inclusion in qemu code base... and further would be a great contribution to it as well as it would solve the availability on other platforms.

Hope qemu devs Will accept it soon

Thanks spirit for this clarification
Keep it on yall
 
Marco, Tarax, Flow,


Thanks for your efforts on VMA.

I regret Tom's tone in your exchanges (* offtopic and in the bottom) and thank you for the persistence and tolerance to keep up with his remarks. He's clearly under a lot of stress... Dietmar should take over the forum ;)In the last weeks I became positively impressed with Proxmox (knowing most other virtualization solutions) and in the last hour, shocked by these exchanges with Tom surrounding the VMA format - (the arrogance, not the technical side)

How did this work for you? Any way to automate it? Pitfalls?

For others google-reaching here because:
- wanting to do incremental backup of the new VMA format - http://ayufan.eu/projects/proxmox-ve-differential-backups/ [Thanks to Flow for point out]
- wanting to extract the filesystem in order to "convert" VMA snapshot formats to VMWare, VirtualBox, XEN, etc, formats VMDK, VDI, etc, this, below seems to be the best suggestion yet.
(by Tarax , adapted by Marco ; unconfirmed)




SMacedo
m.ardito said:
should something like this reasonably work, for someone needing what the OP was looking for?/*add pve repository/key*/echo deb "http://download.proxmox.com/debian wheezy pve" >> /etc/apt/sources.listwget -O- "http://download.proxmox.com/debian/key.asc" | apt-key add -/*find which pve package has vma*/#dpkg -S /usr/bin/vmapve-qemu-kvm: /usr/bin/vma/*just download the deb package (in the current folder)*/#apt-get download pve-qemu-kvm/*verify the name of the deb package*/#ls ./pve-qemu-kvm*./pve-qemu-kvm_1.4-13_amd64.deb/*extract the vma file from package*/#dpkg --fsys-tarfile ./pve-qemu-kvm_1.4-13_amd64.deb | tar xOf - ./usr/bin/vma > ./vma/*make vma executable*/#chmod u+x ./vma/*(solve dependencies)*/#aptitude install libglib2.0-0 libiscsi1 librbd1 libaio1 lzop/*( if you encounter some problems by invoking the binary,)*/#aptitude install open-iscsi qemu gcc-4.7/*copy lzo backup archive locally*/cp /path/to/backup/file.vma.lzo ./*extract lzo archive (gives you vma archive)*/#lzop -d ./file.vma.lzo/*extract vma archive*/#vma extract ./file.vma -v ./vmaextract/*list extracted content*/#ls ./vmaextract./config_file./raw_backup_fileI'm going to try...Marco
Click to expand...
* on Tom's tone:


Finally, to understand Proxmox's Tom contributor stance on this , have a look at:
http://forum.proxmox.com/threads/13...hode-vma-not-rdiff-friendly?p=72868#post72868
"Instead of complaining about the really great docs (e.g. compared with similar projects) you should start contribution" - and Marco did:
http://pve.proxmox.com/wiki/VMA


Another uncalled for response from Tom:
http://forum.proxmox.com/threads/14226-VMA-archive-restore-outside-of-Proxmox?p=76369#post76369
"I recommend you calm down and respecting the forum rules, be friendly and thankful for help."


and the now famous "crying lika a baby" one:
http://forum.proxmox.com/threads/14226-VMA-archive-restore-outside-of-Proxmox?p=76369#post76369


Simple emails like "This is the best for your customers. Our customers pay us, you don't. Please deal with it - you're lucky enough to use our work" - this sort of thing, though harsh, would be ok imho; not the almost insulting rubbish above...
 
Last edited by a moderator:
  • Like
Reactions: Joshua M and gpshead
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back