consistent frozen kvm backup with vzdump

[Mario Wolff]

Hi all,
i had a deeper view to vzdump and qm. Even in suspend mode a backup of a kvm is not in a clean state. The vm_suspend call does only a stop-call on the kvm-monitor. I would like to migrate to a file after the stop-call and add this file to the archive.

I could do modifications to the proxmox-stuff or simply write a new dumper for my use-case without any publishing. Is there a chance to get modifications committed?

I also would extend vzdump to act as a bacula-helper. ClientRunBeforeJob = "...vzdump prepare"; File="\\| ...vzdump filelist"; ClientRunAfterJob = "...vzdump cleanup".

Once more: Is there a chance to get my modifications committed?

Regards,
Mario

PS: Waiting on response to "Local console..." thread.
PPS: Is there interest on proxmox-side to get contributed code?

 

[dietmar]

Hi all,
i had a deeper view to vzdump and qm. Even in suspend mode a backup of a kvm is not in a clean state. The vm_suspend call does only a stop-call on the kvm-monitor. I would like to migrate to a file after the stop-call and add this file to the archive.

I could do modifications to the proxmox-stuff or simply write a new dumper for my use-case without any publishing. Is there a chance to get modifications committed?

I am not particularly interested in that feature. IMHO, the only way to get 100% consistent backups is to use 'stop' mode (vmstate contains time, which is old when you restore, and can lead to many errors).

I also would extend vzdump to act as a bacula-helper. ClientRunBeforeJob = "...vzdump prepare"; File="\\| ...vzdump filelist"; ClientRunAfterJob = "...vzdump cleanup".

Yes, that sounds interesting. But what is 'vzdump prepare' expected to do?

Once more: Is there a chance to get my modifications committed?

PPS: Is there interest on proxmox-side to get contributed code?

yes

 

[Mario Wolff]

I am not particularly interested in that feature. IMHO, the only way to get 100% consistent backups is to use 'stop' mode (vmstate contains time, which is old when you restore, and can lead to many errors).

True if your vm has working acpi-support that takes the shutdown call, false if it stays up and gets killed. Stop-start cycle takes much more time than stopping, migrate to file and continue!

What errors do you expect? We use this stuff for about one year without proxmox on vmware-server-1.x and bare kvm. We take a backup every workday and had about 50 restores. Restore is most for cloning - restore and rename, run with halted cpu, disconnect network, continue, reconfigure on console, shutdown and rerun. Never had problems!

Yes, that sounds interesting. But what is 'vzdump prepare' expected to do?

Suspending/stopping/snapshot creation.... all the stuff vzdump does before calling vmtar...

 

[dietmar]

Never had problems!

That is the point - i never had any problems using vzdump as it works today - there is simply no need to store vm state. Snapshot/suspend mode is good enough. Or what errors do you expect?

Suspending/stopping/snapshot creation.... all the stuff vzdump does before calling vmtar...

Don't they have an interface to pipe the output of 'vzdump' to the tape/backup server?

 

[Mario Wolff]

Don't they have an interface to pipe the output of 'vzdump' to the tape/backup server?

I've taken your comment to to redesign my bacula plans with using vzdump and qmrestore. I've started with some minor changes to let them support writing to stdout and reading from stdin.
There is a new option for vzdump named "--stdout" and qmrestore treats '-' as "read from stdin" and not as a filename. It should be clear, that --stdout is only usable for single archive operations!
I think this changes could be useful for other backup-solutions too.
A patch is applied!
Regards,
Mario

 

[dietmar]

I've taken your comment to to redesign my bacula plans with using vzdump and qmrestore.

Please can you describe how that can be used with bacula now?

It would be great if you download the source packages, and send me diffs against them ("diff -u")

see: ftp://download.proxmox.com/sources/

 

[Mario Wolff]

Please can you describe how that can be used with bacula now?

To say it like the proxmox-people:
See http://wiki.bacula.org/doku.php?id=...kups:postgresql#single_pass_backup_with_bpipe to get an idea.

I would like to have a "qm rename <vmid> <new-vmid>". Are there any plans to implement something like this?

Regards,
Mario

 

[dietmar]

I would like to have a "qm rename <vmid> <new-vmid>". Are there any plans to implement something like this?

No.

 

[Mario Wolff]

Please can you describe how that can be used with bacula now?

If my patches gets committed, i would place a description in your wiki. But i will not cover a bacula-fd update that in fact is required. With lenny comes bacula-fd 2.4.4, current is 5.0.3(!) and to use bpipe-plugin you need at least 3.x.

It would be great if you download the source packages, and send me diffs against them ("diff -u")

See attached files! Like this or only a file-diff?

 

[dietmar]

If my patches gets committed, i would place a description in your wiki. But i will not cover a bacula-fd update that in fact is required. With lenny comes bacula-fd 2.4.4, current is 5.0.3(!) and to use bpipe-plugin you need at least 3.x.

well, squeeze will have 5.0.2

See attached files! Like this or only a file-diff?

thanks. I will try to merge that next week.

 

[Mario Wolff]

Just a note: With vzdump/qmrestore-stdio capabilities you can also simplify cloning!

vzdump --stdout --snapshot 101 | qmrestore --unique - 102

 

[dietmar]

Just a note: With vzdump/qmrestore-stdio capabilities you can also simplify cloning!

Yes, I already noticed that - nice.

Just uploaded a new version to the pvetest repository. Also uploaded new sources to the ftp server. Please can you test?

Sometimes I get error 141 from tar (seems SIG PIPE related, but error is not reproducible anymore).

- Dietmar

 

[Mario Wolff]

After some days of successful backups of multiple vms on to pve-hosts and one restore I think the solution is working!

I could not reproduce your broken pipe problem.

What is the recommended way to install packages from pvetest? It looks like the same distribution in both repositories. I have added the pvetest to my source.list, reread the repositories, reinstalled qemu-server and vzdump and commented the pvetest in the source.list. Is this the right way?

Would you recommend to keep the documentation more straight and simple or more comfortable use but more complex to implement?

Would it be possible to get a lenny bacula 3-5 file-daemon into your repository? That would simplify the required steps for a bacula-bpipe-proxmox setup!

Regards,
Mario

 

[dietmar]

What is the recommended way to install packages from pvetest? It looks like the same distribution in both repositories. I have added the pvetest to my source.list, reread the repositories, reinstalled qemu-server and vzdump and commented the pvetest in the source.list. Is this the right way?

That way you get all packages from 'pvetest'

Would you recommend to keep the documentation more straight and simple or more comfortable use but more complex to implement?

keep it simply - but what exactly do you plan to do?

Would it be possible to get a lenny bacula 3-5 file-daemon into your repository? That would simplify the required steps for a bacula-bpipe-proxmox setup!

Why don't you use the backports repository?

 

[apmuthu]

Alter string in patch:

dos not lock like a QemuServer vzdump backup

to be:

does not look like a QemuServer vzdump backup

 

[dietmar]

Alter string in patch:

Oh - just found the typo - thanks for that hint.

 

[Mario Wolff]

That way you get all packages from 'pvetest'

No, only that both packages! Why is that distribution lenny an not pvetest? With that one can use pinning!

keep it simply - but what exactly do you plan to do?

If you had a look to the linked page, you may have noted that one need to set the restore-vmid when you configure the backup job. To avoid that, i have a wrapper script that takes the arguments for the next restore.

Why don't you use the backports repository?

Well, the question was just for the documentation! Bacula is too simple to compile to handle distribution mixup.

 

[dietmar]

No, only that both packages! Why is that distribution lenny an not pvetest? With that one can use pinning!

We use that repository to test the final repositry.

If you had a look to the linked page, you may have noted that one need to set the restore-vmid when you configure the backup job. To avoid that, i have a wrapper script that takes the arguments for the next restore.

How does that relate to 'simple/complex documentation'?

Well, the question was just for the documentation! Bacula is too simple to compile to handle distribution mixup.

A simple wget from the backport repository would also do the job? (no, i will not add it to our repository)