Proxmox Backups - An option to not use HCP Snapshots

[marotori]

Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

Hello,

Will it be possible to get an extention added in to the backup system. (vzdumps)

Essentially.. rather than running an LVM snapshot..... run and HCP snapshot?

The HCP snapshot system 'is' free software but not GPL.

So yes.. get the point not to install the software by default. But the option for the vzdump scripts to support this would be good?

Why?

They work! I have found recently LVM snapshots to be very broken. no idea if this is a kernel bug - hardware issue - or what - but they cause my machine to die - and a reboot to be needed. Seems this issue is occcuring is loads of distro's.

The also work on any partition. So you are not forced to use EXT3. if you want a different file system. No problem!

Anyway.. would be great to know thoughts on this?

In theory the chance is simple.

An example:
lvcreate -d -L 2G --snapshot --name pve-data-snap /dev/pve/data ; mount -o ro /dev/pve/pve-data-snap /var/lib/vz.snapshot

becomes:
hcp -m /var/lib/vz.snapshot /dev/mapper/pve-data

Simple?

Rob

Note. Ty

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

have you tested this?

i am interested in implementing a working backup system because i am tired now of system hangs.

it would also be great if the proxmox staff confirms HCP is safe to use.

thanks
u.

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

I have used the latest version of HCP not problem.

My backups simply use bacula.. then a before and after script to run an hcp snapshot /remove on the /var/lib/vz volume.

No hangs.. lockups.. just works.

IMHO - LVM is very broken. Yes.. it may work.. but it seems that the 'risk' of a hang (1 out of every 3 backups for me) is too high for a production backup.

The way I see it is that either.. something like an HCP option is included into VZdump... or I will just write my own 'hcpvzdump' script.

Rob

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

then, as i have moderate administrating knowledge, could you give a short tutorial on implementing a backup system like yours?
i don't want to break my system

thanks
u.

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

Do you know how to use bacula?

It is a bit of a beast!

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

well, didn't try it, yet
you could help me out with a simple cron script, i think.

ty
u.

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

I have not yet written a replacement for vzdumps to do this.

My original request you will see is to suggest this is added as an option

I will be doing something to get this working soon!

Rob

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

yes i know, i thought you could share your current backup script.
in the meantime, i will learn something about bacula (and hcp)

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

There is really not much to share.

Bacula is a client server backup system; install of it is a matter of getting a central backup server; and then installing an agent on each machine. There is tons of documentation on this on the bacula site - so much so that it is impossible for me to summarise.

The key thing I am doing is using the bacula options to run a script before and after the backup.

This script used to use the lvm command to create a snapshot.. now I use the hcp command to do the backup.

e.g: hcp -m /var/lib/vz.snapshot /dev/mapper/pve-data


All this does is to take a snapshot of /dev/mapper/pve-data and mount it in /var/lib/vz.snapshot

I then backup /var/lib/vz.snapshot

Techncially.. the script is just a shell script with the 1 command in it!

Rob

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

thanks Rob,

if i don't want to play with bacula i can do it with tar, dd or whatever, i think.

if i have 2 disks for a vm i should make 2 snapshots, and then do the packups, am i right?

u.

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

Yes. Essentially that!

The only issue is you also need to grab the config files from /etc/vz/conf and /etc/qemu-server to match the machine id.

I have had a look at the vzdump script & code. It really should not be a major issue for me to extend this code. It is rather simple

The question really is.. do I have support from proxmox team to do this?

Yes.. I can make a completely independant version of the system to do this; but is makes more sense to simply add an option to the existing vzdump!

Rob

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

then,
start the snapshot,
backup vmid.conf,
backup volumes,
delete snapshot.

and how to restore? should i create an empty vm and copy the backup data to the created volumes?

thanks
u.

STAFF: could you create a section in the forums where we can discuss proxmox-related, but "basic" server administration stuff?
posting stupid questions here here may be annoying to experts but having a beginners' corner could be useful for people like me.

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

Hello,

Pretty much!

vzdump pretty much does exactly that!

Rob

 

[tom]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

Hello,

Will it be possible to get an extention added in to the backup system. (vzdumps)

Essentially.. rather than running an LVM snapshot..... run and HCP snapshot?

The HCP snapshot system 'is' free software but not GPL.

I do not see that this is free software (see http://en.wikipedia.org/wiki/Free_software). If I am wrong, please provide a link to the license terms. I assume its just freeware (http://en.wikipedia.org/wiki/Freeware)

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

Hello,

Pretty much!

vzdump pretty much does exactly that!

Rob

and how should i know the exact size of the volumes to be created?

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

Hi Tom,

I said it was 'free'. (note the quotes) It is not opensource/gpl - but you get a free licence.

I personally would prefer to use lvm - as this is all provided by the OS; but the fact is that LVM is not working properly.. for allot of people. (or rather snapshots are buggy)

I have used the r1soft product in the past - and it is very good. Light weight.. and just works! No problems

So.. my point with all of this is.

1. - dont include the software in the distro. That would break the GPL licensing that makes Proxmox so attractive.
2. - do provide an option. vzdump --mode hcp etc.. This will make the option avaliable for those who have problems with LVM; and generally make a more versatile solution

Rob

 

[marotori]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

You dont need to?

hcp simply uses empty space on the disk to store the files.

You are in effect creating a snapshot - and copying the data off it. The aim being to have a filesystem that is not being modified while you back it up. The result is that your snapshot will not actually really take any space at all!

Rob

 

[tom]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

does this work for block devices or just for virtual disk images stored on ext3/ext4 filesystems?

 

[udi]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

You dont need to?

hcp simply uses empty space on the disk to store the files.

You are in effect creating a snapshot - and copying the data off it. The aim being to have a filesystem that is not being modified while you back it up. The result is that your snapshot will not actually really take any space at all!

Rob

i meant the restore, i have the backup files, then i create the vm with the disks, then copy the data to the disks.
how should i know the disk sizes to create?

or am i misunderstanding something?

 

[dietmar]

Re: Proxmox Backups - An option to not use LVM Snapshots - but HCP Instead!

I personally would prefer to use lvm - as this is all provided by the OS; but the fact is that LVM is not working properly.. for allot of people. (or rather snapshots are buggy)

I suggest to debug LVM instead (Besides, LVM works for many,many people!)