help me understanding backup snaphots on lvm storage

[m.ardito]

Hi,
i am learning how to backup/restore vm (only kvm at the moment) in pve.
i have a iscsi NAS with
- 1 iscsi target (100GB)
- 1 nfs share (100 GB)

due to the test environment network is limited do 100MB by my test server nic, obviously i'll switch to gb nics later

i created in pve:
1 iscsi storage pointing to my target ("iscsi1", with Use LUNs directly: no)
1 lvm storage on that iscsi storage ("lvm_iscsi", with content: Virtual Disks)
1 nfs storage pointing to my nfs share ("backup_vm", with content: VZDump Backups)
1 kvm vm on that lvm storage ("VMID 112", ubuntu 9.04 with 8gb hard disk, cd install, no updates)

i wish to to a live "snapshots" backup of that vm, supposing to save both disk space and time doing so.

i created in pve 1 backup definition for that vm with:
Storage: the nfs storage created above
Mode: snapshot
Compress files: yes

being on lvm storage, if i got it right, i expect the pve to do a live lvm snapshot of that vm each time the backup job runs so the first time i expect to have a "full" backup snapshot (raw image or qcow2?), the next time i expect to find another (much smaller) "diff" backup snapshot o the same vm.

i started 2 times that backup since i created the vm (always running):
- the first backup was of the freshly installed vm), when it finished i got:
these are the only files i find on the nfs share:
vzdump-qemu-112-2010_03_25-14_00_01.tgz
vzdump-qemu-112-2010_03_25-14_00_01.log

and the log was:
Mar 25 14:00:01 INFO: Starting Backup of VM 112 (qemu)
Mar 25 14:00:01 INFO: running
Mar 25 14:00:01 INFO: status = running
Mar 25 14:00:02 INFO: backup mode: snapshot
Mar 25 14:00:02 INFO: bandwidth limit: 10240 KB/s
Mar 25 14:00:02 INFO: Logical volume "vzsnap-debian-0" created
Mar 25 14:00:02 INFO: creating archive '/mnt/pve/backup_vm/vzdump-qemu-112-2010_03_25-14_00_01.tgz'
Mar 25 14:00:02 INFO: adding '/mnt/pve/backup_vm/vzdump-qemu-112-2010_03_25-14_00_01.tmp/qemu-server.conf' to archive ('qemu-server.conf')
Mar 25 14:00:02 INFO: adding '/dev/vg1/vzsnap-debian-0' to archive ('vm-disk-ide0.raw')
Mar 25 14:16:27 INFO: Total bytes written: 2067845120 (2.00 MiB/s)
Mar 25 14:16:32 INFO: archive file size: 756MB
Mar 25 14:16:34 INFO: Logical volume "vzsnap-debian-0" successfully removed
Mar 25 14:16:34 INFO: Finished Backup of VM 112 (00:16:33)

- then i updated the machine with update manager (nearly 240 MB)

- the second time, when the backup finished i got:
these are the only files i find on the nfs share:
vzdump-qemu-112-2010_03_25-15_30_01.log
Name : vzdump-qemu-112-2010_03_25-15_30_01.tgz

and the log was:
Mar 25 15:30:01 INFO: Starting Backup of VM 112 (qemu)
Mar 25 15:30:02 INFO: running
Mar 25 15:30:02 INFO: status = running
Mar 25 15:30:02 INFO: backup mode: snapshot
Mar 25 15:30:02 INFO: bandwidth limit: 10240 KB/s
Mar 25 15:30:02 INFO: Logical volume "vzsnap-debian-0" created
Mar 25 15:30:02 INFO: creating archive '/mnt/pve/backup_vm/vzdump-qemu-112-2010_03_25-15_30_01.tgz'
Mar 25 15:30:02 INFO: adding '/mnt/pve/backup_vm/vzdump-qemu-112-2010_03_25-15_30_01.tmp/qemu-server.conf' to archive ('qemu-server.conf')
Mar 25 15:30:02 INFO: adding '/dev/vg1/vzsnap-debian-0' to archive ('vm-disk-ide0.raw')
Mar 25 15:47:53 INFO: Total bytes written: 2797036544 (2.49 MiB/s)
Mar 25 15:47:55 INFO: archive file size: 1.14GB
Mar 25 15:47:55 INFO: delete old backup '/mnt/pve/backup_vm/vzdump-qemu-112-2010_03_25-14_00_01.tgz'
Mar 25 15:47:59 INFO: Logical volume "vzsnap-debian-0" successfully removed
Mar 25 15:47:59 INFO: Finished Backup of VM 112 (00:17:58)

here are my questions...
1) why the hell it deletes the old backup? can't i keep al muy backups there?
2) is there any way to do incremental "snapshot" backups?
3) why the second backup is much more than the first+updates even if compressed?

here allof the pve versions. running pve on standard lenny64

admin@debian:/var/lib/vz$ pveversion -v
pve-manager: 1.5-7 (pve-manager/1.5/4660)
running kernel: 2.6.26-2-amd64
proxmox-ve-2.6.18: 1.5-5
pve-kernel-2.6.18-1-pve: 2.6.18-4
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-11
pve-firmware: 1.0-3
libpve-storage-perl: 1.0-10
vncterm: 0.9-2
vzctl: 3.0.23-1pve8
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-5

Thanks...
Marco

 

[tom]

reading the manual will help. http://pve.proxmox.com/wiki/Vzdump_manual

see the 'maxfiles' option.

 

[dietmar]

1) why the hell it deletes the old backup? can't i keep al muy backups there?

You can use the --maxfiles option to keep more than one file (set in /etc/vzdump.conf). The idea is that you save those files to a tape somehow.

2) is there any way to do incremental "snapshot" backups?

no

3) why the second backup is much more than the first+updates even if compressed?

because there is more (non zero) data on the disk image.

 

[m.ardito]

Ok, thanks Tom and Dietmar, i understand those were newbie questions, but i am just starting with pve and all the tools

You can use the --maxfiles option to keep more than one file (set in /etc/vzdump.conf). The idea is that you save those files to a tape somehow.

ok, now i relize i should modify the configuration file, as man says:"Global configuration is stored in /etc/vzdump.conf." Do i need to restart something or vzdump will read that each run?
of course i will backup to tape saved files, but hoped to keep most recent versions there: should i assume that vzdump will keep always the more recent X files (being X the value specified for --maxfiles)?
Or? I need to know this in order to plan how to keep thinks clean and safe there.

forgive me this probably silly question:
i read also "LIMITATIONS: VZDump does not save ACLs.", can you please point out which ACLs it's referring to? i think it's the tar/tgz/log files?

m.ardito[/B];20309]>is there any way to do incremental "snapshot" backups?.

no

ok, so vzdump will not make incremental snapshots...
is there any way i could have qcow2 "incremental" snapshots on LVM storage with pve without using vzdump like in
http://www.linux-kvm.com/content/how-you-can-use-qemukvm-base-images-be-more-productive-part-1
or do you think it would be useful and could be implemented in pve somehow, sometime?

because there is more (non zero) data on the disk image.

yes, i suppose that it is a guest OS "feature"?
how can i have info about the running disk image files (used by the guest), in this cas on the LVM-iscsi storage, apart from the pve (debian command line perhaps?)?

from the pve web gui i see, for that storage
<<
Volume ID VMID Size (GB) Used (GB)
lvm_iscsi:vm-112-disk-1 112 8.00 -
>>
it reports 8.00 GB size, and used "-"...
can't get info on really used space on that machine, apart from the guest OS itself (which i think will report just its free space, not (non zero) data)
in order to understand how it is used? i mean where pve phisicaly store those files (are they RAW type images?)

i am just willing to completely understand this system before start real testing for production...

Thanks
Marco

 

[dietmar]

ok, now i relize i should modify the configuration file, as man says:"Global configuration is stored in /etc/vzdump.conf." Do i need to restart something or vzdump will read that each run?

no need to restart anything.

of course i will backup to tape saved files, but hoped to keep most recent versions there: should i assume that vzdump will keep always the more recent X files

Yes.

forgive me this probably silly question:
i read also "LIMITATIONS: VZDump does not save ACLs.", can you please point out which ACLs it's referring to? i think it's the tar/tgz/log files?

ACLs on the filesystem.

ok, so vzdump will not make incremental snapshots...

We do not plan to implement incremental snapshots now.

how can i have info about the running disk image files (used by the guest), in this cas on the LVM-iscsi storage, apart from the pve (debian command line perhaps?)?

# pvesm list -a

(see 'man pvesm')

it reports 8.00 GB size, and used "-"...
can't get info on really used space on that machine, apart from the guest OS itself (which i think will report just its free space, not (non zero) data)

Only the guest can tell you how much it uses. On the host, the whole 8GB is allocated (on the lvm volume group).