vzdump of KVM in snapshot issue

K

kumarullal

Renowned Member
Jun 17, 2009
184
0
81
LA, USA
vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

I created a windows 2000 server vm on proxmox. WHen I did a vzdump using snapshot option, I lost connection to the server. I could not even ping the server while the vzdump job was going on.
Is this normal? What should I do to ensure that the connectivity is not lost. Should I try with suspend option?
It took 30 minutes just for a 18GB disk. If I had a 500 GB disk, it would take forever. This may be a big hurdle in taking backups for several servers on a daily basis.
Is there is solution to this?
 
Last edited:
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

what is the output of (please run when there is no load on the server):

# pveperf
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

I Installed the new version of vzdump, still no joy.
It still takes 30 minutes for 18GB. I included the --size 4096, --ionice 2, still it takes 30 minutes.
Even when I am backing up on local storage. I tried both locations / (root) and also on /var/lib/vz/snap (Created a directory called snap.) with same result of 30 minutes.
How can I drastically increase the speed of vzdump.
I would like to use this in production with a windows 2000 server VM of 350GB capacity. I cant afford to wait for hours while vzdump finishes the job, because the users wont be able to use the server for that much time.
Is there any way around to speed up the process?
I love Proxmox so much that I am trying to migrate all my VM (In production) from vmware esx 3.5 to proxmox.
The only hitch is that the vzdump is taking a very long time to backup KVM images.
Once I am successful in shortening the vzdump time, I can start migrating slowly.
Please help.
Thanks in advance.
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

In order to increase the speed of vzdump backups and maintain uninterrupted service at the same time, you are advised to change the default linux I/O scheduler from "cfq" (which is probably buggy) to "deadline" - which seems to be the best choice for single disk setups - or "noop", which performs best with SSD and HW RAID setups with more than 2-3 disks.

When you have done that, you can even increase the bandwidth limit from 10 to 20-30 MB/s and your server will still be responsive to your users.

Here is a thread with more information:
http://forum.proxmox.com/threads/434-I-O-scheduler

To change the scheduler permanently, you have to edit your /boot/grub/menu.lst file, detailed at the bottom of this post:
http://tombuntu.com/index.php/2008/09/04/four-tweaks-for-using-linux-with-solid-state-drives/
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

You can also use a second (or remote NFS) disk to store backups.
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

And please can you post the output of 'pveperf'?
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

HI gkovacs,
even 30 MB/s would take a long time (194 Minutes) to backup a 350GB disk. That is too long. Image having 3 or 4 servers with large disks would take forever to backup.

dietmar,
I have the test storage (Openfiler) setup on vmware. Even when I backup the VM having disk on the local storage it takes the same amount of time. vzdump backups are stored on local disk.

Here is the output of pvepref

pveperf
CPU BOGOMIPS: 15960.33
REGEX/SECOND: 671701
HD SIZE: 36.67 GB (/dev/mapper/pve-root)
BUFFERED READS: 69.11 MB/sec
AVERAGE SEEK TIME: 11.35 ms
FSYNCS/SECOND: 1312.38
DNS EXT: 37.59 ms
DNS INT: 0.80 ms (abcdxx.com)
 
Last edited:
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

kumarullal said:
even 30 MB/s would take a long time (194 Minutes) to backup a 350GB disk. That is too long.
Click to expand...

But your harddisk seems to be quite slow - what kind of disk subsystem is that? The data you posted looks like a single disk installation, or do you have RAID?
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

If I had correctly understood, the problem is more that the snapshot mode does not work than the backup duration.
If the snapshot mode of vzdump was chosen and functionnal, the VM should be unresponsive for 2 or 3 seconds, not more.
After that, the backup can take an hour, it is not a problem for the users.

Can you show us the output of your vzdump ?
Snapshot mode can fail if your disks are not on LVM, or there is not enought space in the VG to create the snapshot LV.
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

dietmar said:
But your harddisk seems to be quite slow - what kind of disk subsystem is that? The data you posted looks like a single disk installation, or do you have RAID?
Click to expand...
I had created a test VM of openfiler storage on vmware
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

Here is the output of the vzdump
Aug 26 13:49:36 INFO: Starting Backup of VM 102 (qemu)
Aug 26 13:49:36 INFO: stopped
Aug 26 13:49:36 INFO: status = stopped
Aug 26 13:49:37 INFO: backup mode: stop
Aug 26 13:49:37 INFO: ionice priority: 7
Aug 26 13:49:37 INFO: creating archive '/var/lib/vz/snap/server5/vzdump-qemu-102-2010_08_26-13_49_36.tgz'
Aug 26 13:49:37 INFO: adding '/var/lib/vz/snap/server5/vzdump-qemu-102-2010_08_26-13_49_36.tmp/qemu-server.conf' to archive ('qemu-server.conf')
Aug 26 13:49:37 INFO: adding '/mnt/pve/openfiler/images/102/vm-102-disk-1.raw' to archive ('vm-disk-ide0.raw')
Aug 26 14:18:52 INFO: Total bytes written: 808956416 (0.44 MiB/s)
Aug 26 14:18:52 INFO: archive file size: 409MB
Aug 26 14:18:53 INFO: Finished Backup of VM 102 (00:29:17)
Click to expand...
 
Since the vzdump may take a long time for a VM having large disk size, I was thinking if there was an alternative for vzdump.
Here is my question.
If I create snapshot schedules on the datastore itself, with multiple restore points, would it be viable to select any restore points from the datastore itself, in the event of a disaster?
If this works, then we don't have to deal with longer time taken for vzdump to complete the jobs.
Any thoughts?
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

kumarullal said:
I had created a test VM of openfiler storage on vmware
Click to expand...

to summarize, you are running a virtualized openfiler storage on vmware and then you benchmark? I suggest you should re-think about your test setup.
 
If you restore from a snapshot taken live on your storage système, it will be the samle as a LVM snapshot with proxmox.
The result can work, it will be the same as a a power failure.
On databases, or other applications holding data in RAM, it is a bit dangerous.

Also, snapshots on a storage array is not really a backup plan, if you loose your array (fire), you loose everything.

The most fast and secure you can do is :
1) stop the VM
2) snapshot the volume where the disks are (Storage array)
3) start the VM
4) mount the snapshot somewhere and back it up (tape or another storage system, in another building if you can)

The VM will not be accessible beetween steps 1 and 3, the longer part will be step 4.

I think the most difficult part will be to script this, and specifically to synchronise step 2. If you can make a snapshot on your storage from a proxmox server (remote command), fine.
 
Re: vzdump of KVM in snapshot issue-took 30 minutes for 18 GB disk

tom said:
to summarize, you are running a virtualized openfiler storage on vmware and then you benchmark? I suggest you should re-think about your test setup.
Click to expand...

Yes, I realize the limitations on a VMWARE based SAN storage, so I have installed openfiler on a good hardware with 4 GB ram and 1 TB HDD.
I also tried using vzdump on the local hard disk space. I created vzdump.conf and just added one line in it, bwlimit=80000
and the vzdump finfished the job for 30GB disk in just 289 seconds. Very fast, however, the speed was 3 mb/sec insted of 80 mb/sec
But the speed surely was very fast.
I will test on the new Hardware based SAN storage (Openfiler)
Thanks
 
gillesMo said:
If you restore from a snapshot taken live on your storage système, it will be the samle as a LVM snapshot with proxmox.
The result can work, it will be the same as a a power failure.
On databases, or other applications holding data in RAM, it is a bit dangerous.

Also, snapshots on a storage array is not really a backup plan, if you loose your array (fire), you loose everything.

The most fast and secure you can do is :
1) stop the VM
2) snapshot the volume where the disks are (Storage array)
3) start the VM
4) mount the snapshot somewhere and back it up (tape or another storage system, in another building if you can)

The VM will not be accessible beetween steps 1 and 3, the longer part will be step 4.

I think the most difficult part will be to script this, and specifically to synchronise step 2. If you can make a snapshot on your storage from a proxmox server (remote command), fine.
Click to expand...

I'm totally agree and ask for the same some time before:
http://forum.proxmox.com/threads/2349-Backup-KVM-with-stop-shapshot
Unfortunately developers don't think it would be used by many people.
dietmar said:
interesting idea - thought I don't know if that would be used by many people.
Click to expand...


Shall we arrange voting?
 
You must log in or register to reply here.
Top Bottom