Backup getting hung

[ejmerkel]

I am running PVE 2.1-14/f32f3f46. This is the second week in a row when an attempted backup on a VM has hung. The backup starts but seems to stop at about 100G. The backup is going to a local partition called /backup and there is plenty of free space on the device (80% free). Here are the files in /backup and they have not changed in size since 2:07am which was over 7 hours ago.

-rw-r--r-- 1 root root 106649251786 Oct 28 02:07 vzdump-qemu-101-2012_10_28-01_00_02.tar.dat
drwxr-xr-x 2 root root 4096 Oct 28 01:00 vzdump-qemu-101-2012_10_28-01_00_02.tmp

#df /backup
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/lvm_local_storage-backup
1032123136 200609320 821028056 20% /backup



Here is the output of the Backup task.

INFO: starting new backup job: vzdump 101 100 108 --quiet 1 --mailto xxx@yyy.com --mode snapshot --compress lzo --storage backups
INFO: skip external VMs: 100
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-dc03r09ds30-0" created
INFO: creating archive '/backup/dump/vzdump-qemu-101-2012_10_28-01_00_02.tar.lzo'
INFO: adding '/backup/dump/vzdump-qemu-101-2012_10_28-01_00_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/dev/lvm_local_storage/vzsnap-dc03r09ds30-0' to archive ('vm-disk-virtio0.raw')

If I do an "lvs" command this is what I see.

LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
backup lvm_local_storage -wi-ao-- 1000.00g
vm-101-disk-1 lvm_local_storage owi-aos- 250.00g
vm-107-disk-1 lvm_local_storage -wi-ao-- 10.00g
vm-108-disk-1 lvm_local_storage -wi-ao-- 150.00g
vzsnap-dc03r09ds30-0 lvm_local_storage Swi-Ios- 1.00g vm-101-disk-1 100.00
data pve -wi-ao-- 49.50g
root pve -wi-ao-- 25.00g
swap pve -wi-ao-- 12.50g


The LV vzsnap-dc03r09ds30-0 is showing 100% which looks suspicious.

Is there anything else I can look at to see why this backup is hanging?


Thanks,
Eric

 

[dietmar]

The LV vzsnap-dc03r09ds30-0 is showing 100% which looks suspicious.

Yes, you run out of snapshot space. Please increase that (size parameter in /etc/vzdump.conf)

 

[ejmerkel]

I changed the /etc/vzdump.conf as follows:

size: 4096

Also had some other functions running on this server at the same time the dump was being taken which generated a fair amount of changes in the snapshot so I rescheduled those for another time. Between the two this allowed the backup to complete.

Thanks for your help!

Eric

 

[tom]

also upgrade to latest packages, backup hang because of LVM out of space problems is now correctly detected

 

[ejmerkel]

I am fairly new to proxmox. What is the process to upgrade to the latest packages?

Eric

 

[tom]

See http://pve.proxmox.com/wiki/Downloads#Update_a_running_Proxmox_Virtual_Environment_2.x_to_latest_2.2

 

[ejmerkel]

Two more quick questions before I upgrade. I have three servers in a cluster.

Do I have to upgrade the all at the same time?

Do I have to upgrade them in any particular order (master/slave/etc)?

Eric

 

[dietmar]

Do I have to upgrade the all at the same time?

yes (but not 'exactly' the same time)

Do I have to upgrade them in any particular order (master/slave/etc)?

No (there is no master or slave role)