Issue with VZDUMP snapshots

[cre8nchaos]

I had a snapshot job that hung, after a reboot of the Proxmox host I have not been able to get any snapshot jobs to complete. Here is the latest log file.

Subject: vzdump backup status (vh02) : backup failed



VMID STATUS TIME SIZE FILENAME
321 err 00:19:12 0.00MB -

Detailed backup logs:

vzdump --quiet --node 2 --snapshot --compress --storage VMSnapshots --mailto --all

321: May 29 16:00:02 INFO: Starting Backup of VM 321 (qemu)
321: May 29 16:00:02 INFO: running
321: May 29 16:00:02 INFO: status = running
321: May 29 16:00:03 INFO: backup mode: snapshot
321: May 29 16:00:03 INFO: ionice priority: 7
321: May 29 16:00:03 INFO: Logical volume "vzsnap-accvh02-0" created
321: May 29 16:00:03 INFO: creating archive '/mnt/samba2/vzdump-qemu-321-2011_05_29-16_00_02.tgz'
321: May 29 16:00:03 INFO: adding '/mnt/samba2/vzdump-qemu-321-2011_05_29-16_00_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
321: May 29 16:00:03 INFO: adding '/dev/VHD02/vzsnap-accvh02-0' to archive ('vm-disk-ide0.raw')
321: May 29 16:19:10 INFO: gzip: stdout: No space left on device
321: May 29 16:19:10 INFO: received signal - terminate process
321: May 29 16:19:13 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
321: May 29 16:19:13 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
321: May 29 16:19:13 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
321: May 29 16:19:13 INFO: setting parameters failed - close failed - No space left on device
321: May 29 16:19:13 INFO: /dev/dm-9: read failed after 0 of 4096 at 0: Input/output error
321: May 29 16:19:13 INFO: /etc/lvm/archive/.lvm_accvh02_5789_1957313333: fclose failed: No space left on device
321: May 29 16:19:13 INFO: Volume group "VHD02" metadata archive failed.
321: May 29 16:19:13 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
321: May 29 16:19:13 ERROR: command 'lvremove -f '/dev/VHD02/vzsnap-accvh02-0'' failed with exit code 5
321: May 29 16:19:14 ERROR: Backup of VM 321 failed - command '/usr/lib/qemu-server/vmtar '/mnt/samba2/vzdump-qemu-321-2011_05_29-16_00_02.tmp/qemu-server.conf' 'qemu-server.conf' '/dev/VHD02/vzsnap-accvh02-0' 'vm-disk-ide0.raw'|gzip - >/mnt/samba2/vzdump-qemu-321-2011_05_29-16_00_02.dat' failed with exit code 1

The Samba share that the backup is going to is also being used by 5 other hosts that are able to take snapshots fine.

The HD Space Root shows 99.30% used.

Any help that can be provided would be greatly appreciated.

 

[tom]

I assume your samba mount does not work.? therefore the data got written on the root partition and this one is now full.

 

[cre8nchaos]

Where would the data have been written on the root partition? I also checked to make sure the samba drive is still mounted correctly and that it is accessible and it is. Anything else I should check?

 

[tom]

use a tool like ncdu to analyse your full root partition. (apt-get install ncdu)

 

[cre8nchaos]

Thank you.

I used ncdu to look at the percentage of disk space that each folder was using. It showed that /mnt/samba was using 80% of the local space. When I opened up the folder their was a snapshot .tgz file that was out there dated from the day vzdump was hung and I rebooted the host. Looking at the samba share from windows the file was not in the share itself. Deleting the tgz file dropped my disc usage back down to where it should be. I will set up another snapshot job this afternoon and test.

What is the best way to kill vzdump when it is hung up?

 

[gillesMo]

Kill the vztar child processes, vzdump can continue and terminate properly, removing LVM snapshots.