backup failed umount /mnt/vzsnap0' failed with exit code 16

[Majerus]

Yesterday we moved about 40GB of data to one of the vms. We did not have the server backup yesterday because of this. However today I noticed that I was emailed this error regarding the backup. My question is did a directory run out of space or what the is the issue. Any help would be appreciated

Thanks,
Majerus

virtual:~# pveversion -v
pve-manager: 1.8-18 (pve-manager/1.8/6070)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.8-33
pve-kernel-2.6.32-4-pve: 2.6.32-33
qemu-server: 1.1-30
pve-firmware: 1.0-11
libpve-storage-perl: 1.0-17
vncterm: 0.9-2
vzctl: 3.0.27-1pve1
vzdump: 1.2-13
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.14.1-1
ksm-control-daemon: 1.0-6

 

[tom]

Send the full backup log with the error.

Sent from my GT-P1000 using Tapatalk

 

[Majerus]

Detailed backup logs:


vzdump --quiet --node 1 --snapshot --compress --storage Backup --mailto *** --all101: Jul 28 12:47:02 INFO: Starting Backup of VM 101 (qemu)101: Jul 28 12:47:02 INFO: running101: Jul 28 12:47:02 INFO: status = running101: Jul 28 12:47:02 INFO: backup mode: snapshot101: Jul 28 12:47:02 INFO: ionice priority: 7101: Jul 28 12:47:03 INFO: Logical volume "vzsnap-virtual-0" created101: Jul 28 12:47:03 INFO: creating archive '/mnt/sdb1/vzdump-qemu-101-2011_07_28-12_47_02.tgz'101: Jul 28 12:47:03 INFO: adding '/mnt/sdb1/vzdump-qemu-101-2011_07_28-12_47_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')101: Jul 28 12:47:03 INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk-1.raw' to archive ('vm-disk-ide0.raw')101: Jul 28 12:52:43 INFO: Total bytes written: 3737582080 (10.48 MiB/s)101: Jul 28 12:52:43 INFO: archive file size: 1.45GB101: Jul 28 12:52:43 INFO: delete old backup '/mnt/sdb1/vzdump-qemu-101-2011_07_27-12_47_02.tgz'101: Jul 28 12:52:46 INFO: Logical volume "vzsnap-virtual-0" successfully removed101: Jul 28 12:52:46 INFO: Finished Backup of VM 101 (00:05:44)103: Jul 28 12:52:46 INFO: Starting Backup of VM 103 (qemu)103: Jul 28 12:52:46 INFO: running103: Jul 28 12:52:46 INFO: status = running103: Jul 28 12:52:46 INFO: backup mode: snapshot103: Jul 28 12:52:46 INFO: ionice priority: 7103: Jul 28 12:52:47 INFO: Logical volume "vzsnap-virtual-0" created103: Jul 28 12:52:47 INFO: creating archive '/mnt/sdb1/vzdump-qemu-103-2011_07_28-12_52_46.tgz'103: Jul 28 12:52:47 INFO: adding '/mnt/sdb1/vzdump-qemu-103-2011_07_28-12_52_46.tmp/qemu-server.conf' to archive ('qemu-server.conf')103: Jul 28 12:52:47 INFO: adding '/mnt/vzsnap0/images/103/vm-103-disk-1.qcow2' to archive ('vm-disk-ide0.qcow2')103: Jul 28 14:41:11 INFO: gzip: stdout: No space left on device103: Jul 28 14:41:11 INFO: received signal - terminate process103: Jul 28 14:41:12 INFO: setting parameters failed - close failed - No space left on device103: Jul 28 14:41:12 INFO: error writing /etc/mtab.tmp: No space left on device103: Jul 28 14:41:12 ERROR: command 'umount /mnt/vzsnap0' failed with exit code 16103: Jul 28 14:41:12 INFO: /etc/lvm/archive/.lvm_virtual_15134_563831323: write error failed: No space left on device103: Jul 28 14:41:12 INFO: Volume group "pve" metadata archive failed.103: Jul 28 14:41:12 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device103: Jul 28 14:41:12 ERROR: command 'lvremove -f '/dev/pve/vzsnap-virtual-0'' failed with exit code 5103: Jul 28 14:41:17 ERROR: Backup of VM 103 failed - command '/usr/lib/qemu-server/vmtar '/mnt/sdb1/vzdump-qemu-103-2011_07_28-12_52_46.tmp/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/103/vm-103-disk-1.qcow2' 'vm-disk-ide0.qcow2'|gzip - >/mnt/sdb1/vzdump-qemu-103-2011_07_28-12_52_46.dat' failed with exit code 1104: Jul 28 14:41:17 INFO: Starting Backup of VM 104 (qemu)104: Jul 28 14:41:17 INFO: stopped104: Jul 28 14:41:17 INFO: status = stopped104: Jul 28 14:41:17 INFO: backup mode: stop104: Jul 28 14:41:17 INFO: ionice priority: 7104: Jul 28 14:41:18 INFO: creating archive '/mnt/sdb1/vzdump-qemu-104-2011_07_28-14_41_17.tgz'104: Jul 28 14:41:18 INFO: adding '/mnt/sdb1/vzdump-qemu-104-2011_07_28-14_41_17.tmp/qemu-server.conf' to archive ('qemu-server.conf')104: Jul 28 14:41:18 INFO: adding '/var/lib/vz/images/104/vm-104-disk-1.raw' to archive ('vm-disk-ide0.raw')104: Jul 28 14:48:17 INFO: Total bytes written: 4513793024 (10.27 MiB/s)104: Jul 28 14:48:17 INFO: archive file size: 2.25GB104: Jul 28 14:48:17 INFO: delete old backup '/mnt/sdb1/vzdump-qemu-104-2011_07_27-14_04_57.tgz'104: Jul 28 14:48:22 INFO: Finished Backup of VM 104 (00:07:05)

 

[dietmar]

Looks like your root filesystem is full, check with

# df /

 

[Majerus]

df /
/dev/mapper/pve-root 99083868 47345084 46705620 51% /

Looks like the volume has ~50gb free



here is virtual:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 99083868 47345144 46705560 51% /
tmpfs 4071400 0 4071400 0% /lib/init/rw
udev 10240 584 9656 6% /dev
tmpfs 4071400 0 4071400 0% /dev/shm
/dev/mapper/pve-data 1811910692 229351720 1582558972 13% /var/lib/vz
/dev/sda1 516040 31640 458188 7% /boot

 

[Majerus]

Because the backups failed it locked the vm. I was unable to start the vm. logs were showing virtual:~# qm start 103VM is locked (backup)


so i issued a qm unlock 103 then the vm started. I am going to disable backups on the vm until I can figure out what is causing the backups to fail. Any other idea Tom.

 

[dietmar]

How large is the VM data - looks like backup is trying to add store more than 50GB, so you run out of space. Also check if the backup target is mounted (/mnt/sdb1). What kind of storage is that?

 

[Majerus]

The storage is a 2tb sata drive.

 

[dietmar]

The storage is a 2tb sata drive.

But the output from 'df' shows that that disk is not even mounted on /mnt/sdb1?

 

[Majerus]

I didn't notice that. I just mounted the drive and started running a backup. Appreciate the help Dietmar.

 

[dietmar]

I didn't notice that. I just mounted the drive and started running a backup.

So it works now?

 

[Majerus]

Yep it is working now.