The problem is essentially vzdump is filling up every bit of disk space on my external USB drive while trying to backup to KVM VM's. The USB drive is 500GB (/mnt/ext), and this process worked fine prior to my last dist-upgrade to pve 1.4. Here's all the good info:
volt:~# pveversion -v
pve-manager: 1.4-10 (pve-manager/1.4/4403)
qemu-server: 1.1-8
pve-kernel: 2.6.24-18
pve-qemu-kvm: 0.11.0-2
pve-firmware: 1
vncterm: 0.9-2
vzctl: 3.0.23-1pve3
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
volt:~# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
101 SBS running 4096 300.00 12586
102 SQL running 2048 80.00 4210
volt:/var/lib/vz/images# ls -lnh
total 8.0K
drwxr-xr-x 2 0 0 4.0K 2009-01-05 07:23 101
drwxr-xr-x 2 0 0 4.0K 2009-01-06 06:02 102
volt:/var/lib/vz/images# df -H /var/lib/vz
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-data 870G 370G 501G 43% /var/lib/vz
volt:/var/lib/vz/images/101# ls -lnh
total 267G
-rw-r--r-- 1 0 0 267G 2010-01-11 15:31 vm-101-disk.qcow2
volt:/var/lib/vz/images/102# ls -lnh
total 78G
-rw-r--r-- 1 0 0 78G 2010-01-11 15:31 vm-102-disk.qcow2
So that's it on this box. 2 KVM VM's. Nothing else.
volt:~#cat /etc/cron.d/vzdump
# Atomatically generated file - do not edit
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
00 20 * * 1,2,3,4,5 root vzdump --quiet --snapshot --dumpdir /mnt/ext --tmpdir /mnt/ext --bwlimit 7500 --maxfiles 1 --mailto xyz@domain.com 101 102
Finally, the vzdump log:
Detailed backup logs:
vzdump --quiet --snapshot --dumpdir /mnt/ext --tmpdir /mnt/ext --bwlimit 7500 --maxfiles 1 --mailto xyz@domain.com 101 102 101: Jan 08 20:00:02 INFO: Starting Backup of VM 101 (qemu)101: Jan 08 20:00:02 INFO: running101: Jan 08 20:00:02 INFO: status = running101: Jan 08 20:00:03 INFO: backup mode: snapshot101: Jan 08 20:00:03 INFO: bandwidth limit: 7500 KB/s101: Jan 08 20:00:03 INFO: Logical volume "vzsnap-volt-0" created101: Jan 08 20:00:03 INFO: creating archive '/mnt/ext/vzdump-qemu-101-2010_01_08-20_00_02.tar'101: Jan 08 20:00:03 INFO: adding '/mnt/ext/vzdumptmp658/qemu-server.conf' to archive ('qemu-server.conf')101: Jan 08 20:00:03 INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk.qcow2' to archive ('vm-disk-ide0.qcow2')101: Jan 09 13:47:44 INFO: 491568132096 B 457.8 GB 64060.6 s (17:47 h) 7673492 B/s 7.32 MB/s101: Jan 09 13:47:44 INFO: write: No space left on device101: Jan 09 13:47:44 INFO: received signal - terminate process101: Jan 09 13:47:45 INFO: /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error101: Jan 09 13:47:55 INFO: Logical volume "vzsnap-volt-0" successfully removed101: Jan 09 14:00:09 ERROR: Backup of VM 101 failed - command '/usr/lib/qemu-server/vmtar '/mnt/ext/vzdumptmp658/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/101/vm-101-disk.qcow2' 'vm-disk-ide0.qcow2' |cstream -t 7680000 >/mnt/ext/vzdump-qemu-101-2010_01_08-20_00_02.dat' failed with exit code 2 102: Jan 09 14:00:09 INFO: Starting Backup of VM 102 (qemu)102: Jan 09 14:00:10 INFO: running102: Jan 09 14:00:10 INFO: status = running102: Jan 09 14:00:25 INFO: backup mode: snapshot102: Jan 09 14:00:25 INFO: bandwidth limit: 7500 KB/s102: Jan 09 14:00:25 INFO: Logical volume "vzsnap-volt-0" created102: Jan 09 14:00:26 INFO: creating archive '/mnt/ext/vzdump-qemu-102-2010_01_09-14_00_09.tar'102: Jan 09 14:00:26 INFO: adding '/mnt/ext/vzdumptmp658/qemu-server.conf' to archive ('qemu-server.conf')102: Jan 09 14:00:26 INFO: adding '/mnt/vzsnap0/images/102/vm-102-disk.qcow2' to archive ('vm-disk-ide0.qcow2')102: Jan 10 07:48:06 INFO: 491568128000 B 457.8 GB 64060.6 s (17:47 h) 7673488 B/s 7.32 MB/s102: Jan 10 07:48:06 INFO: write: No space left on device102: Jan 10 07:48:06 INFO: received signal - terminate process102: Jan 10 07:48:07 INFO: /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error102: Jan 10 07:48:16 INFO: Logical volume "vzsnap-volt-0" successfully removed102: Jan 10 08:00:01 ERROR: Backup of VM 102 failed - command '/usr/lib/qemu-server/vmtar '/mnt/ext/vzdumptmp658/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/102/vm-102-disk.qcow2' 'vm-disk-ide0.qcow2' |cstream -t 7680000 >/mnt/ext/vzdump-qemu-102-2010_01_09-14_00_09.dat' failed with exit code 2
Any help would be much appreciated. Some of the things that vzdump is doing - well, I don't understand them. Why it's writing 457.8 GB to backup a 80GB VM is very confusing to me.
volt:~# pveversion -v
pve-manager: 1.4-10 (pve-manager/1.4/4403)
qemu-server: 1.1-8
pve-kernel: 2.6.24-18
pve-qemu-kvm: 0.11.0-2
pve-firmware: 1
vncterm: 0.9-2
vzctl: 3.0.23-1pve3
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
volt:~# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
101 SBS running 4096 300.00 12586
102 SQL running 2048 80.00 4210
volt:/var/lib/vz/images# ls -lnh
total 8.0K
drwxr-xr-x 2 0 0 4.0K 2009-01-05 07:23 101
drwxr-xr-x 2 0 0 4.0K 2009-01-06 06:02 102
volt:/var/lib/vz/images# df -H /var/lib/vz
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-data 870G 370G 501G 43% /var/lib/vz
volt:/var/lib/vz/images/101# ls -lnh
total 267G
-rw-r--r-- 1 0 0 267G 2010-01-11 15:31 vm-101-disk.qcow2
volt:/var/lib/vz/images/102# ls -lnh
total 78G
-rw-r--r-- 1 0 0 78G 2010-01-11 15:31 vm-102-disk.qcow2
So that's it on this box. 2 KVM VM's. Nothing else.
volt:~#cat /etc/cron.d/vzdump
# Atomatically generated file - do not edit
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
00 20 * * 1,2,3,4,5 root vzdump --quiet --snapshot --dumpdir /mnt/ext --tmpdir /mnt/ext --bwlimit 7500 --maxfiles 1 --mailto xyz@domain.com 101 102
Finally, the vzdump log:
Detailed backup logs:
vzdump --quiet --snapshot --dumpdir /mnt/ext --tmpdir /mnt/ext --bwlimit 7500 --maxfiles 1 --mailto xyz@domain.com 101 102 101: Jan 08 20:00:02 INFO: Starting Backup of VM 101 (qemu)101: Jan 08 20:00:02 INFO: running101: Jan 08 20:00:02 INFO: status = running101: Jan 08 20:00:03 INFO: backup mode: snapshot101: Jan 08 20:00:03 INFO: bandwidth limit: 7500 KB/s101: Jan 08 20:00:03 INFO: Logical volume "vzsnap-volt-0" created101: Jan 08 20:00:03 INFO: creating archive '/mnt/ext/vzdump-qemu-101-2010_01_08-20_00_02.tar'101: Jan 08 20:00:03 INFO: adding '/mnt/ext/vzdumptmp658/qemu-server.conf' to archive ('qemu-server.conf')101: Jan 08 20:00:03 INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk.qcow2' to archive ('vm-disk-ide0.qcow2')101: Jan 09 13:47:44 INFO: 491568132096 B 457.8 GB 64060.6 s (17:47 h) 7673492 B/s 7.32 MB/s101: Jan 09 13:47:44 INFO: write: No space left on device101: Jan 09 13:47:44 INFO: received signal - terminate process101: Jan 09 13:47:45 INFO: /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error101: Jan 09 13:47:55 INFO: Logical volume "vzsnap-volt-0" successfully removed101: Jan 09 14:00:09 ERROR: Backup of VM 101 failed - command '/usr/lib/qemu-server/vmtar '/mnt/ext/vzdumptmp658/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/101/vm-101-disk.qcow2' 'vm-disk-ide0.qcow2' |cstream -t 7680000 >/mnt/ext/vzdump-qemu-101-2010_01_08-20_00_02.dat' failed with exit code 2 102: Jan 09 14:00:09 INFO: Starting Backup of VM 102 (qemu)102: Jan 09 14:00:10 INFO: running102: Jan 09 14:00:10 INFO: status = running102: Jan 09 14:00:25 INFO: backup mode: snapshot102: Jan 09 14:00:25 INFO: bandwidth limit: 7500 KB/s102: Jan 09 14:00:25 INFO: Logical volume "vzsnap-volt-0" created102: Jan 09 14:00:26 INFO: creating archive '/mnt/ext/vzdump-qemu-102-2010_01_09-14_00_09.tar'102: Jan 09 14:00:26 INFO: adding '/mnt/ext/vzdumptmp658/qemu-server.conf' to archive ('qemu-server.conf')102: Jan 09 14:00:26 INFO: adding '/mnt/vzsnap0/images/102/vm-102-disk.qcow2' to archive ('vm-disk-ide0.qcow2')102: Jan 10 07:48:06 INFO: 491568128000 B 457.8 GB 64060.6 s (17:47 h) 7673488 B/s 7.32 MB/s102: Jan 10 07:48:06 INFO: write: No space left on device102: Jan 10 07:48:06 INFO: received signal - terminate process102: Jan 10 07:48:07 INFO: /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error102: Jan 10 07:48:16 INFO: Logical volume "vzsnap-volt-0" successfully removed102: Jan 10 08:00:01 ERROR: Backup of VM 102 failed - command '/usr/lib/qemu-server/vmtar '/mnt/ext/vzdumptmp658/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/102/vm-102-disk.qcow2' 'vm-disk-ide0.qcow2' |cstream -t 7680000 >/mnt/ext/vzdump-qemu-102-2010_01_09-14_00_09.dat' failed with exit code 2
Any help would be much appreciated. Some of the things that vzdump is doing - well, I don't understand them. Why it's writing 457.8 GB to backup a 80GB VM is very confusing to me.