Hi everybody,
I have a simple PVE 2 set up with 4 openvz to backup nightly.
The backup works fine for a number of days, then a backup process apparently for no reason does not finish, here is the log of the backup
/backup is a mounted SAMBA share.
The last lines (starting with umount) are added as soon as I hit the stop-button of the stalled backup task.
When I log in to the host I get
Subsequent backups on the next nights fail with
Only a reboot of the host recovers from the error, after a reboot backups work again for about 3-4 days until the same error appears again.
Any idea what I could do to fix that?
Thanks!
I have a simple PVE 2 set up with 4 openvz to backup nightly.
The backup works fine for a number of days, then a backup process apparently for no reason does not finish, here is the log of the backup
Code:
INFO: starting new backup job: vzdump 101 104 106 108 --quiet 1 --mode snapshot --compress gzip --storage backup
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp281434 for temporary files
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-root100-0')
INFO: Logical volume "vzsnap-root100-0" created
INFO: creating archive '/backup/vms/dump/vzdump-openvz-101-2012_08_04-05_00_01.tar.gz'
INFO: Total bytes written: 623001600 (595MiB, 9.7MiB/s)
INFO: archive file size: 254MB
INFO: delete old backup '/backup/vms/dump/vzdump-openvz-101-2012_08_03-05_00_01.tar.gz'
INFO: Finished Backup of VM 101 (00:01:12)
INFO: Starting Backup of VM 104 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-root100-0" created
INFO: creating archive '/backup/vms/dump/vzdump-qemu-104-2012_08_04-05_01_13.tar.gz'
INFO: adding '/backup/vms/dump/vzdump-qemu-104-2012_08_04-05_01_13.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/104/vm-104-disk-1.qcow2' to archive ('vm-disk-ide0.qcow2')
INFO: Total bytes written: 21172455936 (14.54 MiB/s)
INFO: archive file size: 13.93GB
INFO: delete old backup '/backup/vms/dump/vzdump-qemu-104-2012_08_03-05_01_04.tar.gz'
INFO: Finished Backup of VM 104 (00:23:13)
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp281434 for temporary files
INFO: Starting Backup of VM 106 (openvz)
INFO: CTID 106 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-root100-0')
INFO: Logical volume "vzsnap-root100-0" created
INFO: creating archive '/backup/vms/dump/vzdump-openvz-106-2012_08_04-05_24_27.tar.gz'
INFO: Total bytes written: 25170360320 (24GiB, 9.8MiB/s)
INFO: umount: /mnt/vzsnap0: device is busy.
INFO: (In some cases useful info about processes that use
INFO: the device is found by lsof(8) or fuser(1))
ERROR: command 'umount /mnt/vzsnap0' failed: exit code 1/backup is a mounted SAMBA share.
The last lines (starting with umount) are added as soon as I hit the stop-button of the stalled backup task.
When I log in to the host I get
Code:
root@hostname:~# fuser /mnt/vzsnap0/
root@hostname:~# umount /mnt/vzsnap0
umount: /mnt/vzsnap0: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
root@hostname:~# lsof /mnt/vzsnap0
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 285007 root cwd DIR 253,3 4096 8381789 /mnt/vzsnap0/private/106
gzip 285011 root cwd DIR 253,3 4096 8381789 /mnt/vzsnap0/private/106
root@hostname:~# killall -9 gzip
root@hostname:~# lsof /mnt/vzsnap0
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 285007 root cwd DIR 253,3 4096 8381789 /mnt/vzsnap0/private/106
gzip 285011 root cwd DIR 253,3 4096 8381789 /mnt/vzsnap0/private/106
root@hostname:~# killall gzip
root@hostname:~# lsof /mnt/vzsnap0
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 285007 root cwd DIR 253,3 4096 8381789 /mnt/vzsnap0/private/106
gzip 285011 root cwd DIR 253,3 4096 8381789 /mnt/vzsnap0/private/106Subsequent backups on the next nights fail with
Code:
INFO: trying to get global lock - waiting...
ERROR: can't aquire lock '/var/run/vzdump.lock' - got timeoutOnly a reboot of the host recovers from the error, after a reboot backups work again for about 3-4 days until the same error appears again.
Any idea what I could do to fix that?
Thanks!
Code:
root@hostname:~# pveversion -vpve-manager: 2.1-13 (pve-manager/2.1/bdd3663d)
running kernel: 2.6.32-13-pve
proxmox-ve-2.6.32: 2.1-72
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-13-pve: 2.6.32-72
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-47
pve-firmware: 1.0-17
libpve-common-perl: 1.0-28
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-29
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-6
ksm-control-daemon: 1.1-1
Last edited:
