hi,
not sure whats happening but my backups have been running fine for months and now they decide to hang until I SSH in and kill the backup process then it carries on.
The schedule runs again and the same thing happens. Storage has plenty of free space, locally connected FreeNAS server using NFS on gig connection, the proxmox server has a dedicated NIC for the backups. Server is a Dell Poweredge R710, Hardware RAID. Versions and log outputs are below -
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-12-pve
proxmox-ve-2.6.32: 2.1-68
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-12-pve: 2.6.32-68
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-16
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
Detailed backup logs:
vzdump 101 104 106 107 --quiet 1 --mailto me@me.com --mode snapshot --compress lzo --storage FNAS1 --node pm3
101: Nov 30 01:30:02 INFO: Starting Backup of VM 101 (qemu)
101: Nov 30 01:30:02 INFO: status = running
101: Nov 30 01:30:03 INFO: backup mode: snapshot
101: Nov 30 01:30:03 INFO: ionice priority: 7
101: Nov 30 01:30:03 INFO: Logical volume "vzsnap-pm3-0" created
101: Nov 30 01:30:03 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-101-2012_11_30-01_30_01.tar.lzo'
101: Nov 30 01:30:03 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-101-2012_11_30-01_30_01.tmp/qemu-server.conf' to archive ('qemu-server.conf')
101: Nov 30 01:30:03 INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk-1.raw' to archive ('vm-disk-ide0.raw')
101: Nov 30 01:43:28 INFO: Total bytes written: 161061276160 (190.81 MiB/s)
101: Nov 30 01:54:50 INFO: archive file size: 6.30GB
101: Nov 30 01:54:50 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-101-2012_11_28-01_30_01.tar.lzo'
101: Nov 30 01:54:54 INFO: Finished Backup of VM 101 (00:24:53)
104: Nov 30 01:54:54 INFO: Starting Backup of VM 104 (qemu)
104: Nov 30 01:54:54 INFO: status = running
104: Nov 30 01:54:54 INFO: backup mode: snapshot
104: Nov 30 01:54:54 INFO: ionice priority: 7
104: Nov 30 01:54:54 INFO: Logical volume "vzsnap-pm3-0" created
104: Nov 30 01:54:54 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-104-2012_11_30-01_54_54.tar.lzo'
104: Nov 30 01:54:54 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-104-2012_11_30-01_54_54.tmp/qemu-server.conf' to archive ('qemu-server.conf')
104: Nov 30 01:54:54 INFO: adding '/mnt/vzsnap0/images/104/vm-104-disk-1.raw' to archive ('vm-disk-ide0.raw')
104: Nov 30 17:29:29 INFO: received signal - terminate process
104: Nov 30 17:29:29 INFO: archive file size: 9.13GB
104: Nov 30 17:29:29 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-104-2012_11_28-01_52_06.tar.lzo'
104: Nov 30 17:29:32 INFO: Finished Backup of VM 104 (15:34:38)
106: Nov 30 17:29:32 INFO: Starting Backup of VM 106 (qemu)
106: Nov 30 17:29:32 INFO: status = running
106: Nov 30 17:29:33 INFO: backup mode: snapshot
106: Nov 30 17:29:33 INFO: ionice priority: 7
106: Nov 30 17:29:33 INFO: Logical volume "vzsnap-pm3-0" created
106: Nov 30 17:29:33 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-106-2012_11_30-17_29_32.tar.lzo'
106: Nov 30 17:29:33 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-106-2012_11_30-17_29_32.tmp/qemu-server.conf' to archive ('qemu-server.conf')
106: Nov 30 17:29:33 INFO: adding '/mnt/vzsnap0/images/106/vm-106-disk-1.raw' to archive ('vm-disk-ide0.raw')
106: Nov 30 17:31:50 INFO: Total bytes written: 32212257280 (224.23 MiB/s)
106: Nov 30 17:31:58 INFO: archive file size: 596MB
106: Nov 30 17:31:58 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-106-2012_11_28-10_45_42.tar.lzo'
106: Nov 30 17:32:02 INFO: Finished Backup of VM 106 (00:02:30)
107: Nov 30 17:32:02 INFO: Starting Backup of VM 107 (qemu)
107: Nov 30 17:32:02 INFO: status = running
107: Nov 30 17:32:03 INFO: backup mode: snapshot
107: Nov 30 17:32:03 INFO: ionice priority: 7
107: Nov 30 17:32:03 INFO: Logical volume "vzsnap-pm3-0" created
107: Nov 30 17:32:03 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-107-2012_11_30-17_32_02.tar.lzo'
107: Nov 30 17:32:03 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-107-2012_11_30-17_32_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
107: Nov 30 17:32:03 INFO: adding '/mnt/vzsnap0/images/107/vm-107-disk-1.raw' to archive ('vm-disk-ide0.raw')
107: Nov 30 17:40:33 INFO: Total bytes written: 53687093760 (100.39 MiB/s)
107: Nov 30 17:41:21 INFO: archive file size: 10.03GB
107: Nov 30 17:41:21 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-107-2012_11_28-10_48_14.tar.lzo'
107: Nov 30 17:41:25 INFO: Finished Backup of VM 107 (00:09:23)
as you can see its hung in this instance on 104.
Any ideas anyone?
thanks
matt
not sure whats happening but my backups have been running fine for months and now they decide to hang until I SSH in and kill the backup process then it carries on.
The schedule runs again and the same thing happens. Storage has plenty of free space, locally connected FreeNAS server using NFS on gig connection, the proxmox server has a dedicated NIC for the backups. Server is a Dell Poweredge R710, Hardware RAID. Versions and log outputs are below -pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-12-pve
proxmox-ve-2.6.32: 2.1-68
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-12-pve: 2.6.32-68
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-16
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
Detailed backup logs:
vzdump 101 104 106 107 --quiet 1 --mailto me@me.com --mode snapshot --compress lzo --storage FNAS1 --node pm3
101: Nov 30 01:30:02 INFO: Starting Backup of VM 101 (qemu)
101: Nov 30 01:30:02 INFO: status = running
101: Nov 30 01:30:03 INFO: backup mode: snapshot
101: Nov 30 01:30:03 INFO: ionice priority: 7
101: Nov 30 01:30:03 INFO: Logical volume "vzsnap-pm3-0" created
101: Nov 30 01:30:03 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-101-2012_11_30-01_30_01.tar.lzo'
101: Nov 30 01:30:03 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-101-2012_11_30-01_30_01.tmp/qemu-server.conf' to archive ('qemu-server.conf')
101: Nov 30 01:30:03 INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk-1.raw' to archive ('vm-disk-ide0.raw')
101: Nov 30 01:43:28 INFO: Total bytes written: 161061276160 (190.81 MiB/s)
101: Nov 30 01:54:50 INFO: archive file size: 6.30GB
101: Nov 30 01:54:50 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-101-2012_11_28-01_30_01.tar.lzo'
101: Nov 30 01:54:54 INFO: Finished Backup of VM 101 (00:24:53)
104: Nov 30 01:54:54 INFO: Starting Backup of VM 104 (qemu)
104: Nov 30 01:54:54 INFO: status = running
104: Nov 30 01:54:54 INFO: backup mode: snapshot
104: Nov 30 01:54:54 INFO: ionice priority: 7
104: Nov 30 01:54:54 INFO: Logical volume "vzsnap-pm3-0" created
104: Nov 30 01:54:54 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-104-2012_11_30-01_54_54.tar.lzo'
104: Nov 30 01:54:54 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-104-2012_11_30-01_54_54.tmp/qemu-server.conf' to archive ('qemu-server.conf')
104: Nov 30 01:54:54 INFO: adding '/mnt/vzsnap0/images/104/vm-104-disk-1.raw' to archive ('vm-disk-ide0.raw')
104: Nov 30 17:29:29 INFO: received signal - terminate process
104: Nov 30 17:29:29 INFO: archive file size: 9.13GB
104: Nov 30 17:29:29 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-104-2012_11_28-01_52_06.tar.lzo'
104: Nov 30 17:29:32 INFO: Finished Backup of VM 104 (15:34:38)
106: Nov 30 17:29:32 INFO: Starting Backup of VM 106 (qemu)
106: Nov 30 17:29:32 INFO: status = running
106: Nov 30 17:29:33 INFO: backup mode: snapshot
106: Nov 30 17:29:33 INFO: ionice priority: 7
106: Nov 30 17:29:33 INFO: Logical volume "vzsnap-pm3-0" created
106: Nov 30 17:29:33 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-106-2012_11_30-17_29_32.tar.lzo'
106: Nov 30 17:29:33 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-106-2012_11_30-17_29_32.tmp/qemu-server.conf' to archive ('qemu-server.conf')
106: Nov 30 17:29:33 INFO: adding '/mnt/vzsnap0/images/106/vm-106-disk-1.raw' to archive ('vm-disk-ide0.raw')
106: Nov 30 17:31:50 INFO: Total bytes written: 32212257280 (224.23 MiB/s)
106: Nov 30 17:31:58 INFO: archive file size: 596MB
106: Nov 30 17:31:58 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-106-2012_11_28-10_45_42.tar.lzo'
106: Nov 30 17:32:02 INFO: Finished Backup of VM 106 (00:02:30)
107: Nov 30 17:32:02 INFO: Starting Backup of VM 107 (qemu)
107: Nov 30 17:32:02 INFO: status = running
107: Nov 30 17:32:03 INFO: backup mode: snapshot
107: Nov 30 17:32:03 INFO: ionice priority: 7
107: Nov 30 17:32:03 INFO: Logical volume "vzsnap-pm3-0" created
107: Nov 30 17:32:03 INFO: creating archive '/mnt/pve/FNAS1/dump/vzdump-qemu-107-2012_11_30-17_32_02.tar.lzo'
107: Nov 30 17:32:03 INFO: adding '/mnt/pve/FNAS1/dump/vzdump-qemu-107-2012_11_30-17_32_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
107: Nov 30 17:32:03 INFO: adding '/mnt/vzsnap0/images/107/vm-107-disk-1.raw' to archive ('vm-disk-ide0.raw')
107: Nov 30 17:40:33 INFO: Total bytes written: 53687093760 (100.39 MiB/s)
107: Nov 30 17:41:21 INFO: archive file size: 10.03GB
107: Nov 30 17:41:21 INFO: delete old backup '/mnt/pve/FNAS1/dump/vzdump-qemu-107-2012_11_28-10_48_14.tar.lzo'
107: Nov 30 17:41:25 INFO: Finished Backup of VM 107 (00:09:23)
as you can see its hung in this instance on 104.
Any ideas anyone?
thanks
matt
