Backup failed

[103]

After updating Proxmox VE to 2.2 and making a cluster with two nodes I have been getting backup errors. This is a log fragment:

102: Nov 02 10:32:20 INFO: gzip: stdout: Input/output error 102: Nov 02 10:32:20 INFO: Total bytes written: 231731200 (221MiB, ?/s) 102: Nov 02 10:32:20 INFO: tar: -: Wrote only 4096 of 10240 bytes 102: Nov 02 10:32:20 INFO: tar: Error is not recoverable: exiting now 102: Nov 02 10:32:20 ERROR: Backup of VM 102 failed - command '(cd /mnt/vzsnap0/private/102;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system --null -T -|gzip) >/mnt/pve/backups-alternate/dump/vzdump-openvz-102-2012_11_02-10_31_26.tar.dat' failed: exit code 1

The interesting part is that if I make a manual backup there's no error at all.

Best regards

 

[tom]

post your 'pveversion -v'

and the full backup log.

 

[103]

pveversion:

pve-manager: 2.2-26 (pve-manager/2.2/c1614c8c)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.2-80
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-16-pve: 2.6.32-80
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-1
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-28
qemu-server: 2.0-64
pve-firmware: 1.0-21
libpve-common-perl: 1.0-37
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-34
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.2-7
ksm-control-daemon: 1.1-1

Log here:


VMID STATUS TIME SIZE FILENAME 101 ok 00:01:24 137MB /mnt/pve/backups-alternate/dump/vzdump-openvz-101-2012_11_02-10_30_02.tar.gz 102 err 00:00:54 0.00MB - 103 err 00:06:08 0.00MB - 104 ok 00:01:34 188MB /mnt/pve/backups-alternate/dump/vzdump-openvz-104-2012_11_02-10_38_28.tar.gz Detailed backup logs: vzdump 101 102 103 104 105 106 107 --quiet 1 --mailto mailhere --mode snapshot --compress gzip --storage backups-alternate 101: Nov 02 10:30:02 INFO: Starting Backup of VM 101 (openvz) 101: Nov 02 10:30:02 INFO: CTID 101 exist mounted running 101: Nov 02 10:30:02 INFO: status = running 101: Nov 02 10:30:02 INFO: backup mode: snapshot 101: Nov 02 10:30:02 INFO: ionice priority: 7 101: Nov 02 10:30:02 INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0') 101: Nov 02 10:30:03 INFO: Logical volume "vzsnap-proxmox-0" created 101: Nov 02 10:30:03 INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-101-2012_11_02-10_30_02.tar.gz' 101: Nov 02 10:31:17 INFO: Total bytes written: 357376000 (341MiB, 5.3MiB/s) 101: Nov 02 10:31:25 INFO: archive file size: 137MB 101: Nov 02 10:31:25 INFO: delete old backup '/mnt/pve/backups-alternate/dump/vzdump-openvz-101-2012_11_01-10_43_37.tar.gz' 101: Nov 02 10:31:26 INFO: Finished Backup of VM 101 (00:01:24) 102: Nov 02 10:31:26 INFO: Starting Backup of VM 102 (openvz) 102: Nov 02 10:31:26 INFO: CTID 102 exist mounted running 102: Nov 02 10:31:26 INFO: status = running 102: Nov 02 10:31:26 INFO: backup mode: snapshot 102: Nov 02 10:31:26 INFO: ionice priority: 7 102: Nov 02 10:31:26 INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0') 102: Nov 02 10:31:27 INFO: Logical volume "vzsnap-proxmox-0" created 102: Nov 02 10:31:27 INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-102-2012_11_02-10_31_26.tar.gz' 102: Nov 02 10:32:20 INFO: gzip: stdout: Input/output error 102: Nov 02 10:32:20 INFO: Total bytes written: 231731200 (221MiB, ?/s) 102: Nov 02 10:32:20 INFO: tar: -: Wrote only 4096 of 10240 bytes 102: Nov 02 10:32:20 INFO: tar: Error is not recoverable: exiting now 102: Nov 02 10:32:20 ERROR: Backup of VM 102 failed - command '(cd /mnt/vzsnap0/private/102;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system --null -T -|gzip) >/mnt/pve/backups-alternate/dump/vzdump-openvz-102-2012_11_02-10_31_26.tar.dat' failed: exit code 1 103: Nov 02 10:32:21 INFO: Starting Backup of VM 103 (openvz) 103: Nov 02 10:32:21 INFO: CTID 103 exist mounted running 103: Nov 02 10:32:21 INFO: status = running 103: Nov 02 10:32:21 INFO: backup mode: snapshot 103: Nov 02 10:32:21 INFO: ionice priority: 7 103: Nov 02 10:32:21 INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0') 103: Nov 02 10:32:21 INFO: Logical volume "vzsnap-proxmox-0" created 103: Nov 02 10:32:22 INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-103-2012_11_02-10_32_20.tar.gz' 103: Nov 02 10:38:27 INFO: gzip: stdout: Input/output error 103: Nov 02 10:38:27 INFO: Total bytes written: 1701867520 (1.6GiB, ?/s) 103: Nov 02 10:38:27 INFO: tar: -: Wrote only 4096 of 10240 bytes 103: Nov 02 10:38:27 INFO: tar: Error is not recoverable: exiting now 103: Nov 02 10:38:28 ERROR: Backup of VM 103 failed - command '(cd /mnt/vzsnap0/private/103;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system --null -T -|gzip) >/mnt/pve/backups-alternate/dump/vzdump-openvz-103-2012_11_02-10_32_20.tar.dat' failed: exit code 1 104: Nov 02 10:38:28 INFO: Starting Backup of VM 104 (openvz) 104: Nov 02 10:38:28 INFO: CTID 104 exist mounted running 104: Nov 02 10:38:28 INFO: status = running 104: Nov 02 10:38:28 INFO: backup mode: snapshot 104: Nov 02 10:38:28 INFO: ionice priority: 7 104: Nov 02 10:38:28 INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0') 104: Nov 02 10:38:28 INFO: Logical volume "vzsnap-proxmox-0" created 104: Nov 02 10:38:29 INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-104-2012_11_02-10_38_28.tar.gz' 104: Nov 02 10:39:52 INFO: Total bytes written: 474081280 (453MiB, 5.9MiB/s) 104: Nov 02 10:40:01 INFO: archive file size: 188MB 104: Nov 02 10:40:01 INFO: delete old backup '/mnt/pve/backups-alternate/dump/vzdump-openvz-104-2012_11_01-12_35_06.tar.gz' 104: Nov 02 10:40:02 INFO: Finished Backup of VM 104 (00:01:34)

 

[tom]

you upgraded your kernel but you did not reboot.?

> running kernel: 2.6.32-11-pve

reboot to activate latest installed kernel, 2.6.32-16 - and try again.

 

[103]

You're right, I rebooted the system, then I got 2.6.32-16, I will notify when programmed backup take effect. I didn't reboot the system avoiding downtime.

Thanks for the quick answers.

Best regards

 

[103]

Oh, backup was going fine but:

INFO: starting new backup job: vzdump 101 102 103 104 105 106 107 --quiet 1 --mailto mailhere --mode snapshot --compress gzip --storage backups-alternate
INFO: skip external VMs: 105, 106, 107
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0')
INFO: Logical volume "vzsnap-proxmox-0" created
INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-101-2012_11_02-15_30_02.tar.gz'
INFO: Total bytes written: 360663040 (344MiB, 5.5MiB/s)
INFO: archive file size: 137MB
INFO: delete old backup '/mnt/pve/backups-alternate/dump/vzdump-openvz-101-2012_11_02-10_30_02.tar.gz'
INFO: Finished Backup of VM 101 (00:01:17)
INFO: Starting Backup of VM 102 (openvz)
INFO: CTID 102 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0')
INFO: Logical volume "vzsnap-proxmox-0" created
INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-102-2012_11_02-15_31_19.tar.gz'
INFO: Total bytes written: 1289113600 (1.3GiB, 5.9MiB/s)
INFO: archive file size: 529MB
INFO: delete old backup '/mnt/pve/backups-alternate/dump/vzdump-openvz-102-2012_11_02-13_16_55.tar.gz'
INFO: Finished Backup of VM 102 (00:03:50)
INFO: Starting Backup of VM 103 (openvz)
INFO: CTID 103 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0')
INFO: Logical volume "vzsnap-proxmox-0" created
INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-103-2012_11_02-15_35_10.tar.gz'
INFO: gzip: stdout: Input/output error
INFO: Total bytes written: 1606717440 (1.5GiB, ?/s)
INFO: tar: -: Wrote only 4096 of 10240 bytes
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 103 failed - command '(cd /mnt/vzsnap0/private/103;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system --null -T -|gzip) >/mnt/pve/backups-alternate/dump/vzdump-openvz-103-2012_11_02-15_35_10.tar.dat' failed: exit code 1
INFO: Starting Backup of VM 104 (openvz)
INFO: CTID 104 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox-0')
INFO: Logical volume "vzsnap-proxmox-0" created
INFO: creating archive '/mnt/pve/backups-alternate/dump/vzdump-openvz-104-2012_11_02-15_40_36.tar.gz'
INFO: Total bytes written: 474234880 (453MiB, 6.0MiB/s)
INFO: archive file size: 188MB
INFO: delete old backup '/mnt/pve/backups-alternate/dump/vzdump-openvz-104-2012_11_02-10_38_28.tar.gz'
INFO: Finished Backup of VM 104 (00:01:31)
INFO: Backup job finished with errors
TASK ERROR: job errors

This time only one CT got errors.

EDITED: this time I rebooted both nodes, then I ran a programmed backup successfully.

 

[dietmar]

Is there enough free space on the backup storage?

 

[103]

Well, I have to delete every existent backup, then trigger automated backup again. So far, no problems.

Best regards