one of 12 backup failed, need advice

[RobFantini]

Hello, one vm failled backup:

  2446: 2017-09-21 21:10:02 INFO: Starting Backup of VM 2446 (lxc)
  2446: 2017-09-21 21:10:02 INFO: status = running
  2446: 2017-09-21 21:10:02 INFO: CT Name: p4lxceph
  2446: 2017-09-21 21:10:02 INFO: backup mode: snapshot
  2446: 2017-09-21 21:10:02 INFO: bandwidth limit: 500000 KB/s
  2446: 2017-09-21 21:10:02 INFO: ionice priority: 7
  2446: 2017-09-21 21:10:02 INFO: suspend vm to make snapshot
  2446: 2017-09-21 21:10:02 INFO: create storage snapshot 'vzdump'
  2446: 2017-09-21 21:10:05 INFO: resume vm
  2446: 2017-09-21 21:10:05 INFO: vm is online again after 3 seconds
  2446: 2017-09-21 21:10:05 INFO: creating archive '/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_21-21_10_02.tar.lzo'
  2446: 2017-09-21 21:10:46 INFO: tar: ./rsnapshot/hourly.0/localhost/fbc/linux-install/13-do-on-most-systems: Cannot stat:
Structure needs cleaning
  2446: 2017-09-21 21:10:46 INFO: tar: ./rsnapshot/hourly.0/localhost/root/.monit.state: Cannot stat: Structure needs cleaning
  2446: 2017-09-21 21:32:03 INFO: Total bytes written: 142330716160 (133GiB, 103MiB/s)
  2446: 2017-09-21 21:32:03 INFO: tar: Exiting with failure status due to previous errors
  2446: 2017-09-21 21:32:11 INFO: remove vzdump snapshot
  2446: 2017-09-21 21:32:16 ERROR: Backup of VM 2446 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p
--sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability'
'--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored'
'--directory=/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_21-21_10_02.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0'
--no-anchored '--exclude=lost+found' --anchored ./ ./bkup | cstream -t 512000000 | lzop
>/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_21-21_10_02.tar.dat' failed: exit code 2

now I'll try again, but could get same result EDIT :failed again:

INFO: starting new backup job: vzdump 2446 --mode snapshot --storage bkup-nfs --remove 0 --node sys5 --compress lzo
INFO: Starting Backup of VM 2446 (lxc)
INFO: status = running
INFO: CT Name: p4lxceph
INFO: backup mode: snapshot
INFO: bandwidth limit: 500000 KB/s
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
/dev/rbd5
/dev/rbd6
INFO: resume vm
INFO: vm is online again after 2 seconds
INFO: creating archive '/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_21-21_55_45.tar.lzo'
INFO: tar: ./rsnapshot/hourly.0/localhost/fbc/linux-install/13-do-on-most-systems: Cannot stat: Structure needs cleaning
INFO: tar: ./rsnapshot/hourly.0/localhost/root/.monit.state: Cannot stat: Structure needs cleaning

any clues on what causes this:

tar: ./rsnapshot/hourly.0/localhost/fbc/linux-install/13-do-on-most-systems: Cannot stat:
Structure needs cleaning

this is a LXC system so if fsck should be done - could someone give a clue on where to do that?

 

[dcsapak]

you can use pct for a fsck

pct fsck <vmdi>

see

man pct

for more details

 

[RobFantini]

fsck done, backup fails again:

# pct fsck 2446 --device mp0
fsck from util-linux 2.29.2
/dev/rbd3: clean, 245759/16384000 files, 32960256/65536000 blocks

# pct fsck 2446 --device mp0 --force 1
fsck from util-linux 2.29.2
/dev/rbd3: 245759/16384000 files (0.9% non-contiguous), 32960256/65536000 blocks

NFO: starting new backup job: vzdump 2446 --mode snapshot --storage bkup-nfs --remove 0 --compress lzo --node sys5
INFO: Starting Backup of VM 2446 (lxc)
INFO: status = running
INFO: CT Name: p4lxceph
INFO: backup mode: snapshot
INFO: bandwidth limit: 500000 KB/s
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
/dev/rbd5
/dev/rbd6
INFO: resume vm
INFO: vm is online again after 2 seconds
INFO: creating archive '/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_22-06_22_24.tar.lzo'
INFO: tar: ./var/run/sshd.pid: Cannot stat: Structure needs cleaning
INFO: tar: ./var/run/zabbix-agent/zabbix_agentd.pid: Cannot stat: Structure needs cleaning
INFO: tar: ./var/run/monit.pid: Cannot stat: Structure needs cleaning
INFO: tar: ./var/run/atd.pid: Cannot stat: Structure needs cleaning
INFO: Total bytes written: 139885352960 (131GiB, 115MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: remove vzdump snapshot
Removing snap: 100% complete...done.
Removing snap: 100% complete...done.
ERROR: Backup of VM 2446 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_22-06_22_24.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored ./ ./bkup | cstream -t 512000000 | lzop >/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_22-06_22_24.tar.dat' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors

more info soon.

 

[RobFantini]

- bad files are on 1ST disk. 2ND disk is a much larger disk for data.
- I turned off backup of 2ND disk to speed up the process.
- the backup works:

INFO: starting new backup job: vzdump 2446 --mode snapshot --remove 0 --compress lzo --node sys5 --storage bkup-nfs
INFO: Starting Backup of VM 2446 (lxc)
INFO: status = running
INFO: CT Name: p4lxceph
INFO: excluding volume mount point mp0 ('/bkup') from backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 500000 KB/s
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd5
INFO: creating archive '/mnt/pve/bkup-nfs/dump/vzdump-lxc-2446-2017_09_22-06_51_49.tar.lzo'
INFO: Total bytes written: 8669276160 (8.1GiB, 162MiB/s)
INFO: archive file size: 1.93GB
INFO: remove vzdump snapshot
Removing snap: 100% complete...done.
INFO: Finished Backup of VM 2446 (00:00:54)
INFO: Backup job finished successfully
TASK OK

now I do not understand why the fails only when both disks are backed up.

 

[RobFantini]

update, same issue, to different vm . same sized lxc .

123: 2017-09-30 21:29:08 INFO: Starting Backup of VM 123 (lxc)
  123: 2017-09-30 21:29:08 INFO: status = running
  123: 2017-09-30 21:29:08 INFO: CT Name: pro4bkup
  123: 2017-09-30 21:29:08 INFO: backup mode: snapshot
  123: 2017-09-30 21:29:08 INFO: bandwidth limit: 500000 KB/s
  123: 2017-09-30 21:29:08 INFO: ionice priority: 7
  123: 2017-09-30 21:29:08 INFO: suspend vm to make snapshot
  123: 2017-09-30 21:29:08 INFO: create storage snapshot 'vzdump'
  123: 2017-09-30 21:29:11 INFO: resume vm
  123: 2017-09-30 21:29:11 INFO: vm is online again after 3 seconds
  123: 2017-09-30 21:29:11 INFO: creating archive '/mnt/pve/bkup-nfs/dump/vzdump-lxc-123-2017_09_30-21_29_08.tar.lzo'

  123: 2017-09-30 21:29:14 INFO: tar: ./fbc/bin/.do.not.delete.this: Cannot stat: Structure needs cleaning  

  123: 2017-09-30 21:50:01 INFO: Total bytes written: 139812311040 (131GiB, 107MiB/s)
  123: 2017-09-30 21:50:01 INFO: tar: Exiting with failure status due to previous errors
  123: 2017-09-30 21:50:10 INFO: remove vzdump snapshot
  123: 2017-09-30 21:50:17 ERROR: Backup of VM 123 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p
--sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability'
'--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored'
'--directory=/mnt/pve/bkup-nfs/dump/vzdump-lxc-123-2017_09_30-21_29_08.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0'
--no-anchored '--exclude=lost+found' --anchored ./ ./bkup | cstream -t 512000000 | lzop
>/mnt/pve/bkup-nfs/dump/vzdump-lxc-123-2017_09_30-21_29_08.tar.dat' failed: exit code 2

without doing a fsck , I just tried backing up again and that worked.