[SOLVED] Can't backup CT: "Cannot stat: Structure needs cleaning"

[morfair]

Hi, all!
I have CT with root disk on CEPH rbd. Now I can't to make backup:

---------------------
INFO: starting new backup job: vzdump 187 --node pve05 --compress lzo --storage storage6 --mode snapshot --remove 0
INFO: Starting Backup of VM 187 (lxc)
INFO: status = running
INFO: CT Name: dom.mtt.ru
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd15
INFO: creating archive '/mnt/pve/storage6/dump/vzdump-lxc-187-2016_12_27-09_50_54.tar.lzo'
INFO: tar: ./var/lib/php5/sess_bg325j1sf9a93q9a1f479e15q5: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_ivr5dmm2frhad73sufgggseka5: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_vu7q6b6jmh4t6so8vvkq09eig4: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_bi8oimfl8oq25c72k5l5pj2h10: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_pu8ns09ce96lh4quep2q5a7eo7: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_i3ngm0df47tql0aa2m3jekp657: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_eihjg7nv9v85f0iln1l067dfe1: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_469cns1726i777ndpgacnfabc0: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_fei1csc4nfqv6egptpak6dffj5: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_sb7ofncvjqksp28fu8elq9cqi6: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_5iufcpdm4k11lj75mo6l49dk01: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_sm2e96f7kigdm5uodtak9962g4: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_65afcjcuhmthf1m48dqro9fen0: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_eo2bi3iue6fbsst7ljce00du07: Cannot stat: Structure needs cleaning
INFO: Total bytes written: 2518620160 (2.4GiB, 7.8MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: remove vzdump snapshot
ERROR: Backup of VM 187 failed - command 'set -o pipefail && tar cpf - --totals --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/storage6/dump/vzdump-lxc-187-2016_12_27-09_50_54.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/mnt/pve/storage6/dump/vzdump-lxc-187-2016_12_27-09_50_54.tar.dat' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors
---------------------

How to fix this?

 

[dietmar]

Seems the file-system is corrupt. Try to correct using fsck (shutdown the CT first):

# pct shutdown 187
# pct fsck 187

 

[morfair]

dietmar, thanks! It's solve the problem.

 

[Ruben Waitz]

Hi @morfair,

I've got the same issues you're describing on a CEPH-storage. Repairing with fsck as @dietmar mentions, doesn't help because the filesystem isn't broken.
It looks like it only occurs on files which are in use by the system during the backup-process, like PHP-session files, logfiles and cache-files.

I'm able to reduce the errors by adding this line to /etc/vzdump.conf
exclude-path: /var/lib/php/session/?* /var/cache/*

What are your experiences?

 

[wbumiller]

Is the error message the same? ("Structure needs cleaning")

 

[Ruben Waitz]

Hi Wolfgang,

Yes, the same error. Here are some examples.

6070: Jan 31 05:19:10 INFO: creating archive '/mnt/pve/ib-onsite-backup2/dump/vzdump-lxc-6070-2017_01_31-05_19_07.tar.gz'
6070: Jan 31 05:27:58 INFO: tar: ./var/lib/php/session/sess_nk8q3hh36qn9vgo09imou6sph5: Cannot stat: Structure needs cleaning
6070: Jan 31 05:29:46 INFO: Total bytes written: 26294200320 (25GiB, 40MiB/s)
6070: Jan 31 05:29:46 INFO: tar: Exiting with failure status due to previous errors

6080: Jan 30 03:41:54 INFO: creating archive '/mnt/pve/ib-onsite-backup2/dump/vzdump-lxc-6080-2017_01_30-03_41_52.tar.gz'

6080: Jan 30 03:48:48 INFO: tar: ./var/log/httpd/mydomain.com/01302017-access.log: Cannot stat: Structure needs cleaning

6080: Jan 30 03:49:10 INFO: Total bytes written: 17257349120 (17GiB, 38MiB/s)
6080: Jan 30 03:49:10 INFO: tar: Exiting with failure status due to previous errors

6070: Jan 27 04:14:23 INFO: creating archive '/mnt/pve/ib-onsite-backup2/dump/vzdump-lxc-6070-2017_01_27-04_14_21.tar.gz'
6070: Jan 27 04:23:34 INFO: tar: ./var/cache/mod_pagespeed/v2/anotherdomain.com/http,3A/,2Fwww.anotherdomain.com/nl/vacatures,3Fview=listing,26similar=3243,26start=60,: Cannot stat: Structure needs cleaning

I took a long-shot by upgrading ceph from hammer to jewel (as you know from the other thread https://forum.proxmox.com/threads/ceph-upgrade-to-jewel-ha-migrate-not-working-anymore.32083/ ), but that didn't help.

Unfortunately the whole backup is aborted, so you end up with no backup.

Best Regards,
Ruben