[SOLVED] Can't backup CT: "Cannot stat: Structure needs cleaning"

morfair

Member
Feb 20, 2013
48
1
8
Hi, all!
I have CT with root disk on CEPH rbd. Now I can't to make backup:

---------------------
INFO: starting new backup job: vzdump 187 --node pve05 --compress lzo --storage storage6 --mode snapshot --remove 0
INFO: Starting Backup of VM 187 (lxc)
INFO: status = running
INFO: CT Name: dom.mtt.ru
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd15
INFO: creating archive '/mnt/pve/storage6/dump/vzdump-lxc-187-2016_12_27-09_50_54.tar.lzo'
INFO: tar: ./var/lib/php5/sess_bg325j1sf9a93q9a1f479e15q5: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_ivr5dmm2frhad73sufgggseka5: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_vu7q6b6jmh4t6so8vvkq09eig4: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_bi8oimfl8oq25c72k5l5pj2h10: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_pu8ns09ce96lh4quep2q5a7eo7: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_i3ngm0df47tql0aa2m3jekp657: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_eihjg7nv9v85f0iln1l067dfe1: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_469cns1726i777ndpgacnfabc0: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_fei1csc4nfqv6egptpak6dffj5: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_sb7ofncvjqksp28fu8elq9cqi6: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_5iufcpdm4k11lj75mo6l49dk01: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_sm2e96f7kigdm5uodtak9962g4: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_65afcjcuhmthf1m48dqro9fen0: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php5/sess_eo2bi3iue6fbsst7ljce00du07: Cannot stat: Structure needs cleaning
INFO: Total bytes written: 2518620160 (2.4GiB, 7.8MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: remove vzdump snapshot
ERROR: Backup of VM 187 failed - command 'set -o pipefail && tar cpf - --totals --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/storage6/dump/vzdump-lxc-187-2016_12_27-09_50_54.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/mnt/pve/storage6/dump/vzdump-lxc-187-2016_12_27-09_50_54.tar.dat' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors
---------------------

How to fix this?
 
Seems the file-system is corrupt. Try to correct using fsck (shutdown the CT first):

# pct shutdown 187
# pct fsck 187
 
Hi @morfair,

I've got the same issues you're describing on a CEPH-storage. Repairing with fsck as @dietmar mentions, doesn't help because the filesystem isn't broken.
It looks like it only occurs on files which are in use by the system during the backup-process, like PHP-session files, logfiles and cache-files.

I'm able to reduce the errors by adding this line to /etc/vzdump.conf
exclude-path: /var/lib/php/session/?* /var/cache/*

What are your experiences?
 
Is the error message the same? ("Structure needs cleaning")
 
Hi Wolfgang,

Yes, the same error. Here are some examples.

6070: Jan 31 05:19:10 INFO: creating archive '/mnt/pve/ib-onsite-backup2/dump/vzdump-lxc-6070-2017_01_31-05_19_07.tar.gz'
6070: Jan 31 05:27:58 INFO: tar: ./var/lib/php/session/sess_nk8q3hh36qn9vgo09imou6sph5: Cannot stat: Structure needs cleaning
6070: Jan 31 05:29:46 INFO: Total bytes written: 26294200320 (25GiB, 40MiB/s)
6070: Jan 31 05:29:46 INFO: tar: Exiting with failure status due to previous errors

6080: Jan 30 03:41:54 INFO: creating archive '/mnt/pve/ib-onsite-backup2/dump/vzdump-lxc-6080-2017_01_30-03_41_52.tar.gz'

6080: Jan 30 03:48:48 INFO: tar: ./var/log/httpd/mydomain.com/01302017-access.log: Cannot stat: Structure needs cleaning

6080: Jan 30 03:49:10 INFO: Total bytes written: 17257349120 (17GiB, 38MiB/s)
6080: Jan 30 03:49:10 INFO: tar: Exiting with failure status due to previous errors

6070: Jan 27 04:14:23 INFO: creating archive '/mnt/pve/ib-onsite-backup2/dump/vzdump-lxc-6070-2017_01_27-04_14_21.tar.gz'
6070: Jan 27 04:23:34 INFO: tar: ./var/cache/mod_pagespeed/v2/anotherdomain.com/http,3A/,2Fwww.anotherdomain.com/nl/vacatures,3Fview=listing,26similar=3243,26start=60,: Cannot stat: Structure needs cleaning


I took a long-shot by upgrading ceph from hammer to jewel (as you know from the other thread https://forum.proxmox.com/threads/ceph-upgrade-to-jewel-ha-migrate-not-working-anymore.32083/ :)), but that didn't help.

Unfortunately the whole backup is aborted, so you end up with no backup.

Best Regards,
Ruben
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!