[SOLVED] Backup issue

[Elfy]

Hi all,
I've getting an error with the backup of an LXC container to my NFS shared storage for a while now. I was hoping someone could make some sense of the error message and point me in the right direction to get it backing up again. I have other CT's and VM's backing up to this shared storage without issue.

INFO: starting new backup job: vzdump 100 --mode snapshot --remove 0 --compress lzo --storage shared-nfs --node Wash
INFO: Starting Backup of VM 100 (lxc)
INFO: status = running
INFO: CT Name: Teamspeak
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: Teamspeak
INFO: starting first sync /proc/30490/root// to /tmp/backup_tmp/vzdumptmp32759
INFO: rsync: readlink_stat("/proc/30490/root/usr/lib/X11/xedit/lisp/xedit.lsp") failed: Bad message (74)
INFO: IO error encountered -- skipping file deletion
INFO: Number of files: 23,942 (reg: 18,583, dir: 2,306, link: 3,021, dev: 2, special: 30)
INFO: Number of created files: 23,941 (reg: 18,583, dir: 2,305, link: 3,021, dev: 2, special: 30)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 18,565
INFO: Total file size: 1,253,051,716 bytes
INFO: Total transferred file size: 1,133,995,011 bytes
INFO: Literal data: 1,133,995,011 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 458,729
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 1,135,756,180
INFO: Total bytes received: 375,978
INFO: sent 1,135,756,180 bytes received 375,978 bytes 50,494,762.58 bytes/sec
INFO: total size is 1,253,051,716 speedup is 1.10
INFO: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
ERROR: Backup of VM 100 failed - command 'rsync --stats -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/30490/root///./ /tmp/backup_tmp/vzdumptmp32759' failed: exit code 23
INFO: Backup job finished with errors
TASK ERROR: job errors

If I try shutting down the CT and backing up to local storage, I get this error:

()
INFO: starting new backup job: vzdump 100 --compress lzo --remove 0 --node Wash --storage local --mode snapshot
INFO: Starting Backup of VM 100 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: Teamspeak
INFO: creating archive '/var/lib/vz/dump/vzdump-lxc-100-2018_04_17-10_45_43.tar.lzo'
INFO: tar: ./usr/lib/X11/xedit/lisp/xedit.lsp: Cannot stat: Bad message
INFO: Total bytes written: 1174835200 (1.1GiB, 65MiB/s)
INFO: tar: Exiting with failure status due to previous errors
ERROR: Backup of VM 100 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/tmp/backup_tmp/vzdumptmp16230' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/var/lib/vz/dump/vzdump-lxc-100-2018_04_17-10_45_43.tar.dat' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors

Thank you to anyone that can help me out!

 

[dietmar]

What kind of file ist that /root/usr/lib/X11/xedit/lisp/xedit.lsp exactly? Please can you also post your container configuration file?

 

[Elfy]

On the CT, I get no such file or directory unless I use the directory ./usr/lib/X11/xedit/lisp/. If I try to list the directory, I get:

root@Teamspeak:/usr/lib/X11/xedit/lisp# ls
ls: cannot access 'xedit.lsp': Bad message
indent.lsp  lisp.lsp  progmodes  syntax.lsp  xedit.lsp

And trying to do anything to the file gives lots of "Bad message" errors. Could this be a corrupt file?

root@Teamspeak:/usr/lib/X11/xedit/lisp# mv xedit.lsp xedit.old
mv: cannot stat 'xedit.lsp': Bad message
root@Teamspeak:/usr/lib/X11/xedit/lisp# sudo chmod 775 xedit.lsp
chmod: cannot access 'xedit.lsp': Bad message
root@Teamspeak:/usr/lib/X11/xedit/lisp# rm xedit.lsp
rm: cannot remove 'xedit.lsp': Bad message
root@Teamspeak:/usr/lib/X11/xedit/lisp# sudo rm -i xedit.lsp
rm: cannot remove 'xedit.lsp': Bad message
root@Teamspeak:/usr/lib/X11/xedit/lisp# rm -- xedit.lsp
rm: cannot remove 'xedit.lsp': Bad message

Here is my CT's configuration:

lxc.arch = amd64
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.monitor.unshare = 1
lxc.tty.max = 2
lxc.environment = TERM=linux
lxc.uts.name = Teamspeak
lxc.cgroup.memory.limit_in_bytes = 1077936128
lxc.cgroup.memory.memsw.limit_in_bytes = 1614807040
lxc.cgroup.cpu.cfs_period_us = 100000
lxc.cgroup.cpu.cfs_quota_us = 100000
lxc.cgroup.cpu.shares = 2000
lxc.rootfs.path = /var/lib/lxc/100/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth100i0
lxc.net.0.hwaddr = <redacted>
lxc.net.0.name = eth0
lxc.cgroup.cpuset.cpus = 0

 

[dietmar]

Please try to run a file system check:

# pct fsck 100

 

[Elfy]

Here's what I get:

root@Wash:~# pct fsck 100
fsck from util-linux 2.29.2
/dev/mapper/shared-vm--100--disk--2 contains a file system with errors, check forced.
/dev/mapper/shared-vm--100--disk--2: Deleted inode 4849 has zero dtime.  FIXED.
/dev/mapper/shared-vm--100--disk--2: Entry 'xedit.lsp' in /usr/lib/X11/xedit/lisp (4828) has deleted/unused inode 4849.  CLEARED.
/dev/mapper/shared-vm--100--disk--2: 23943/327680 files (0.3% non-contiguous), 324261/1310720 blocks
command 'fsck -a -l /dev/shared/vm-100-disk-2' failed: exit code 1

And...

.
.
.
INFO: Finished Backup of VM 100 (00:01:26)
INFO: Backup job finished successfully
TASK OK

Looks like that did the trick! Thank you so much!

I guess for future reference, in general a "Deleted inode XXXX has zero dtime" error is usually due to improper shutdown of either the CT or the host.