TurnKey Nextcloud backup problem

C

Cussy

Member
Aug 9, 2020
8
2
23
24
Hi everyone.

I just recently dicided to try using ceph for my home servers. I backed up all VMs and LXCs and then restored them to ceph on the new cluster. Everything was working fine except for the Nextcloud LXCs. I have two nextcloud LXCs running and I am having same problem with both. I have used NFS share for backups as well as tried backing up to the local disk. Decided to try using PBS but the problem persists.

The log from backups:
PBS:
INFO: starting new backup job: vzdump 450 --storage backup --mode snapshot --node pve-node02 --remove 0 INFO: Starting Backup of VM 450 (lxc) INFO: Backup started at 2020-08-10 08:03:17 INFO: status = running INFO: CT Name: cloud INFO: including mount point rootfs ('/') in backup INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: create storage snapshot 'vzdump' /dev/rbd3 INFO: creating Proxmox Backup Server archive 'ct/450/2020-08-10T06:03:17Z' INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp532052/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 450 --backup-time 1597039397 --repository pve-ceph@pbs@10.23.23.240:backup-1tb INFO: Starting backup: ct/450/2020-08-10T06:03:17Z INFO: Client name: pve-node02 INFO: Starting protocol: 2020-08-10T08:03:17+02:00 INFO: Upload config file '/var/tmp/vzdumptmp532052/etc/vzdump/pct.conf' to 'pve-ceph@pbs' as pct.conf.blob INFO: Upload directory '/mnt/vzsnap0' to 'pve-ceph@pbs' as root.pxar.didx INFO: catalog upload error - channel closed INFO: Error: error at "var/lib/php/sessions": stat failed on "var/lib/php/sessions/sess_8an1gbh5j5tph06nahvcdjmopn": EUCLEAN: Structure needs cleaning INFO: remove vzdump snapshot Removing snap: 100% complete...done. ERROR: Backup of VM 450 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp532052/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 450 --backup-time 1597039397 --repository pve-ceph@pbs' failed: exit code 255 INFO: Failed at 2020-08-10 08:03:46 INFO: Backup job finished with errors TASK ERROR: job errors

Local Disk:
INFO: starting new backup job: vzdump 450 --compress zstd --mode snapshot --storage local --node pve-node02 --remove 0 INFO: Starting Backup of VM 450 (lxc) INFO: Backup started at 2020-08-10 07:39:37 INFO: status = running INFO: CT Name: cloud INFO: including mount point rootfs ('/') in backup INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: create storage snapshot 'vzdump' /dev/rbd3 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-450-2020_08_10-07_39_37.tar.zst' INFO: tar: ./var/lib/php/sessions/sess_h1kn7s0mtmrui2ki50ct03391s: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_13mqeebhf6iu27i2apg9qabbu5: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_u26he0miscmlgg7j92f6oh7hdr: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_kdqdkbb244j3tkll7o832og7du: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_auqblldlnc82cer906aktgoaor: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_f17qfcrcbdmhd5tq15ksrcosu7: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_bvoh9emtumo22b34epp3l30n3b: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_3a4gb297ktun55dlma35t2kaim: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_jo5n2gimnh5im9c8ftg7b5nsof: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_nsm15jmm81ntmnrtr1n7279rrt: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_td3n1sf3po01ks6i88cs5brvsr: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_p6qij7o6shvd3v6b41dd9gkegu: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_1mcoi78u6u0190tvakl9miqckm: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_q708dpqg14aqgjjn9fnd4bs4he: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_0e8dql4c8b0dnuou75kqud4rj0: Cannot stat: Structure needs cleaning INFO: tar: ./var/lib/php/sessions/sess_qfak439ie9sa78ss88j5leleuc: Cannot stat: Structure needs cleaning INFO: Total bytes written: 2119700480 (2.0GiB, 36MiB/s) INFO: tar: Exiting with failure status due to previous errors INFO: remove vzdump snapshot Removing snap: 100% complete...done. ERROR: Backup of VM 450 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/lib/vz/dump/vzdump-lxc-450-2020_08_10-07_39_37.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./var/lib/php/session/?*' '--exclude=./var/cache/*' '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/var/lib/vz/dump/vzdump-lxc-450-2020_08_10-07_39_37.tar.dat' failed: exit code 2 INFO: Failed at 2020-08-10 07:40:36 INFO: Backup job finished with errors TASK ERROR: job errors

I am running a four node proxmox cluster with ceph. All nodes are version 6.2-10 and ceph version is 14.2.10. PBS version is 0.8-9.
Nextcloud is set up with debian-10-turnkey-nextcloud_16.0-1_amd64.tar.gz template. The nextcloud version is 18.0.7.
Have tried setting up a new nextcloud LXC and tried backing it up, but I get the same problem.

Thanks for answers:)
 
Hi,
please try using pct fsck 450 when the container is shut down. Does it work afterwards?
If not, does the issue also occur if you create the container on a different storage than Ceph?
 
I have tried pct fsck 450 with the container shutted down but it doesen't help, but I did manage to backup container when it was shutted down. So now I just tried restoreing it to local storage and then take a backup and that worked without any problems.
Seems like ceph doesen't like the session files in "/var/lib/php/sessions".

After backing up the container when it was running on local storage I tried restoring it back to ceph again since i knew backup worked on that container when on local storage. But when trying to backup the container when it was running on ceph storage I got a different error message.
INFO: starting new backup job: vzdump 451 --remove 0 --node pve-node02 --mode snapshot --storage backup INFO: Starting Backup of VM 451 (lxc) INFO: Backup started at 2020-08-10 09:45:17 INFO: status = running INFO: CT Name: cloud INFO: including mount point rootfs ('/') in backup INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: create storage snapshot 'vzdump' /dev/rbd3 INFO: creating Proxmox Backup Server archive 'ct/451/2020-08-10T07:45:17Z' INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp613851/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 451 --backup-time 1597045517 --repository pve-ceph@pbs@10.23.23.240:backup-1tb INFO: Starting backup: ct/451/2020-08-10T07:45:17Z INFO: Client name: pve-node02 INFO: Starting protocol: 2020-08-10T09:45:18+02:00 INFO: Upload config file '/var/tmp/vzdumptmp613851/etc/vzdump/pct.conf' to 'pve-ceph@pbs@10.23.23.240:backup-1tb' as pct.conf.blob INFO: Upload directory '/mnt/vzsnap0' to 'pve-ceph@pbs@10.23.23.240:backup-1tb' as root.pxar.didx INFO: catalog upload error - channel closed INFO: Error: error at "var/lib/php/sessions": stat failed on "var/lib/php/sessions/sess_m7up7te1kmhfr805e5jo5pqkeh": EBADMSG: Not a data message INFO: remove vzdump snapshot Removing snap: 100% complete...done. ERROR: Backup of VM 451 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp613851/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 451 --backup-time 1597045517 --repository pve-ceph@pbs@10.23.23.240:backup-1tb' failed: exit code 255 INFO: Failed at 2020-08-10 09:46:08 INFO: Backup job finished with errors TASK ERROR: job errors

It's INFO: Error: error at "var/lib/php/sessions": stat failed on "var/lib/php/sessions/sess_m7up7te1kmhfr805e5jo5pqkeh": EBADMSG: Not a data message that seems to be the error.
 
Last edited:
Yes, such errors normally come from file system corruption. Did the pct fsck command show any repairs? Does ceph -w print something while you're running a backup? Anything Ceph-related in the /var/log/syslog around that time?
 
pct fsck showed no repairs. ceph -w shows nothing out of the ordinary and there is nothing ceph related in
/var/log/syslog.

It looks like the problem is in the php sessions files. Is there a way to exclude those from backup? I have heard you can exclude in /etc/pve/vzdump.cron, but from what I have read the files are still backed up during the initial backup, but are then excluded from the backup file. So this didn't fix the problem. Not sure about this though so correct me if I'm wrong.
 
Hi,
the root cause of the problem seems to be the interaction of krbd/vzdump and temporary directories, see this bug report.
if you use the --exclude-path option with a PBS backup storage and snapshot mode, it's currently ignored. I filed a bug.

So sadly it seems like there's no nice workaround with a
LXC on Ceph + snapshot mode + PBS backup server
setup at the moment. Using a different backup storage, you could use --exclude-path. Using suspend mode for these containers is another alternative. Or you can try the patch in the first bug report.
 
It's been a while, but as I'm currently working on the exclude-path bug, here is some quick update. The snapshot mode actually does apply the exclude options, but there was a typo in one of your paths. The s from sessions is missing:
Code: 
--exclude=./var/lib/php/session/?*

The bug only happens with the stop mode, and a patch for that should be available soonish.
 
Thanks for keeping me updated.
I will do some testing and see if maybe it works now that PBS has been updated more, and if the problem persists then hopefully it will work with future updates :)
 
@Fabian_E I have been testing now for some days and it seems to be working fine now. I'm not getting the "Cannot stat: Structure needs cleaning" error anymore :D
 
  • Like
Reactions: fiona
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back