TurnKey Nextcloud backup problem

Cussy

New Member
Aug 9, 2020
3
0
1
20
Hi everyone.

I just recently dicided to try using ceph for my home servers. I backed up all VMs and LXCs and then restored them to ceph on the new cluster. Everything was working fine except for the Nextcloud LXCs. I have two nextcloud LXCs running and I am having same problem with both. I have used NFS share for backups as well as tried backing up to the local disk. Decided to try using PBS but the problem persists.

The log from backups:
PBS:
INFO: starting new backup job: vzdump 450 --storage backup --mode snapshot --node pve-node02 --remove 0
INFO: Starting Backup of VM 450 (lxc)
INFO: Backup started at 2020-08-10 08:03:17
INFO: status = running
INFO: CT Name: cloud
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd3
INFO: creating Proxmox Backup Server archive 'ct/450/2020-08-10T06:03:17Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp532052/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 450 --backup-time 1597039397 --repository pve-ceph@pbs@10.23.23.240:backup-1tb
INFO: Starting backup: ct/450/2020-08-10T06:03:17Z
INFO: Client name: pve-node02
INFO: Starting protocol: 2020-08-10T08:03:17+02:00
INFO: Upload config file '/var/tmp/vzdumptmp532052/etc/vzdump/pct.conf' to 'pve-ceph@pbs' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'pve-ceph@pbs' as root.pxar.didx
INFO: catalog upload error - channel closed
INFO: Error: error at "var/lib/php/sessions": stat failed on "var/lib/php/sessions/sess_8an1gbh5j5tph06nahvcdjmopn": EUCLEAN: Structure needs cleaning
INFO: remove vzdump snapshot
Removing snap: 100% complete...done.
ERROR: Backup of VM 450 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp532052/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 450 --backup-time 1597039397 --repository pve-ceph@pbs' failed: exit code 255
INFO: Failed at 2020-08-10 08:03:46
INFO: Backup job finished with errors
TASK ERROR: job errors


Local Disk:
INFO: starting new backup job: vzdump 450 --compress zstd --mode snapshot --storage local --node pve-node02 --remove 0
INFO: Starting Backup of VM 450 (lxc)
INFO: Backup started at 2020-08-10 07:39:37
INFO: status = running
INFO: CT Name: cloud
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd3
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-450-2020_08_10-07_39_37.tar.zst'
INFO: tar: ./var/lib/php/sessions/sess_h1kn7s0mtmrui2ki50ct03391s: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_13mqeebhf6iu27i2apg9qabbu5: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_u26he0miscmlgg7j92f6oh7hdr: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_kdqdkbb244j3tkll7o832og7du: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_auqblldlnc82cer906aktgoaor: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_f17qfcrcbdmhd5tq15ksrcosu7: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_bvoh9emtumo22b34epp3l30n3b: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_3a4gb297ktun55dlma35t2kaim: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_jo5n2gimnh5im9c8ftg7b5nsof: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_nsm15jmm81ntmnrtr1n7279rrt: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_td3n1sf3po01ks6i88cs5brvsr: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_p6qij7o6shvd3v6b41dd9gkegu: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_1mcoi78u6u0190tvakl9miqckm: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_q708dpqg14aqgjjn9fnd4bs4he: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_0e8dql4c8b0dnuou75kqud4rj0: Cannot stat: Structure needs cleaning
INFO: tar: ./var/lib/php/sessions/sess_qfak439ie9sa78ss88j5leleuc: Cannot stat: Structure needs cleaning
INFO: Total bytes written: 2119700480 (2.0GiB, 36MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: remove vzdump snapshot
Removing snap: 100% complete...done.
ERROR: Backup of VM 450 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/lib/vz/dump/vzdump-lxc-450-2020_08_10-07_39_37.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./var/lib/php/session/?*' '--exclude=./var/cache/*' '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/var/lib/vz/dump/vzdump-lxc-450-2020_08_10-07_39_37.tar.dat' failed: exit code 2
INFO: Failed at 2020-08-10 07:40:36
INFO: Backup job finished with errors
TASK ERROR: job errors


I am running a four node proxmox cluster with ceph. All nodes are version 6.2-10 and ceph version is 14.2.10. PBS version is 0.8-9.
Nextcloud is set up with debian-10-turnkey-nextcloud_16.0-1_amd64.tar.gz template. The nextcloud version is 18.0.7.
Have tried setting up a new nextcloud LXC and tried backing it up, but I get the same problem.

Thanks for answers:)
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
474
48
28
Hi,
please try using pct fsck 450 when the container is shut down. Does it work afterwards?
If not, does the issue also occur if you create the container on a different storage than Ceph?
 

Cussy

New Member
Aug 9, 2020
3
0
1
20
I have tried pct fsck 450 with the container shutted down but it doesen't help, but I did manage to backup container when it was shutted down. So now I just tried restoreing it to local storage and then take a backup and that worked without any problems.
Seems like ceph doesen't like the session files in "/var/lib/php/sessions".

After backing up the container when it was running on local storage I tried restoring it back to ceph again since i knew backup worked on that container when on local storage. But when trying to backup the container when it was running on ceph storage I got a different error message.
INFO: starting new backup job: vzdump 451 --remove 0 --node pve-node02 --mode snapshot --storage backup
INFO: Starting Backup of VM 451 (lxc)
INFO: Backup started at 2020-08-10 09:45:17
INFO: status = running
INFO: CT Name: cloud
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd3
INFO: creating Proxmox Backup Server archive 'ct/451/2020-08-10T07:45:17Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp613851/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 451 --backup-time 1597045517 --repository pve-ceph@pbs@10.23.23.240:backup-1tb
INFO: Starting backup: ct/451/2020-08-10T07:45:17Z
INFO: Client name: pve-node02
INFO: Starting protocol: 2020-08-10T09:45:18+02:00
INFO: Upload config file '/var/tmp/vzdumptmp613851/etc/vzdump/pct.conf' to 'pve-ceph@pbs@10.23.23.240:backup-1tb' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'pve-ceph@pbs@10.23.23.240:backup-1tb' as root.pxar.didx
INFO: catalog upload error - channel closed
INFO: Error: error at "var/lib/php/sessions": stat failed on "var/lib/php/sessions/sess_m7up7te1kmhfr805e5jo5pqkeh": EBADMSG: Not a data message
INFO: remove vzdump snapshot
Removing snap: 100% complete...done.
ERROR: Backup of VM 451 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp613851/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 451 --backup-time 1597045517 --repository pve-ceph@pbs@10.23.23.240:backup-1tb' failed: exit code 255
INFO: Failed at 2020-08-10 09:46:08
INFO: Backup job finished with errors
TASK ERROR: job errors


It's INFO: Error: error at "var/lib/php/sessions": stat failed on "var/lib/php/sessions/sess_m7up7te1kmhfr805e5jo5pqkeh": EBADMSG: Not a data message that seems to be the error.
 
Last edited:

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
474
48
28
Yes, such errors normally come from file system corruption. Did the pct fsck command show any repairs? Does ceph -w print something while you're running a backup? Anything Ceph-related in the /var/log/syslog around that time?
 

Cussy

New Member
Aug 9, 2020
3
0
1
20
pct fsck showed no repairs. ceph -w shows nothing out of the ordinary and there is nothing ceph related in
/var/log/syslog.

It looks like the problem is in the php sessions files. Is there a way to exclude those from backup? I have heard you can exclude in /etc/pve/vzdump.cron, but from what I have read the files are still backed up during the initial backup, but are then excluded from the backup file. So this didn't fix the problem. Not sure about this though so correct me if I'm wrong.
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
474
48
28
Hi,
the root cause of the problem seems to be the interaction of krbd/vzdump and temporary directories, see this bug report.
if you use the --exclude-path option with a PBS backup storage and snapshot mode, it's currently ignored. I filed a bug.

So sadly it seems like there's no nice workaround with a
LXC on Ceph + snapshot mode + PBS backup server
setup at the moment. Using a different backup storage, you could use --exclude-path. Using suspend mode for these containers is another alternative. Or you can try the patch in the first bug report.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!