Unable to back up lxc container

[rlaroy]

Hello all,

We have been experiencing an ongoing issue with only one container where it fails to back up. The other 50 containers back up fine, just not this one. I have migrated it to different hosts and tried backing up to an NFS point as well as with PBS (which has been working great for everything else). Each time we get a similar issue. The container is not locked and we performed a fsck on the container. Any thoughts?

Here is the most recent error with domain information removed:
INFO: starting new backup job: vzdump 124 --node pve1 --mode snapshot --remove 0 --storage backup1
INFO: Starting Backup of VM 124 (lxc)
INFO: Backup started at 2020-08-20 17:29:12
INFO: status = running
INFO: CT Name:
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd1
INFO: creating Proxmox Backup Server archive 'ct/124/2020-08-20T23:29:12Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3279199/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 124 --backup-time 1597966152 --repository backup@pbs@pve1-backup:datastore7
INFO: Starting backup: ct/124/2020-08-20T23:29:12Z
INFO: Client name: pve1
INFO: Starting protocol: 2020-08-20T17:29:13-06:00
INFO: Upload config file '/var/tmp/vzdumptmp3279199/etc/vzdump/pct.conf' to 'backup@pbs@pve1-backup:datastore7' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'backup@pbs@pve1-backup:datastore7' as root.pxar.didx
INFO: Error: error at "var/lib/php/session": stat failed on "var/lib/php/session/sess_eef2eql9hmo53qo39f2memsf7m": EUCLEAN: Structure needs cleaning
INFO: remove vzdump snapshot
2020-08-20 17:30:45.059 7f73357fa700 -1 librbd:bject_map::InvalidateRequest: 0x7f7338036aa0 should_complete: r=0
Removing snap: 100% complete...done.
ERROR: Backup of VM 124 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp3279199/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 124 --backup-time 1597966152 --repository backup@pbs@pve1:datastore7' failed: exit code 255
INFO: Failed at 2020-08-20 17:30:45
INFO: Backup job finished with errors
TASK ERROR: job errors

Container unlocked:
root@pve2:~# qm unlock 124
Configuration file 'nodes/pve2/qemu-server/124.conf' does not exist
root@pve2:~# pct unlock 124
no lock found trying to remove any lock
root@pve2:~# qm conf 124
Configuration file 'nodes/pve2/qemu-server/124.conf' does not exist

fsck on the container
root@pve2:~# pct shutdown 124
root@pve2:~# pct fsck 124
/dev/rbd4
fsck from util-linux 2.33.1
/dev/rbd4: clean, 100367/1769472 files, 1214257/7077888 blocks

 

[dietmar]

Seems the file system of the container has errors (EUCLEAN: Structure needs cleaning).
Does it work again after the fsck run (should)?

 

[rlaroy]

I tried the fsck on the container, the fsck ran but did not fix the problem

 

[grin]

It seems that the problem is not in the container but the method itself; I suspec that somewhere in the snapshotting the FS gets an inconsistent snapshotted state and the backup complains. Visibly the inconsistent files are recent ones and there is no reason to have any problem there, and a restart doesn't give any errors either.

It is a problem since it seems randomly appear on really busy containers.