LXC (unprivileged) backup task failing

[Pavel Hruška]

Hello everyone, I've created and set up my first unprivileged LXC container, everything was okay until I've run into problems with backing up this LXC. The backup is failing with this error:

ERROR: Backup of VM 320 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/XXX/dump/vzdump-lxc-320-2018_11_05-07_23_29.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ ./var/ftp | pigz -p 2 >/mnt/pve/XXX/dump/vzdump-lxc-320-2018_11_05-07_23_29.tar.dat' failed: exit code 2

Any ideas? I am lost how to debug this...

 

[dcsapak]

do you backup to an nfs/cifs server? if yes please set a tmpdir in /etc/vzdump.conf (see also https://pve.proxmox.com/wiki/Backup_and_Restore )

 

[Pavel Hruška]

Great, that did the trick (EDIT: Yes, backup target is on NFS storage).

May I ask why the tmpdir is needed for the backup? And is the unprivileged LXC cause of the problem?

Thank you.

 

[Pavel Hruška]

I'm just curious if the backup data are passing thru the tmp dir when backing up - just in case my tmp is on SSD, would it wear my drive out by backups?

 

[dcsapak]

I'm just curious if the backup data are passing thru the tmp dir when backing up

yes, that is the purpose of the tmpdir

just in case my tmp is on SSD, would it wear my drive out by backups?

depends on your disk/how much you backup/the amount of data/etc

 

[Pavel Hruška]

Okay, that's not the best news (SSD will be (slowly) worn out by buckups), but okay. I'm glad to know how things works. LXC is full of pros/cons needed to be considered when designing stuff. Good to know.

Thank you for answers!

 

[GPLExpert]

Hello,

For this tmpdir option, what capacity we need to use ?

 

[dcsapak]

For this tmpdir option, what capacity we need to use ?

the maximum size of the biggest container you want to backup, since all files of a container will be copied there temporarily

 

[olepinguin]

do you backup to an nfs/cifs server? if yes please set a tmpdir in /etc/vzdump.conf (see also https://pve.proxmox.com/wiki/Backup_and_Restore )

Hi and sorry for the stupip question:
the temp dir, needs to be on PVE host, or on the NFS-Server?? what is needed 775?

I can't backup my unprivileged containers anymore ..
INFO: creating vzdump archive '/mnt/pve/TrueNAS/dump/vzdump-lxc-103-2022_04_21-10_15_09.tar.zst' INFO: tar: ./etc/shadow-: Cannot open: Permission denied INFO: tar: ./etc/ssl/private: Cannot open: Permission denied INFO: tar: ./etc/gshadow-: Cannot open: Permission denied INFO: tar: ./etc/security/opasswd: Cannot open: Permission denied INFO: tar: ./etc/ssh/ssh_host_ed25519_key: Cannot open: Permission denied INFO: tar: ./etc/ssh/ssh_host_dsa_key: Cannot open: Permission denied INFO: tar: ./etc/ssh/ssh_host_rsa_key: Cannot open: Permission denied INFO: tar: ./etc/ssh/ssh_host_ecdsa_key: Cannot open: Permission denied INFO: tar: ./etc/.pwd.lock: Cannot open: Permission denied ... INFO: tar: Exiting with failure status due to previous errors INFO: cleanup temporary 'vzdump' snapshot ERROR: Backup of VM 103 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/tmpfs/vzdumptmp2056191_103/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/TrueNAS/dump/vzdump-lxc-103-2022_04_21-10_15_09.tar.dat' failed: exit code 2 INFO: Failed at 2022-04-21 10:16:16 INFO: Backup job finished with errors TASK ERROR: job errors

Thanks,
Ole

Edit, really don't understand what happened, even local folder on host as backup target with 777 didn't work anymore.
Backups worked - for ages on several target, also for replicated HA-containers ...suddenly it stoped working

root@PVE:~# chmod 777 -R /var/lib/vz/dump root@PVE:~# ls -la /var/lib/vz/ total 20 drwxr-xr-x 5 root root 4096 Apr 13 21:09 . drwxr-xr-x 37 root root 4096 Apr 13 21:09 .. drwxrwxrwx 2 root root 4096 Apr 21 11:26 dump drwxr-xr-x 2 root root 4096 Nov 24 18:32 images drwxr-xr-x 4 root root 4096 Apr 14 07:25 template root@PVE:~#

result is the same:

INFO: including mount point rootfs ('/') in backup INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: create storage snapshot 'vzdump' INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-101-2022_04_21-11_30_25.tar.zst' INFO: tar: ./etc/ssl/private: Cannot open: Permission denied ...... INFO: tar: ./var/spool/postfix/incoming: Cannot open: Permission denied INFO: Total bytes written: 845670400 (807MiB, 75MiB/s) INFO: tar: Exiting with failure status due to previous errors INFO: cleanup temporary 'vzdump' snapshot ERROR: Backup of VM 101 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=tmpfs/vzdumptmp2564247_101/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/var/lib/vz/dump/vzdump-lxc-101-2022_04_21-11_30_25.tar.dat' failed: exit code 2 INFO: Failed at 2022-04-21 11:30:37 INFO: Backup job finished with errors TASK ERROR: job errors


edit II:
and done, I restored the volume from ext-storage again to "get" user/group 100000 for the root-disk again and with that and/or the temp-dir in "/etc/vzdump.conf" it works..

drwxr-xr-x 20 root root 4096 Apr 21 09:58 .. drwxr-xr-x 21 100000 100000 21 Apr 21 12:37 subvol-102-disk-0