Backup of some LXC's failing (Permission Denied)

[FunnyManVA]

I've searched a lot and everyone else's issues seem to be related to NFS permissions and not being able to create the directory on temp space on NFS. This is not the problem I'm having. I don't have enough space on the Proxmox nodes to use local temp so I use a big NFS share to store everything (which is backed by a ZFS pool). Most of the LXC containers are working fine. Just a few get a Permission Denied when creating the .tar.gz from the temp folder. It creates the folder for the backup in a temp directory off NFS just fine, it just seems to fail on some files when making the archive. Not sure why just some containers and not others. I took one of the smaller ones that's having issues and created a local backup (which works fine), then deleted and restored it hoping it would clean out any odd issues, but even after all that it still fails backups. Any ideas are appreciated as I've tried just about everything I can think of. Below is a log of one of them.

INFO: Starting Backup of VM 150 (lxc)
INFO: status = running
INFO: CT Name: ubuntu-desktop
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: ubuntu-desktop
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/24535/root// to /mnt/pve/cloud_storage/dump/vzdump-lxc-150-2019_08_02-01_58_53.tmp
INFO: Number of files: 109,381 (reg: 74,844, dir: 8,634, link: 25,870, special: 33)
INFO: Number of created files: 109,380 (reg: 74,844, dir: 8,633, link: 25,870, special: 33)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 74,818
INFO: Total file size: 3,935,408,653 bytes
INFO: Total transferred file size: 3,784,864,019 bytes
INFO: Literal data: 3,784,864,019 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 3,800,567
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 3,792,892,173
INFO: Total bytes received: 1,547,702
INFO: sent 3,792,892,173 bytes received 1,547,702 bytes 2,938,009.97 bytes/sec
INFO: total size is 3,935,408,653 speedup is 1.04
INFO: first sync finished (1291 seconds)
INFO: suspend vm
INFO: starting final sync /proc/24535/root// to /mnt/pve/cloud_storage/dump/vzdump-lxc-150-2019_08_02-01_58_53.tmp
INFO: Number of files: 109,381 (reg: 74,844, dir: 8,634, link: 25,870, special: 33)
INFO: Number of created files: 0
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 3
INFO: Total file size: 3,935,408,653 bytes
INFO: Total transferred file size: 6,120 bytes
INFO: Literal data: 2,184 bytes
INFO: Matched data: 3,936 bytes
INFO: File list size: 327,597
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 3,861,141
INFO: Total bytes received: 9,583
INFO: sent 3,861,141 bytes received 9,583 bytes 126,908.98 bytes/sec
INFO: total size is 3,935,408,653 speedup is 1,016.71
INFO: final sync finished (30 seconds)
INFO: resume vm
INFO: vm is online again after 30 seconds
INFO: creating archive '/mnt/pve/cloud_storage/dump/vzdump-lxc-150-2019_08_02-01_58_53.tar.lzo'
INFO: tar: ./var/log/speech-dispatcher: Cannot open: Permission denied
INFO: tar: ./var/log/auth.log.1: Cannot open: Permission denied
INFO: tar: ./var/log/syslog.1: Cannot open: Permission denied
INFO: tar: ./var/log/auth.log: Cannot open: Permission denied
INFO: tar: ./var/log/mail.log: Cannot open: Permission denied
INFO: tar: ./var/log/syslog: Cannot open: Permission denied
INFO: tar: ./var/log/mail.log.1: Cannot open: Permission denied
INFO: tar: ./var/log/syslog.2.gz: Cannot open: Permission denied
INFO: tar: ./var/spool/rsyslog: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/bounce: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/flush: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/active: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/saved: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/hold: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/deferred: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/private: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/public: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/incoming: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/maildrop: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/trace: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/corrupt: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/defer: Cannot open: Permission denied
INFO: tar: ./var/cache/apt/archives/partial: Cannot open: Permission denied
INFO: tar: ./var/cache/cups/ppds.dat: Cannot open: Permission denied
INFO: tar: ./var/lib/colord/.cache: Cannot open: Permission denied
INFO: tar: ./var/lib/apt/lists/partial: Cannot open: Permission denied
INFO: tar: ./var/lib/lightdm-data/lightdm: Cannot open: Permission denied
INFO: tar: ./var/lib/lightdm-data/sysadmin: Cannot open: Permission denied
INFO: tar: ./var/lib/snapd/void: Cannot open: Permission denied
INFO: tar: ./var/lib/lightdm: Cannot open: Permission denied
INFO: tar: ./var/lib/postfix/master.lock: Cannot open: Permission denied
INFO: tar: ./var/lib/update-notifier/package-data-downloads/partial: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/evolution: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/nautilus/scripts: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/gvfs-metadata: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/recently-used.xbel: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/applications: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/sounds: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/keyrings: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.local/share/gnome-shell: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.xsession-errors: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.thunderbird: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.ssh: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.gnupg: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.mozilla: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.config: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.gvfs: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.ICEauthority: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.bash_history: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.Xauthority: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.dbus: Cannot open: Permission denied
INFO: tar: ./home/sysadmin/.cache: Cannot open: Permission denied
INFO: Total bytes written: 3886632960 (3.7GiB, 2.7MiB/s)
INFO: tar: Exiting with failure status due to previous errors
ERROR: Backup of VM 150 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/cloud_storage/dump/vzdump-lxc-150-2019_08_02-01_58_53.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/pve/cloud_storage/dump/vzdump-lxc-150-2019_08_02-01_58_53.tmp' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | lzop >/mnt/pve/cloud_storage/dump/vzdump-lxc-150-2019_08_02-01_58_53.tar.dat' failed: exit code 2
INFO: Backup job finished with errors

 

[nununo]

Having exactly the same problem here. Did you find anything? Thanks!

Update: I noticed that the PVEs were backed up correctly and only the VXC containers failed. Maybe I'll have to use another mode instead of snapshot. I'll give it a try late today.

 

[FunnyManVA]

I still haven't figured out why it copies the files over to the NFS fine, but has permission denied when trying to create the tar file for the backup on some containers. What's so strange is other containers on the same node work fine.

 

[sumshouse]

I am facing the same isse in 2021.

 

[lifeboy]

Indeed, how can it be that this happens to multiple people and no-one knows why? Surely someone at Proxmox knows?

It happens with a particular container on my side as well.

 

[lifeboy]

More particularly, this seems to have to do with the LXC file system. I have:

# ls -la /var/spool/
total 20
drwxr-xr-x  5 root   root    4096 Jan 24  2019 .
drwxr-xr-x 11 root   root    4096 Jan 24  2019 ..
drwxr-xr-x  3 root   root    4096 Jan 24  2019 cron
lrwxrwxrwx  1 root   root       7 Jan 24  2019 mail -> ../mail
drwxr-xr-x 20 root   root    4096 Dec 31  2019 postfix
drwx------  2 nobody nogroup 4096 Apr 24  2018 rsyslog

and the vzdump complains that it can't read /var/spool/rsyslog, which is of course true.

The problem is, one cannot change the ownership or modify the rights, even as root...

# chown root:root /var/spool/rsyslog
chown: changing ownership of '/var/spool/rsyslog': Operation not permitted

# chmod o+r /var/spool/rsyslog
chmod: changing permissions of '/var/spool/rsyslog': Operation not permitted

The container is running Ubuntu 18.04

 

[fabian]

the issue is the temp dir on NFS. fix that, and the problem should go away. if it does not. please post the full task log, container config and permissions of the relevant dirs from within the container.

 

[lifeboy]

The problem is described here: https://unix.stackexchange.com/questions/563942/debian-change-owner-of-nobodynogroup

The files listed as not accessible all belong to nobody:nogroup. Fix that and the backup will run fine.

Or backup from the command line with vzdump and use --exclude-path to skip the files that cannot be read.

 

[acdoussan]

If anyone else wanders their way in here and the file permissions look fine, and you don't want to use a local temp dir, take a look at this post:

https://blog.doussan.info/posts/container-backup-permission-denied-nfs/

TL;DR - you want to swap your nfs share to map all users, rather than just the root user. In ZFS, you can change this on the NFS share under advanced -> access -> set mapall user to your nfs user, remove maproot user (leave it empty).

 

[spratt]

the issue is the temp dir on NFS. fix that, and the problem should go away. if it does not. please post the full task log, container config and permissions of the relevant dirs from within the container.

If I'm understanding this correctly, the problem is because the tar operation is being done as user 100000 because it's an unprivileged container, and since that user doesn't have permission to read protected files (as far as the NFS server is concerned), then it returns permission denied, as it should.
However, this is a backup operation being initiated by the host. Why is the tar operation being run with the privileges of the guest container rather than being run entirely on the host?
Apologies if I'm misunderstanding the problem.

 

[fabian]

If I'm understanding this correctly, the problem is because the tar operation is being done as user 100000 because it's an unprivileged container, and since that user doesn't have permission to read protected files (as far as the NFS server is concerned), then it returns permission denied, as it should.
However, this is a backup operation being initiated by the host. Why is the tar operation being run with the privileges of the guest container rather than being run entirely on the host?
Apologies if I'm misunderstanding the problem.

because the backup needs to be done from the container's perspective, not the host's.

 

[spratt]

because the backup needs to be done from the container's perspective, not the host's.

Why? Does the host not have access to the container's data?

 

[fabian]

Why? Does the host not have access to the container's data?

obviously it does (one way or another), but the problem is exactly the inverse of the one you have - all the file and dir owners would be wrong in the backup archive, and not valid when restoring in a different context (e.g., unprivileged -> privileged, or when custom mappings are involved).

 

[spratt]

obviously it does (one way or another), but the problem is exactly the inverse of the one you have - all the file and dir owners would be wrong in the backup archive, and not valid when restoring in a different context (e.g., unprivileged -> privileged, or when custom mappings are involved).

So it's more complicated than simply using the raw UIDs and subtracting 100000 from them during the backup process?

 

[fabian]

So it's more complicated than simply using the raw UIDs and subtracting 100000 from them during the backup process?

yes, that is just the default mapping, it could be very different in practice. also manually mapping the owners (and possibly, ACLs) for each file/dir is not practical. there are some developments in that area (like id-mapped mounting) that might allow us to skip it, but since we'd then need to support both variants basically forever (not just for backup/restore, but for mounting volumes!), that is added complexity for very little gain other than this edge case.