Backup LXC Containers fail

[dsimoes]

Hi everyone.

I'm running Proxmox 6.4 with some LXC containers and a couple of QVM's.

One LXC Container is failing to backup to my Synology NAS (NFS). I don't understand why.

Here are the logs for the LXC that fails:

INFO: starting new backup job: vzdump 100 --compress zstd --storage SynologyNAS --mode snapshot --remove 0 --node proxmox
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2021-11-18 11:24:27
INFO: status = running
INFO: CT Name: Koala
INFO: including mount point rootfs ('/') in backup
INFO: excluding volume mount point mp1 ('/mnt/media') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
Logical volume "snap_vm-100-disk-0_vzdump" created.
/dev/sdc: open failed: No medium found
/dev/sdc: open failed: No medium found
INFO: creating vzdump archive '/mnt/pve/SynologyNAS/dump/vzdump-lxc-100-2021_11_18-11_24_27.tar.zst'
INFO: tar: ./var/cache/private: Cannot open: Permission denied
INFO: tar: ./var/lib/private: Cannot open: Permission denied
INFO: tar: ./var/log/mail.err: Cannot open: Permission denied
INFO: tar: ./var/log/btmp: Cannot open: Permission denied
INFO: tar: ./var/log/private: Cannot open: Permission denied
INFO: Total bytes written: 1660692480 (1.6GiB, 30MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-100-disk-0_vzdump" successfully removed
/dev/sdc: open failed: No medium found
ERROR: Backup of VM 100 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/SynologyNAS/dump/vzdump-lxc-100-2021_11_18-11_24_27.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/SynologyNAS/dump/vzdump-lxc-100-2021_11_18-11_24_27.tar.dat' failed: exit code 2
INFO: Failed at 2021-11-18 11:25:24
INFO: Backup job finished with errors
TASK ERROR: job errors

Is it failing because of the Permission denied errors? If so, what's wrong with this particular LXC that has these files/dirs with no read permissions?
When I ssh to the LXC and try to sudo access the files I still can't read them. I'm a bit lost. Here is a screenshot of the /var/log directory.

Thanks for the help.

 

[Moayad]

Hi,

INFO: tar: ./var/cache/private: Cannot open: Permission denied
INFO: tar: ./var/lib/private: Cannot open: Permission denied
INFO: tar: ./var/log/mail.err: Cannot open: Permission denied
INFO: tar: ./var/log/btmp: Cannot open: Permission denied
INFO: tar: ./var/log/private: Cannot open: Permission denied

It indicates that you do not have written permission for this directory with the unprivileged container (unprivileged containers use an assigned "userID" on the host, and backups for containers are made in the context of this "userID").

To fix this, you can edit the "/etc/vzdump.conf" and set the "tmpdir" to e.g. "/var/tmp/".

vi /etc/vzdump.conf
# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR

to:
# vzdump default settings

tmpdir:  /var/tmp/
#tmpdir: DIR
#dumpdir: DIR

 

[dsimoes]

Hi,

Thank you for your quick answer.

I edited vzdump.conf and uncommented/pointed tmpdir to /var/tmp.
Then I went to interface and initialized a new Backup. I got to see a temp file being created on /var/tmp for that LXC but the backup still failed:

INFO: creating vzdump archive '/mnt/pve/SynologyNAS/dump/vzdump-lxc-100-2021_11_18-13_27_29.tar.zst'
INFO: tar: ./var/cache/private: Cannot open: Permission denied
INFO: tar: ./var/lib/private: Cannot open: Permission denied
INFO: tar: ./var/log/mail.err: Cannot open: Permission denied
INFO: tar: ./var/log/btmp: Cannot open: Permission denied
INFO: tar: ./var/log/private: Cannot open: Permission denied
INFO: Total bytes written: 1660631040 (1.6GiB, 30MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-100-disk-0_vzdump" successfully removed
/dev/sdc: open failed: No medium found
ERROR: Backup of VM 100 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp9820_100/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/SynologyNAS/dump/vzdump-lxc-100-2021_11_18-13_27_29.tar.dat' failed: exit code 2
INFO: Failed at 2021-11-18 13:28:26
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[Moayad]

Could you please post the container config pct config 100 and the output of pveversion -v

 

[dsimoes]

Sure:
pct config 100:

arch: amd64
cores: 2
hostname: Koala
memory: 1024
mp1: data-hdd:vm-100-disk-0,mp=/mnt/media,acl=0,size=150G
nameserver: 192.168.1.3 8.8.8.8
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=6E:7E:A6:65:43:24,ip=192.168.1.22/24,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-100-disk-0,size=8G
startup: order=4
swap: 1024
unprivileged: 1

pveversion -v:

proxmox-ve: 6.4-1 (running kernel: 5.4.119-1-pve)
pve-manager: 6.4-9 (running version: 6.4-9/5f5c0e3f)
pve-kernel-5.4: 6.4-3
pve-kernel-helper: 6.4-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-6
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

 

[gridiron]

Bumping this thread as I appear to be running into the same issue.

Error logs:

INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp991849_103 for temporary files
INFO: Starting Backup of VM 103 (lxc)
INFO: Backup started at 2022-01-24 04:00:55
INFO: status = running
INFO: CT Name: jellyfin
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/movies') from backup (not a volume)
INFO: excluding bind mount point mp1 ('/tvshows') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating vzdump archive '/tank/backups/pve/dump/vzdump-lxc-103-2022_01_24-04_00_55.tar.zst'
INFO: tar: ./var/cache/private: Cannot open: Permission denied
INFO: tar: ./var/lib/private: Cannot open: Permission denied
INFO: tar: ./var/log/private: Cannot open: Permission denied
INFO: tar: ./var/log/mail.err: Cannot open: Permission denied
INFO: tar: ./var/log/btmp: Cannot open: Permission denied
INFO: Total bytes written: 1373440000 (1.3GiB, 105MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 103 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:44 -m g:44:44:1 -m g:45:100045:60 -m g:105:103:1 -m g:106:100106:65430 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp991849_103' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/tank/backups/pve/dump/vzdump-lxc-103-2022_01_24-04_00_55.tar.dat' failed: exit code 2
INFO: Failed at 2022-01-24 04:01:08

My container config:

arch: amd64
cores: 4
features: nesting=1
hostname: jellyfin
memory: 2048
mp0: /tank/media/movies,mp=/movies,ro=1
mp1: /tank/media/tvshows,mp=/tvshows,ro=1
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.2.1,hwaddr=<removed>,ip=192.168.2.103/24,type=veth
onboot: 1
ostype: debian
rootfs: local-zfs:subvol-103-disk-0,mountoptions=noatime,size=32G
swap: 512
unprivileged: 1
lxc.cgroup2.devices.allow: c 226:1 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/card1 dev/dri/card1 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 60
lxc.idmap: g 105 103 1
lxc.idmap: g 106 100106 65430

My PVE version:

proxmox-ve: 7.1-1 (running kernel: 5.15.7-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-5.15: 7.1-8
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.15.12-1-pve: 5.15.12-3
pve-kernel-5.15.7-1-pve: 5.15.7-1
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.3-1
proxmox-backup-file-restore: 2.1.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

The GID mapping was put in place to allow ffmpeg to use the /dev/dri/renderD128 device on the host for HW-accelerated transcoding. The transcoding works, but now the automated backups are failing.

Any ideas?

 

[fabian]

if the files are showing as user/group nobody inside the container, then they are 'outside' of the uid/gid mapping. you either need to give them correctly mapped owners (from the host namespace, e.g. using 'pct mount') or adjust the mapping so that it covers whatever uid/gid is missing there..

pct mount and then ls -ln /path for each of the paths should give you more information (the numeric IDs).

 

[gridiron]

Thanks Fabian.

Here is the output of ls -ln on the paths in question:

root@jellyfin:~# ls -ln /var/cache /var/lib /var/log
/var/cache:
total 19
drwxr-xr-x  3     0     0  5 Jan 25 06:38 apt
drwxr-xr-x  2     0     0  7 Jan  8 23:45 debconf
drwxr-x---  6   107     4  6 Jan 16 01:54 jellyfin
drwx------  2     0     0  3 Jan 25 06:38 ldconfig
drwxr-xr-x 26 65534 65534 28 Jan 23 13:47 man
drwx------  2 65534 65534  2 Jan  8 15:08 private

/var/lib:
total 58
drwxr-xr-x 5     0     0  8 Jan 25 06:38 apt
drwxr-xr-x 2     0     0  3 Jan  8 15:08 dbus
drwxr-xr-x 2     0     0  2 May 27  2021 dhcp
drwxr-xr-x 3     0     0  3 Aug 16 05:36 dictionaries-common
drwxr-xr-x 7     0     0 17 Jan 25 06:38 dpkg
drwxr-x--- 9   107     4  9 Jan  9 13:54 jellyfin
drwxr-xr-x 2     0     0  3 Jan 25 00:00 logrotate
drwxr-xr-x 2     0     0  3 Aug 16 05:36 man-db
drwxr-xr-x 2     0     0  2 Apr 10  2021 misc
drwxr-xr-x 2     0     0  8 Jan  8 15:10 pam
drwxr-xr-x 3     0     0  3 Aug 16 05:35 polkit-1
drwxr-xr-x 2   101   106  3 Jan  8 15:08 postfix
drwx------ 2 65534 65534  2 Jan  8 15:08 private
drwxr-xr-x 2     0     0  3 Aug 16 05:36 python
drwxr-xr-x 3     0     0  3 Jan 10 10:15 sudo
drwxr-xr-x 9 65534 65534  9 Jan  8 15:11 systemd
drwxr-xr-x 3     0     0 14 Jan  8 23:45 ucf
drwxr-xr-x 2     0     0  2 Feb 19  2021 unattended-upgrades
drwxr-xr-x 3     0     0  3 Aug 16 05:34 vim
drwxr-xr-x 2     0     0  2 Feb 28  2020 xkb

/var/log:
total 6821
-rw-r--r--  1     0     0    15517 Jan 25 06:37 alternatives.log
drwxr-xr-x  2     0     0        5 Jan 25 06:37 apt
-rw-r-----  1     0     4   832522 Jan 25 09:27 auth.log
-rw-r-----  1     0     4     8862 Jan 23 13:47 auth.log.1
-rw-r-----  1     0     4     5557 Jan 15 23:03 auth.log.2.gz
-rw-rw----  1 65534 65534     3072 Jan 13 09:31 btmp
-rw-r-----  1     0     4 35134294 Jan 25 09:27 daemon.log
-rw-r-----  1     0     4    17558 Jan 23 13:47 daemon.log.1
-rw-r-----  1     0     4    31405 Jan 16 00:00 daemon.log.2.gz
-rw-r--r--  1     0     0   275450 Jan 25 06:38 dpkg.log
-rw-r--r--  1     0     0    32032 Jan 10 10:15 faillog
drwxr-x---  2   107     4       83 Jan 25 09:21 jellyfin
drwxr-sr-x+ 3 65534 65534        3 Jan  8 15:08 journal
-rw-rw-r--  1 65534 65534   292292 Jan 13 20:14 lastlog
-rw-r-----  1 65534 65534      204 Jan 23 16:55 mail.err
-rw-r-----  1     0     4     4854 Jan 25 09:19 mail.info
-rw-r-----  1     0     4      281 Jan 13 15:36 mail.info.2.gz
-rw-r-----  1     0     4     4854 Jan 25 09:19 mail.log
-rw-r-----  1     0     4      281 Jan 13 15:36 mail.log.2.gz
-rw-r-----  1     0     4     1959 Jan 25 09:19 mail.warn
-rw-r-----  1     0     4      167 Jan 13 15:36 mail.warn.2.gz
-rw-r-----  1     0     4     1998 Jan 25 09:19 messages
-rw-r-----  1     0     4      149 Jan 16 00:10 messages.1
-rw-r-----  1     0     4      250 Jan 13 15:36 messages.2.gz
drwx------  2 65534 65534        2 Jan  8 15:08 private
drwxr-xr-x  3     0     0        3 Aug 16 05:35 runit
-rw-r--r--  1     0     0 35148790 Jan 25 09:27 syslog
-rw-r--r--  1     0     0    22205 Jan 23 13:47 syslog.1
-rw-r--r--  1     0     0    34977 Jan 16 00:00 syslog.2.gz
drwxr-x---  2     0     4        5 Jan  9 06:16 unattended-upgrades
-rw-r-----  1     0     4      495 Jan 25 09:19 user.log
-rw-r-----  1     0     4      135 Jan 13 15:36 user.log.2.gz
-rw-rw-r--  1 65534 65534    66048 Jan 23 16:55 wtmp

So, it looks like 65534:65534 is the UID and GID of the directories causing issues. Do these need to be remapped in the lxc config file as well? If so, to what on the host?

 

[gridiron]

To close the loop on this, I ended up restoring from a backup made before the lxc.idmap: ... lines were added to the 103.conf file. After re-adding the lines and rebooting the container, the backups now run without issue.

 

[Michael.Uray]

I think I am having the same issue, I end up in an error when I try to run a backup:

INFO: starting new backup job: vzdump 110 --notes-template '{{guestname}}' --remove 0 --mode stop --storage local --compress zstd --node pve1
INFO: Starting Backup of VM 110 (lxc)
INFO: Backup started at 2023-07-03 13:03:56
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: vpn
INFO: including mount point rootfs ('/') in backup
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-110-2023_07_03-13_03_56.tar.zst'
INFO: tar: ./var/log/btmp: Cannot open: Permission denied
INFO: tar: ./var/log/mail.err: Cannot open: Permission denied
INFO: tar: ./var/log/kern.log: Cannot open: Permission denied
INFO: tar: ./var/log/private: Cannot open: Permission denied
INFO: tar: ./var/cache/private: Cannot open: Permission denied
INFO: tar: ./var/lib/private: Cannot open: Permission denied
INFO: Total bytes written: 1753088000 (1.7GiB, 67MiB/s)
INFO: tar: Exiting with failure status due to previous errors
ERROR: Backup of VM 110 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp2036673_110/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/var/lib/vz/dump/vzdump-lxc-110-2023_07_03-13_03_56.tar.dat' failed: exit code 2
INFO: Failed at 2023-07-03 13:04:22
INFO: Backup job finished with errors
TASK ERROR: job errors

root@pve1:~# ls -ln /var/lib/private
total 0
root@pve1:~# ls -ln \
/var/log/btmp \
/var/log/mail.err \
/var/log/kern.log \
/var/log/private \
/var/cache/private \
/var/lib/private
ls: cannot access '/var/log/mail.err': No such file or directory
-rw-rw---- 1 0 43     384 Jul  2 13:27 /var/log/btmp
-rw-r----- 1 0  4 2096292 Jul  3 13:14 /var/log/kern.log

/var/cache/private:
total 0

/var/lib/private:
total 0

/var/log/private:
total 0

Any ideas how to solve that?

 

[fabian]

check with "pct mount 110" what the actual ownership is of the dirs in question..

 

[Michael.Uray]

You mean like this?

root@pve1:~# pct mount 110
mounted CT 110 in '/var/lib/lxc/110/rootfs'

root@pve1:/var/lib/lxc/110/rootfs# ls -ln \
./var/log/btmp \
./var/log/mail.err \
./var/log/kern.log
-rw-rw---- 1 0 43 2688 Feb 27  2021 ./var/log/btmp
-rw-r----- 1 0  4 6482 Feb 27  2021 ./var/log/kern.log
-rw-r----- 1 0  4  392 Feb 27  2021 ./var/log/mail.err

root@pve1:/var/lib/lxc/110/rootfs# ls -l
total 92
drwxr-xr-x  2 100000 100000  4096 Apr 24  2022 bin
drwxr-xr-x  2 100000 100000  4096 May 13  2019 boot
drwxr-xr-x  2 100000 100000  4096 Dec 10  2008 dev
drwxr-xr-x 78 100000 100000  4096 Jul  2 14:21 etc
drwxr-xr-x  2 100000 100000  4096 May 13  2019 home
drwxr-xr-x 10 100000 100000  4096 Jul  8  2019 lib
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 lib64
drwx------  2 root   root   16384 Feb 27  2021 lost+found
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 media
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 mnt
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 opt
drwxr-xr-x  2 100000 100000  4096 May 13  2019 proc
drwx------  4 100000 100000  4096 Dec 12  2022 root
drwxr-xr-x  3 100000 100000  4096 Jul  8  2019 run
drwxr-xr-x  2 100000 100000  4096 Mar 28  2021 sbin
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 srv
drwxr-xr-x  2 100000 100000  4096 May 13  2019 sys
drwxrwxrwt  7 root   root    4096 Jul  3 12:59 tmp
drwxr-xr-x 10 100000 100000  4096 Jul  8  2019 usr
drwxr-xr-x 12 100000 100000  4096 Feb 27  2021 var

root@pve1:/var/lib/lxc/110/rootfs/var/cache# ls -ln
total 24
drwxr-xr-x  3 100000 100000 4096 Feb 27  2021 apache2
drwxr-xr-x  3 100000 100000 4096 Apr 24  2022 apt
drwxr-xr-x  2 100000 100000 4096 Mar 26  2021 debconf
drwx------  2 100000 100000 4096 Mar 26  2021 ldconfig
drwxr-xr-x 32      6     12 4096 Feb 27  2021 man
drwx------  2      0      0 4096 Feb 27  2021 private

root@pve1:/var/lib/lxc/110/rootfs/var/lib# ls -ln
total 92
drwxr-xr-x 5 100000 100000 4096 Feb 27  2021 apache2
drwxr-xr-x 5 100000 100000 4096 Apr 24  2022 apt
drwxr-xr-x 2 100000 100000 4096 Jul  8  2019 dbus
drwxr-xr-x 2 100000 100000 4096 Dec 11  2018 dhcp
drwxr-xr-x 3 100000 100000 4096 Jul  8  2019 dictionaries-common
drwxr-xr-x 7 100000 100000 4096 Apr 24  2022 dpkg
drwxr-xr-x 2 100000 100000 4096 Feb 27  2021 fail2ban
drwxr-xr-x 2 100107 100115 4096 Jul  2 14:21 ldap
drwxr-xr-x 2 100000 100000 4096 Jul  3 02:00 logrotate
drwxr-xr-x 2 100000 100000 4096 Jul  8  2019 man-db
drwxr-xr-x 2 100000 100000 4096 May 13  2019 misc
drwxr-xr-x 2 100000 100000 4096 Feb 27  2021 pam
drwxr-xr-x 4 100000 100000 4096 Feb 27  2021 php
drwxr-xr-x 3 100000 100000 4096 Jul  8  2019 polkit-1
drwxr-xr-x 2 100101 100107 4096 Feb 27  2021 postfix
drwx------ 2      0      0 4096 Feb 27  2021 private
drwxr-xr-x 2 100000 100000 4096 Jul  8  2019 python
drwxr-xr-x 2 100000 100000 4096 Feb 14  2021 slapd
drwxr-xr-x 3 100000 100000 4096 Feb 27  2021 sudo
drwxr-xr-x 6      0      0 4096 Feb 27  2021 systemd
drwxr-xr-x 3 100000 100000 4096 Mar  1  2021 ucf
drwxr-xr-x 3 100000 100000 4096 Jul  8  2019 vim
drwxr-xr-x 2 100000 100000 4096 Apr 30  2018 xkb

 

[fabian]

yes. so you can see a few dirs with wrong ownership, except lost+found nothing should be owned by the host root user '0'. you need to shift those by 100000 , unless you did some custom mapping, in that case you need to provide more information

 

[Michael.Uray]

To run

root@pve1:/var/lib/lxc/110/rootfs/var/log# chown 100000 *

on these directory fix that and the backup is working now, thank you very much Fabian.

root@pve1:/var/lib/lxc/110/rootfs/var/log# ls -ln
total 86500
-rw-r--r-- 1 100000 100000        0 Mar  1  2021 alternatives.log
-rw-r--r-- 1 100000 100000    19664 Feb 27  2021 alternatives.log.1
drwxr-x--- 2 100000 100004     4096 Jul 14 02:00 apache2
drwxr-xr-x 2 100000 100000     4096 Jul  4 02:00 apt
-rw-r----- 1 100000 100004 72689480 Jul 14 07:59 auth.log
-rw-rw---- 1 100000     43     2688 Feb 27  2021 btmp
-rw-r----- 1 100000 100004 12404734 Jul 14 07:39 daemon.log
-rw-r----- 1 100000 100004  2370330 Jul 14 07:59 debug
-rw-r--r-- 1 100000 100000        0 Jul  4 02:00 dpkg.log
-rw-r--r-- 1 100000 100000     1570 Jul  3 20:59 dpkg.log.1
-rw-r--r-- 1 100000 100000      175 Apr 24  2022 dpkg.log.2.gz
-rw-r--r-- 1 100000 100000      298 Dec 13  2021 dpkg.log.3.gz
-rw-r--r-- 1 100000 100000     1211 Mar 28  2021 dpkg.log.4.gz
-rw-r--r-- 1 100000 100000    28465 Feb 27  2021 dpkg.log.5.gz
-rw-r----- 1 100000 100004   215726 Jul 14 07:49 fail2ban.log
-rw-r----- 1 100000 100004    80546 Jul  9 00:44 fail2ban.log.1
-rw-r----- 1 100000 100004    15381 Jul  2 01:42 fail2ban.log.2.gz
-rw-r----- 1 100000 100004    13908 Jun 25 01:54 fail2ban.log.3.gz
-rw-r----- 1 100000 100004    23151 Jun 18 01:50 fail2ban.log.4.gz
-rw-r--r-- 1 100000 100000     3456 Feb 27  2021 faillog
-rw-r----- 1      0      4     6482 Feb 27  2021 kern.log
-rw-rw-r-- 1      0     43    31536 Feb 27  2021 lastlog
-rw-r----- 1      0      4      392 Feb 27  2021 mail.err
-rw-r----- 1 100000 100004    73397 Jul  3 19:44 mail.info
-rw-r----- 1 100000 100004    73397 Jul  3 19:44 mail.log
-rw-r----- 1 100000 100004    10477 Jul  3 19:44 mail.warn
-rw-r----- 1 100000 100004   149696 Jul 14 02:00 messages
drwxr-xr-x 2 100000 100000     4096 Feb 27  2021 openvpn
drwx------ 2      0      0     4096 Feb 27  2021 private
-rw-r--r-- 1 100000 100000    32158 Jul 14 07:59 syslog
-rw-r--r-- 1 100000 100000   131603 Jul 14 02:00 syslog.1
-rw-r--r-- 1 100000 100000     8209 Jul 13 02:00 syslog.2.gz
-rw-r--r-- 1 100000 100000     8377 Jul 12 02:00 syslog.3.gz
-rw-r--r-- 1 100000 100000     8404 Jul 11 02:00 syslog.4.gz
-rw-r--r-- 1 100000 100000     8238 Jul 10 02:00 syslog.5.gz
-rw-r--r-- 1 100000 100000     8019 Jul  9 02:00 syslog.6.gz
-rw-r--r-- 1 100000 100000     8022 Jul  8 02:00 syslog.7.gz
drwxr-x--- 2 100000 100004     4096 Jul  3 02:00 unattended-upgrades
-rw-r----- 1 100000 100004     5924 Jul  3 19:44 user.log
-rw-rw-r-- 1      0     43    12672 Feb 27  2021 wtmp

root@pve1:/var/lib/lxc/110/rootfs/var/log# chown 100000 *

root@pve1:/var/lib/lxc/110/rootfs/var/log# ls -ln
total 86500
-rw-r--r-- 1 100000 100000        0 Mar  1  2021 alternatives.log
-rw-r--r-- 1 100000 100000    19664 Feb 27  2021 alternatives.log.1
drwxr-x--- 2 100000 100004     4096 Jul 14 02:00 apache2
drwxr-xr-x 2 100000 100000     4096 Jul  4 02:00 apt
-rw-r----- 1 100000 100004 72691263 Jul 14 08:00 auth.log
-rw-rw---- 1 100000     43     3456 Jul 14 08:00 btmp
-rw-r----- 1 100000 100004 12404734 Jul 14 07:39 daemon.log
-rw-r----- 1 100000 100004  2370504 Jul 14 08:00 debug
-rw-r--r-- 1 100000 100000        0 Jul  4 02:00 dpkg.log
-rw-r--r-- 1 100000 100000     1570 Jul  3 20:59 dpkg.log.1
-rw-r--r-- 1 100000 100000      175 Apr 24  2022 dpkg.log.2.gz
-rw-r--r-- 1 100000 100000      298 Dec 13  2021 dpkg.log.3.gz
-rw-r--r-- 1 100000 100000     1211 Mar 28  2021 dpkg.log.4.gz
-rw-r--r-- 1 100000 100000    28465 Feb 27  2021 dpkg.log.5.gz
-rw-r----- 1 100000 100004   216059 Jul 14 08:00 fail2ban.log
-rw-r----- 1 100000 100004    80546 Jul  9 00:44 fail2ban.log.1
-rw-r----- 1 100000 100004    15381 Jul  2 01:42 fail2ban.log.2.gz
-rw-r----- 1 100000 100004    13908 Jun 25 01:54 fail2ban.log.3.gz
-rw-r----- 1 100000 100004    23151 Jun 18 01:50 fail2ban.log.4.gz
-rw-r--r-- 1 100000 100000     3456 Feb 27  2021 faillog
-rw-r----- 1 100000      4     6482 Feb 27  2021 kern.log
-rw-rw-r-- 1 100000     43    31536 Feb 27  2021 lastlog
-rw-r----- 1 100000      4      392 Feb 27  2021 mail.err
-rw-r----- 1 100000 100004    73397 Jul  3 19:44 mail.info
-rw-r----- 1 100000 100004    73397 Jul  3 19:44 mail.log
-rw-r----- 1 100000 100004    10477 Jul  3 19:44 mail.warn
-rw-r----- 1 100000 100004   149696 Jul 14 02:00 messages
drwxr-xr-x 2 100000 100000     4096 Feb 27  2021 openvpn
drwx------ 2 100000      0     4096 Feb 27  2021 private
-rw-r--r-- 1 100000 100000    32332 Jul 14 08:00 syslog
-rw-r--r-- 1 100000 100000   131603 Jul 14 02:00 syslog.1
-rw-r--r-- 1 100000 100000     8209 Jul 13 02:00 syslog.2.gz
-rw-r--r-- 1 100000 100000     8377 Jul 12 02:00 syslog.3.gz
-rw-r--r-- 1 100000 100000     8404 Jul 11 02:00 syslog.4.gz
-rw-r--r-- 1 100000 100000     8238 Jul 10 02:00 syslog.5.gz
-rw-r--r-- 1 100000 100000     8019 Jul  9 02:00 syslog.6.gz
-rw-r--r-- 1 100000 100000     8022 Jul  8 02:00 syslog.7.gz
drwxr-x--- 2 100000 100004     4096 Jul  3 02:00 unattended-upgrades
-rw-r----- 1 100000 100004     5924 Jul  3 19:44 user.log
-rw-rw-r-- 1 100000     43    12672 Feb 27  2021 wtmp

I actually can't remember that I did change the ownership of these files.
Does the group membership also need to get changed? There are some files with group 0.

 

[fabian]

yes, if something is relying on group-based access

 

[Michael.Uray]

This fixed the issue with the user and group ownership:

Show current ownership of directoy:

root@pve1:/var/lib/lxc/110/rootfs# ls -ln
total 92
drwxr-xr-x  2 100000 100000  4096 Aug  6 11:10 bin
drwxr-xr-x  2 100000 100000  4096 May 13  2019 boot
drwxr-xr-x  2 100000 100000  4096 Dec 10  2008 dev
drwxr-xr-x 80 100000 100000  4096 Aug  6 11:08 etc
drwxr-xr-x  2 100000 100000  4096 May 13  2019 home
drwxr-xr-x 11 100000 100000  4096 Aug  6 11:07 lib
drwxr-xr-x  2 100000 100000  4096 Aug  6 11:07 lib64
drwx------  2      0      0 16384 Feb 27  2021 lost+found
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 media
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 mnt
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 opt
drwxr-xr-x  2 100000 100000  4096 May 13  2019 proc
drwx------  4 100000 100000  4096 Aug  6 11:05 root
drwxr-xr-x  3 100000 100000  4096 Jul  8  2019 run
drwxr-xr-x  2 100000 100000  4096 Aug  6 11:08 sbin
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 srv
drwxr-xr-x  2 100000 100000  4096 May 13  2019 sys
drwxrwxrwt  9      0      0  4096 Aug  6 11:09 tmp
drwxr-xr-x 10 100000 100000  4096 Jul  8  2019 usr
drwxr-xr-x 12 100000 100000  4096 Feb 27  2021 var

Find files and directories of all users with ID 0 and list them:

root@pve1:/var/lib/lxc/110/rootfs# find . -user 0
./lost+found
./tmp
./lib/systemd/system/container-getty@.service

Find files and directories of all users with ID 0 and change its ownership

root@pve1:/var/lib/lxc/110/rootfs# find . -user 0 -exec chown 100000:100000 {} \;

Check new ownership:

root@pve1:/var/lib/lxc/110/rootfs# ls -ln
total 92
drwxr-xr-x  2 100000 100000  4096 Aug  6 11:30 bin
drwxr-xr-x  2 100000 100000  4096 May 13  2019 boot
drwxr-xr-x  2 100000 100000  4096 Dec 10  2008 dev
drwxr-xr-x 80 100000 100000  4096 Aug  6 11:08 etc
drwxr-xr-x  2 100000 100000  4096 May 13  2019 home
drwxr-xr-x 11 100000 100000  4096 Aug  6 11:07 lib
drwxr-xr-x  2 100000 100000  4096 Aug  6 11:07 lib64
drwx------  2 100000 100000 16384 Feb 27  2021 lost+found
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 media
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 mnt
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 opt
drwxr-xr-x  2 100000 100000  4096 May 13  2019 proc
drwx------  4 100000 100000  4096 Aug  6 11:05 root
drwxr-xr-x  3 100000 100000  4096 Jul  8  2019 run
drwxr-xr-x  2 100000 100000  4096 Aug  6 11:08 sbin
drwxr-xr-x  2 100000 100000  4096 Jul  8  2019 srv
drwxr-xr-x  2 100000 100000  4096 May 13  2019 sys
drwxrwxrwt  9 100000 100000  4096 Aug  6 11:30 tmp
drwxr-xr-x 10 100000 100000  4096 Jul  8  2019 usr
drwxr-xr-x 12 100000 100000  4096 Feb 27  2021 var

What could actually lead to this situation?
I think this issue appeared after a PVE upgrade.

 

[Repave2544]

Hi,




It indicates that you do not have written permission for this directory with the unprivileged container (unprivileged containers use an assigned "userID" on the host, and backups for containers are made in the context of this "userID").

To fix this, you can edit the "/etc/vzdump.conf" and set the "tmpdir" to e.g. "/var/tmp/".

vi /etc/vzdump.conf
# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR

to:
# vzdump default settings

tmpdir:  /var/tmp/
#tmpdir: DIR
#dumpdir: DIR

This worked for me thanks!