[Solved] Issues with LXC container backup

[Vittorio]

Hi

I'm having issues backing up two LXC Containers.
I have other CT and one VM and they are backing up fine.

VMID	NAME	STATUS	TIME	SIZE	FILENAME
103	Influxdb-Grafana	FAILED	00:10:09	command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_103/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_103/' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | zstd --rsyncable '--threads=1' >/mnt/pve/NFS-QNAP-Proxmox-Backup/dump/vzdump-lxc-103-2021_02_19-01_04_28.tar.dat' failed: exit code 2
105	AdGuard	FAILED	00:05:07	command 'rsync --stats -h --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/2176/root//./ /mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_105/' failed: exit code 23

103: 2021-02-19 01:04:28 INFO: Starting Backup of VM 103 (lxc)
103: 2021-02-19 01:04:28 INFO: status = running
103: 2021-02-19 01:04:28 INFO: CT Name: Influxdb-Grafana
103: 2021-02-19 01:04:28 INFO: including mount point rootfs ('/') in backup
103: 2021-02-19 01:04:28 INFO: mode failure - some volumes do not support snapshots
103: 2021-02-19 01:04:28 INFO: trying 'suspend' mode instead
103: 2021-02-19 01:04:28 INFO: backup mode: suspend
103: 2021-02-19 01:04:28 INFO: ionice priority: 7
103: 2021-02-19 01:04:28 INFO: CT Name: Influxdb-Grafana
103: 2021-02-19 01:04:28 INFO: including mount point rootfs ('/') in backup
103: 2021-02-19 01:04:28 INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
103: 2021-02-19 01:04:28 INFO: starting first sync /proc/4704/root/ to /mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_103/
103: 2021-02-19 01:10:59 INFO: first sync finished - transferred 2.30G bytes in 391s
103: 2021-02-19 01:10:59 INFO: suspending guest
103: 2021-02-19 01:10:59 INFO: starting final sync /proc/4704/root/ to /mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_103/
103: 2021-02-19 01:11:10 INFO: final sync finished - transferred 44.03M bytes in 11s
103: 2021-02-19 01:11:10 INFO: resuming guest
103: 2021-02-19 01:11:10 INFO: guest is online again after 11 seconds
103: 2021-02-19 01:11:10 INFO: creating vzdump archive '/mnt/pve/NFS-QNAP-Proxmox-Backup/dump/vzdump-lxc-103-2021_02_19-01_04_28.tar.zst'
103: 2021-02-19 01:13:27 INFO: tar: ./var/cache/apt/archives/partial: Cannot open: Permission denied
103: 2021-02-19 01:13:32 INFO: tar: ./var/lib/apt/lists/partial: Cannot open: Permission denied
103: 2021-02-19 01:13:32 INFO: tar: ./var/lib/chronograf/backup: Cannot open: Permission denied
103: 2021-02-19 01:13:32 INFO: tar: ./var/lib/chronograf/chronograf-v1.db: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/grafana/plugins: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/grafana/png: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/grafana/grafana.db: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/influxdb/data/_internal: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/influxdb/data/homeassistant: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/influxdb/data/telegraf: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/influxdb/wal: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/kapacitor/kapacitor.db: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/lib/postfix/master.lock: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/grafana/grafana.log: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/grafana/grafana.log.2021-02-14.001: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/grafana/grafana.log.2021-02-15.001: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/grafana/grafana.log.2021-02-16.001: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/grafana/grafana.log.2021-02-17.001: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/grafana/grafana.log.2021-02-18.001: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/kapacitor/kapacitor.log: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/kapacitor/kapacitor.log-20210215.gz: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/kapacitor/kapacitor.log-20210216.gz: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/kapacitor/kapacitor.log-20210217.gz: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/kapacitor/kapacitor.log-20210218.gz: Cannot open: Permission denied
103: 2021-02-19 01:13:36 INFO: tar: ./var/log/kapacitor/kapacitor.log-20210219.gz: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/active: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/bounce: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/corrupt: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/defer: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/deferred: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/flush: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/hold: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/incoming: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/maildrop: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/private: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/public: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/saved: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: tar: ./var/spool/postfix/trace: Cannot open: Permission denied
103: 2021-02-19 01:13:37 INFO: Total bytes written: 1808834560 (1.7GiB, 12MiB/s)
103: 2021-02-19 01:13:37 INFO: tar: Exiting with failure status due to previous errors
103: 2021-02-19 01:14:37 ERROR: Backup of VM 103 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_103/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_103/' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | zstd --rsyncable '--threads=1' >/mnt/pve/NFS-QNAP-Proxmox-Backup/dump/vzdump-lxc-103-2021_02_19-01_04_28.tar.dat' failed: exit code 2

105: 2021-02-19 01:21:19 INFO: Starting Backup of VM 105 (lxc)
105: 2021-02-19 01:21:19 INFO: status = running
105: 2021-02-19 01:21:19 INFO: CT Name: AdGuard
105: 2021-02-19 01:21:19 INFO: including mount point rootfs ('/') in backup
105: 2021-02-19 01:21:19 INFO: mode failure - some volumes do not support snapshots
105: 2021-02-19 01:21:19 INFO: trying 'suspend' mode instead
105: 2021-02-19 01:21:19 INFO: backup mode: suspend
105: 2021-02-19 01:21:19 INFO: ionice priority: 7
105: 2021-02-19 01:21:19 INFO: CT Name: AdGuard
105: 2021-02-19 01:21:19 INFO: including mount point rootfs ('/') in backup
105: 2021-02-19 01:21:19 INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
105: 2021-02-19 01:21:19 INFO: starting first sync /proc/2176/root/ to /mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_105/
105: 2021-02-19 01:26:26 ERROR: Backup of VM 105 failed - command 'rsync --stats -h --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/2176/root//./ /mnt/pve/NFS-QNAP-Proxmox-Backup/tmp/vzdumptmp19655_105/' failed: exit code 23

In vzdump.conf I have
tmpdir: /mnt/pve/NFS-QNAP-Proxmox-Backup/tmp

What can I check?

Thanks

 

[mira]

Please provide the output of pveversion -v.

Have you tried setting tmpdir to a local directory instead of NFS? This way everything has to be sent over the network multiple times.

 

[Vittorio]

Thanks @mira

setting tmpdir to a local folder solved the backup of the CT 103, /mnt/storage2 is a second local SSD.

While for the 105 I still have an issue.

This is the result of pveversion -v

root@proxmox:/etc# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.55-1-pve: 5.4.55-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-3
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.8-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-5
pve-cluster: 6.2-1
pve-container: 3.3-3
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-1
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

And this is the error of the 105 backup...

INFO: starting new backup job: vzdump 105 --storage NFS-QNAP-Proxmox --compress zstd --mode suspend --node proxmox --remove 0
INFO: Starting Backup of VM 105 (lxc)
INFO: Backup started at 2021-02-19 14:44:43
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: AdGuard
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/2176/root/ to /mnt/storage2/tmp/vzdumptmp10869_105/
ERROR: Backup of VM 105 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/2176/root//./ /mnt/storage2/tmp/vzdumptmp10869_105/' failed: exit code 23
INFO: Failed at 2021-02-19 14:45:05
INFO: Backup job finished with errors
TASK ERROR: job errors

Thanks

 

[mira]

Error 23 is: Partial transfer due to error
This might be an issue with permissions. Is this an unprivileged container?
Are there any files in the container you can't access?

 

[Vittorio]

Yes, this is an unprivileged container

arch: amd64
cores: 1
cpuunits: 512
features: mount=fuse,nesting=1
hostname: AdGuard
memory: 512
nameserver: 1.0.0.1
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.10,hwaddr=22:A2:28:85:EA:A0,ip=192.168.1.120/24,type=veth
onboot: 1
ostype: debian
rootfs: local2:105/vm-105-disk-0.raw,size=10G
searchdomain: 1.1.1.1
startup: order=1
swap: 512
unprivileged: 1
lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file,optional
lxc.mount.auto: cgroup:rw

and yes, there are some files where I have permission denied messages

This a container with AdGuard Home and was not so easy to install, it uses squashfs.
I tried to install via snapd, but it wasn't possible, so I installed via curl.

At the end, it works fine now.

Thanks

 

[Vittorio]

I've now moved to a new CT with Docker and Adguard in it and Backup it's working fine.

Thanks

 

[mira]

Glad it is now working fine.

Please consider marking the thread as 'Solved'. You can do so by pressing 'Edit thread' above the first post and selecting 'Solved'.

 

[kalasnikov]

I've now moved to a new CT with Docker and Adguard in it and Backup it's working fine.

Thanks

You created a brand new CT?

 

[Vittorio]

You created a brand new CT?

Yes, I've created a new CT.
In any case, I've found also someone who created an LXC container already with Adguard.
You can find it here.