Problem with backups of LXC Containers

[dthompson]

Hi all,

I am all of a sudden having an issue with backing up LXC containers on my nodes. I have a couple of servers in LXC format and since a couple of days ago they are all displaying issues when running a backup. Haven't run any updates in a bit, but the nodes also show that there are no updates available so I do believe I'm all up to date there.

The LXC container are running on 10GB NFS mounts and are backed up to another 10GB NFS Mount. This is the error when a backup is triggered (either by a schedule or manually)

Virtual Environment 6.1-8
Container 107 (ns1.digidns.ca) on node 'vdc4'
Logs
INFO: starting new backup job: vzdump 107 --compress lzo --mode snapshot --remove 0 --storage vdc-backup --node vdc4
INFO: Starting Backup of VM 107 (lxc)
INFO: Backup started at 2020-03-21 08:04:29
INFO: status = running
INFO: CT Name: ns1.digidns.ca
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: ns1.digidns.ca
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/10059/root// to /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp
INFO: Number of files: 28,106 (reg: 21,878, dir: 2,426, link: 3,802)
INFO: Number of created files: 28,105 (reg: 21,878, dir: 2,425, link: 3,802)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 21,860
INFO: Total file size: 961,455,660 bytes
INFO: Total transferred file size: 826,200,991 bytes
INFO: Literal data: 826,200,991 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 1,273,414
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 827,989,273
INFO: Total bytes received: 441,787
INFO: sent 827,989,273 bytes received 441,787 bytes 19,492,495.53 bytes/sec
INFO: total size is 961,455,660 speedup is 1.16
INFO: first sync finished (42 seconds)
INFO: suspend vm
INFO: starting final sync /proc/10059/root// to /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp
INFO: Number of files: 28,106 (reg: 21,878, dir: 2,426, link: 3,802)
INFO: Number of created files: 0
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 0
INFO: Total file size: 961,455,660 bytes
INFO: Total transferred file size: 0 bytes
INFO: Literal data: 0 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 0
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 652,787
INFO: Total bytes received: 2,612
INFO: sent 652,787 bytes received 2,612 bytes 262,159.60 bytes/sec
INFO: total size is 961,455,660 speedup is 1,466.98
INFO: final sync finished (2 seconds)
INFO: resume vm
INFO: guest is online again after 3 seconds
INFO: creating archive '/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tar.lzo'
INFO: tar: /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 107 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | lzop >/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tar.dat' failed: exit code 2
INFO: Failed at 2020-03-21 08:05:19
INFO: Backup job finished with errors
TASK ERROR: job errors

Anyone have any ideas as to why the backup is fialing all of a sudden. I can see in the error log that its complaining about permissions, yet on the node, I am able to read / write to the dump location without issue.

INFO: tar: /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp: Cannot open: Permission denied

Thanks

 

[Dominic]

Hi!

Could you please also post the following

pveversion -v
pct config 107
cat /etc/pve/storage.cfg
cat /etc/vzdump.conf

and try setting your temporary directory to something local?