Problem with backups of LXC Containers

dthompson

Well-Known Member
Nov 23, 2011
146
14
58
Canada
www.digitaltransitions.ca
Hi all,

I am all of a sudden having an issue with backing up LXC containers on my nodes. I have a couple of servers in LXC format and since a couple of days ago they are all displaying issues when running a backup. Haven't run any updates in a bit, but the nodes also show that there are no updates available so I do believe I'm all up to date there.

The LXC container are running on 10GB NFS mounts and are backed up to another 10GB NFS Mount. This is the error when a backup is triggered (either by a schedule or manually)

Code:
Virtual Environment 6.1-8
Container 107 (ns1.digidns.ca) on node 'vdc4'
Logs
INFO: starting new backup job: vzdump 107 --compress lzo --mode snapshot --remove 0 --storage vdc-backup --node vdc4
INFO: Starting Backup of VM 107 (lxc)
INFO: Backup started at 2020-03-21 08:04:29
INFO: status = running
INFO: CT Name: ns1.digidns.ca
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: ns1.digidns.ca
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/10059/root// to /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp
INFO: Number of files: 28,106 (reg: 21,878, dir: 2,426, link: 3,802)
INFO: Number of created files: 28,105 (reg: 21,878, dir: 2,425, link: 3,802)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 21,860
INFO: Total file size: 961,455,660 bytes
INFO: Total transferred file size: 826,200,991 bytes
INFO: Literal data: 826,200,991 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 1,273,414
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 827,989,273
INFO: Total bytes received: 441,787
INFO: sent 827,989,273 bytes received 441,787 bytes 19,492,495.53 bytes/sec
INFO: total size is 961,455,660 speedup is 1.16
INFO: first sync finished (42 seconds)
INFO: suspend vm
INFO: starting final sync /proc/10059/root// to /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp
INFO: Number of files: 28,106 (reg: 21,878, dir: 2,426, link: 3,802)
INFO: Number of created files: 0
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 0
INFO: Total file size: 961,455,660 bytes
INFO: Total transferred file size: 0 bytes
INFO: Literal data: 0 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 0
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 652,787
INFO: Total bytes received: 2,612
INFO: sent 652,787 bytes received 2,612 bytes 262,159.60 bytes/sec
INFO: total size is 961,455,660 speedup is 1,466.98
INFO: final sync finished (2 seconds)
INFO: resume vm
INFO: guest is online again after 3 seconds
INFO: creating archive '/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tar.lzo'
INFO: tar: /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 107 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | lzop >/mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tar.dat' failed: exit code 2
INFO: Failed at 2020-03-21 08:05:19
INFO: Backup job finished with errors
TASK ERROR: job errors

Anyone have any ideas as to why the backup is fialing all of a sudden. I can see in the error log that its complaining about permissions, yet on the node, I am able to read / write to the dump location without issue.

INFO: tar: /mnt/pve/vdc-backup/dump/vzdump-lxc-107-2020_03_21-08_04_29.tmp: Cannot open: Permission denied

Thanks
 
Hi!

Could you please also post the following

Code:
pveversion -v
pct config 107
cat /etc/pve/storage.cfg
cat /etc/vzdump.conf

and try setting your temporary directory to something local?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!