[SOLVED] Problem with backup for specific LXC container

[xtavras]

Hello,

I have proxmox with bunch of LXC containers, I do backups of them regular using "backup" option in Proxmox to my NFS share, everything works fine except one container. I always get this message.

command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/prx-tmpdir/vzdumptmp22152' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/mnt/pve/nfs-on-mysyn/dump/vzdump-lxc-104-2019_09_26-23_32_12.tar.dat' failed: exit code 1

I've tried do backup locally and it gives me the same error. Is there the way to debug whats wrong?

Thanks.

INFO: starting new backup job: vzdump 104 --mode snapshot --node prx --remove 0 --compress lzo --storage nfs-on-mysyn
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2019-10-01 18:26:36
INFO: status = running
INFO: CT Name: plex
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-104-disk-1_vzdump" created.
  WARNING: Sum of all thin volume sizes (176.00 GiB) exceeds the size of thin pool pve/data and the amount of free space in volume group (76.25 GiB).
INFO: creating archive '/mnt/pve/nfs-on-mysyn/dump/vzdump-lxc-104-2019_10_01-18_26_36.tar.lzo'
INFO: tar: ./var/cache/apt/archives/util-linux_2.31.1-0.4ubuntu3.3_amd64.deb: File shrank by 501040 bytes; padding with zeros
INFO: Total bytes written: 12555079680 (12GiB, 44MiB/s)
INFO: remove vzdump snapshot
  Logical volume "snap_vm-104-disk-1_vzdump" successfully removed
ERROR: Backup of VM 104 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/nfs-on-mysyn/dump/vzdump-lxc-104-2019_10_01-18_26_36.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/mnt/pve/nfs-on-mysyn/dump/vzdump-lxc-104-2019_10_01-18_26_36.tar.dat' failed: exit code 1
INFO: Failed at 2019-10-01 18:31:24
INFO: Backup job finished with errors
TASK ERROR: job errors

pveversion -v

proxmox-ve: 6.0-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-5.0.21-2-pve: 5.0.21-3
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-8
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2

 

[ramrot]

Have you checked all the output of the backup job? At some point it gives you the following warning

WARNING: Sum of all thin volume sizes (176.00 GiB) exceeds the size of thin pool pve/data and the amount of free space in volume group (76.25 GiB).

It seems that you have overprovisioned the thin storage and when trying to backup there is not enough free space to support the provision

 

[xtavras]

Thats true, but I do backups to NFS which have pleynty of space, I even deactivated local "tmpdir" in
/etc/vzdump.conf

And like I've said other containers does no have problems with backup.

 

[xtavras]

btw, I fixed backup. The problem lies in this line (somehow it didn't get my attention because of INFO level)

INFO: tar: ./var/cache/apt/archives/util-linux_2.31.1-0.4ubuntu3.3_amd64.deb: File shrank by 501040 bytes; padding with zeros

after removing this file on container, everything works fine.