[SOLVED] Backup Error (exit code 1)

[ninjafella]

Hello all,

I'm experiencing an error with one of my LXCs where the backup keeps failing. I'm stuggling to find the cause or even more desicriptive logs about what is actually happening. It says that the "set -o pipefail && lxc-usernsexec" command is failing to run. I've provided the logs I found and other info bellow.

Proxmox version:

proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.8: 6.8.4-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.2-1
proxmox-backup-file-restore: 3.2.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.6
pve-container: 5.0.11
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.5
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2

pct config output:

arch: amd64
cores: 1
features: nesting=1
hostname: Tailscale
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=9A:95:AE:C0:B2:9F,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-113-disk-0,size=8G
swap: 512
tags: general
unprivileged: 1
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Task Logs:

INFO: starting new backup job: vzdump 113 --notification-mode auto --remove 0 --compress zstd --node hiro --notes-template '{{guestname}}' --storage Penfold-backups --mode stop
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp1533439_113 for temporary files
INFO: Starting Backup of VM 113 (lxc)
INFO: Backup started at 2024-05-09 00:58:38
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: Tailscale
INFO: including mount point rootfs ('/') in backup
INFO: stopping virtual guest
INFO: creating vzdump archive '/mnt/penfold-backups/dump/vzdump-lxc-113-2024_05_09-00_58_38.tar.zst'
INFO: tar: ./var/log/journal/746209223bc541718ac42e598af9fc31/system@a8f219dbf1c349718406021efd92b320-00000000000031e1-0006042c0c277849.journal: File shrank by 7929856 bytes; padding with zeros
INFO: Total bytes written: 1172408320 (1.1GiB, 28MiB/s)
INFO: restarting vm
INFO: guest is online again after 47 seconds
ERROR: Backup of VM 113 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp1533439_113' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/mnt/penfold-backups/dump/vzdump-lxc-113-2024_05_09-00_58_38.tar.dat' failed: exit code 1
INFO: Failed at 2024-05-09 00:59:25
INFO: Backup job finished with errors
INFO: notified via target `Backup-Notifcations`
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Backup config:




The backup is saving to my NAS mounted with cifs in fstab. I have many other LXCs and VMs all backing up with the same parameters to the same storage device but this one just doesn't want to work.

I've tried different modes (snapshot and stop) and without using compression.

Let me know if you can help or if you need anymore information.

 

[fiona]

Hi,
there seems to be a problematic file:

INFO: tar: ./var/log/journal/746209223bc541718ac42e598af9fc31/system@a8f219dbf1c349718406021efd92b320-00000000000031e1-0006042c0c277849.journal: File shrank by 7929856 bytes; padding with zeros

 

[ninjafella]

Hi,
there seems to be a problematic file:

INFO: tar: ./var/log/journal/746209223bc541718ac42e598af9fc31/system@a8f219dbf1c349718406021efd92b320-00000000000031e1-0006042c0c277849.journal: File shrank by 7929856 bytes; padding with zeros

I've deleted that file and all if good now thank you.