[SOLVED] Proxmox CT Backup failed

[proxhelp]

I am running the following proxmox version/kernel: Linux 6.5.11-8-pve (2024-01-30T12:27Z

I am trying to backup (manually) one of my Proxmox CT backups. The backup worked before but now it is not working. When I run the backup, it exits with the following error:

ERROR: Backup of VM 20001 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/backup/dump/vzdump-lxc-20001-2024_06_26-14_42_43.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/backup/dump/vzdump-lxc-20001-2024_06_26-14_42_43.tar.dat' failed: exit code 1
INFO: Failed at 2024-06-26 14:43:28
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

 

[Chris]

I am running the following proxmox version/kernel: Linux 6.5.11-8-pve (2024-01-30T12:27Z

I am trying to backup (manually) one of my Proxmox CT backups. The backup worked before but now it is not working. When I run the backup, it exits with the following error:

ERROR: Backup of VM 20001 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/backup/dump/vzdump-lxc-20001-2024_06_26-14_42_43.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/backup/dump/vzdump-lxc-20001-2024_06_26-14_42_43.tar.dat' failed: exit code 1
INFO: Failed at 2024-06-26 14:43:28
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Hi,
please provide the full backup task log and the container config. The latter you get via the output of pct config <VMID> --current

 

[proxhelp]

Hi, here is the full log:

2024-06-28 08:02:06 INFO: Starting Backup of VM 20001 (lxc)
2024-06-28 08:02:06 INFO: status = running
2024-06-28 08:02:06 INFO: backup mode: stop
2024-06-28 08:02:06 INFO: ionice priority: 7
2024-06-28 08:02:06 INFO: CT Name: [DELETED]
2024-06-28 08:02:06 INFO: including mount point rootfs ('/') in backup
2024-06-28 08:02:06 INFO: stopping virtual guest
2024-06-28 08:02:39 INFO: creating vzdump archive '/backup/dump/vzdump-lxc-20001-2024_06_28-08_02_06.tar.zst'
2024-06-28 08:03:13 INFO: tar: ./var/log/gitlab/gitlab-rails/production_json.log: File shrank by 389286910 bytes; padding with zeros
2024-06-28 08:03:27 INFO: Total bytes written: 12934369280 (13GiB, 254MiB/s)
2024-06-28 08:03:28 INFO: restarting vm
2024-06-28 08:03:29 INFO: guest is online again after 83 seconds
2024-06-28 08:03:30 ERROR: Backup of VM 20001 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/backup/dump/vzdump-lxc-20001-2024_06_28-08_02_06.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/backup/dump/vzdump-lxc-20001-2024_06_28-08_02_06.tar.dat' failed: exit code 1

Here is the container config:

arch: amd64
cores: 4
features: nesting=1
hostname: [DELETED]
memory: 6144
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=BC:24:11:31:FF:07,ip=192.168.1.216/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-20001-disk-0,size=40G
swap: 2048
unprivileged: 1
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

 

[fiona]

Hi,

2024-06-28 08:03:13 INFO: tar: ./var/log/gitlab/gitlab-rails/production_json.log: File shrank by 389286910 bytes; padding with zeros

This is the problematic file. You might want to run pct fsck 20001 and if that doesn't help, manually check/salvage/remove the file.

 

[proxhelp]

Hi,

2024-06-28 08:03:13 INFO: tar: ./var/log/gitlab/gitlab-rails/production_json.log: File shrank by 389286910 bytes; padding with zeros

This is the problematic file. You might want to run pct fsck 20001 and if that doesn't help, manually check/salvage/remove the file.

Hi, it worked perfectly after I manually deleted the file.

I am curious though. How did you know the production_json.log was a problem? It simply says that this file was shrunk and 0s were added? What indicates this file was the culprit?

 

[fiona]

Hi, it worked perfectly after I manually deleted the file.

I am curious though. How did you know the production_json.log was a problem? It simply says that this file was shrunk and 0s were added? What indicates this file was the culprit?

Well, it can be seen from the log: tar logged that the file changed (was padded with zeroes in the backup). And then later tar exited with code 1, because of this mismatch. From man tar, section RETURN VALUE

1      Some  files  differ.  If tar was invoked with the --compare (--diff, -d) command line option, this means that some files in the archive differ from their disk counterparts.  If tar was
              given one of the --create, --append or --update options, this exit code means that some files were changed while being archived and so the resulting archive does not contain the  exact
              copy of the file set.