Backup Completion Error Due to Missing Completion Flag

[NodExplorer]

PBS: (Proxmox backup server)

The scheduled backup did not complete correctly because the completion flag was not set, resulting in the deletion of the backup. This occurred during the backup task on VM/167, and logs indicate that although the chunks were successfully uploaded, the completion flag was not configured.


Observed Result:

The backup did not finish correctly, and it was removed due to the absence of the completion flag. The log shows the following error messages:

2024-06-23T12:13:18+02:00: backup ended and finish failed: backup ended but finished flag is not set.
2024-06-23T12:13:18+02:00: removing unfinished backup
2024-06-23T12:13:18+02:00: TASK ERROR: backup ended but finished flag is not set.

PVE: (Proxmox ve)


The backup process for VM 176 failed. According to the system logs, the failure was due to an exit code 255 from the 'lxc-usernsexec' command.


Log Output:


Jun 23 12:13:19 hp-generic-1 pvescheduler[1994156]: ERROR: Backup of VM 176 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp1994156_176/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 176 --backup-time 1719137530 --repository backup@pbs@10.12.1.218:srs-backup --ns rfci-bkp' failed: exit code 255

Could someone help me how to solve this problem, I don't understand why this happens if on the machine in which several CTs and VMs are stored, like the one that is failing, they run correctly, everything else is completed except that VM that it always fails when it is executed for the backup, why is it, why does this happen and not in the others? how can I solve that?

 

[Chris]

Hi,
please provide the full backup task log for the failed backup as well as the container config, which you get via pct config <VMID> --current. Also, please provide the output of pveversion -v.

 

[NodExplorer]

Hi,
please provide the full backup task log for the failed backup as well as the container config, which you get via pct config <VMID> --current. Also, please provide the output of pveversion -v.

Here i provide you what you ask for,

thanks for you help with that!

config:

root@hp-generic-1:~# pct config 176 --current
arch: amd64
cores: 4
features: nesting=1
hostname: rfci-vnode-x86-fedora30
memory: 12288
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=00:16:3e:c2:76:d5,ip=dhcp,type=veth
onboot: 1
ostype: fedora
rootfs: srs-prxmx:subvol-176-disk-2,size=200G
swap: 0
unprivileged: 1

pve version:

root@hp-generic-1:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.131-2-pve)
pve-manager: 7.4-17 (running version: 7.4-17/513c62be)
pve-kernel-5.15: 7.4-9
pve-kernel-5.15.131-2-pve: 5.15.131-3
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.4-1
proxmox-backup-file-restore: 2.4.4-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.14-pve1

 

[Chris]

Thanks for the output, could you however also provide the backup task log, sorry if I was not clear with what I meant here.
You can get it by navigating to <your-node> > Task History > <select-corresponding-backup-task> > View > Download in the WebUI. This hopefully gives some more error context to help investigate.

On a side note: Please consider upgrading to PVE8 in the not to distant future, as PVE7 is almost at its end of live, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#faq-support-table

You can upgrade by following the steps in https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

 

[NodExplorer]

Thanks for the output, could you however also provide the backup task log, sorry if I was not clear with what I meant here.
You can get it by navigating to <your-node> > Task History > <select-corresponding-backup-task> > View > Download in the WebUI. This hopefully gives some more error context to help investigate.

On a side note: Please consider upgrading to PVE8 in the not to distant future, as PVE7 is almost at its end of live, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#faq-support-table

You can upgrade by following the steps in https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

I appreciate your patience with this issue, I may be wrong but I believe that I have shared the logs of both the pbs and the part from the pve with you, at least the equivalent of the day the process failed me, if I am wrong, forgive me, It's a bit confusing for me and it may be that I haven't shared what you're asking me for. On the other hand, if I update Proxmox, could I have problems with incompatibilities or version conflicts? Is it advisable then to update all the devices on which I use Proxmox to version 8?

 

[Chris]

if I am wrong, forgive me, It's a bit confusing for me and it may be that I haven't shared what you're asking me for

You only provided the systemd journal excerpts for both PBS and PVE, not however the backup task log, as it is generated on the PVE host. Please navigate as described in the WebUI to fetch it.

On the other hand, if I update Proxmox, could I have problems with incompatibilities or version conflicts? Is it advisable then to update all the devices on which I use Proxmox to version 8?

You should not perform the upgrade without having restorable backup, yes. For the rest please see the provided upgrade guide, which has an in depth description of possible issues and how to handle these. Normally the upgrade process is rather straight forward, but this depends a bit on how much you customized your setup.

In any case please do use the pve7to8 helper script! This caches a lot of possible issues before the upgrade.