[SOLVED] Loss of backup configurations in PVE

Y

YAGA

Renowned Member
Feb 15, 2016
76
8
73
58
Hi there,

The problem recently emerged on a 4-node cluster using different filesystems (PBS with multiple datastores, Ceph, and several NFS volumes) for backups.

I am using the PVE Community edition with the latest updates available to date, i.e., 8.0.4.

Suddenly, there were no backups, and some information had disappeared in the GUI under the backup tab.

The information had also vanished from the file containing backup information, /etc/pve/jobs.cfg.

The /etc/pve/jobs.cfg file still matches the GUI.

Here is a short example from the /etc/pve/jobs.cfg file that highlights the issue:

... vzdump: backup-a72814b2-6a82 enabled 1 repeat-missed 1 schedule Sat 00:00 vzdump: backup-09b8f777-8757 enabled 1 repeat-missed 1 schedule Sat 06:00 vzdump: backup-209eb4e4-aa5c enabled 1 repeat-missed 1 schedule Sat 12:00 ...

I noticed that the following information is missing:

all 1 mailnotification always mailto xxx mode snapshot notes-template {{cluster}}, {{node}}, {{guestname}}, {{vmid}} storage xxx

By manually adding the backup information in the GUI, everything works again for a while, but then it disappears again (perhaps during a reboot?).

Do you have any clues to solve this problem?

Kind regards,
YAGA
 
Last edited:
Good evening,

The bug with the /etc/pve/jobs.cfg file has occurred again, the jobs.cfg file no longer contains the information for backups except :

... vzdump: backup-a72814b2-6a82 enabled 1 repeat-missed 1 schedule Sat 00:00 vzdump: backup-09b8f777-8757 enabled 1 repeat-missed 1 schedule Sat 06:00 vzdump: backup-209eb4e4-aa5c enabled 1 repeat-missed 1 schedule Sat 12:00 ...

Since I have several backup filesystems including several disks and several datastores on a PBS server, I have the feeling that this is a problem that may be related to this bug which required a patch made by @Lukas Wagner : https://lists.proxmox.com/pipermail/pve-devel/2023-August/058881.html

In my case it is more critical because the /etc/pve/jobs.cfg file is corrupted, essential information is missing and PVE no longer performs backups.

Any advice welcome,

Kind regards,
YAGA
 
Last edited:
Hey!
The patch that you have linked is not available in the pve-nosubscription and enterprise repositories yet, only in pvetest - so I doubt that this is the issue. The issue was only in relation to sending notifications after backup jobs and should not have an influence on the configuration.

Could you please provide the output of pveversion -v?
Also, would it be able for you to share the full contents of /etc/pve/jobs.cfg?
 
Hello Lukas,

Thanks for your message.

Here is the pveversion -v output :

proxmox-ve: 8.0.2 (running kernel: 6.2.16-12-pve) pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390) pve-kernel-6.2: 8.0.5 proxmox-kernel-helper: 8.0.3 pve-kernel-5.15: 7.4-4 pve-kernel-5.11: 7.0-10 proxmox-kernel-6.2.16-12-pve: 6.2.16-12 proxmox-kernel-6.2: 6.2.16-12 proxmox-kernel-6.2.16-10-pve: 6.2.16-10 pve-kernel-6.2.16-5-pve: 6.2.16-6 pve-kernel-6.2.16-4-pve: 6.2.16-5 pve-kernel-5.15.108-1-pve: 5.15.108-1 pve-kernel-5.13.19-2-pve: 5.13.19-4 pve-kernel-5.11.22-7-pve: 5.11.22-12 pve-kernel-5.11.22-1-pve: 5.11.22-2 ceph: 17.2.6-pve1+3 ceph-fuse: 17.2.6-pve1+3 corosync: 3.1.7-pve3 criu: 3.17.1-2 glusterfs-client: 10.3-5 ifupdown2: 3.2.0-1+pmx4 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-4 libknet1: 1.25-pve1 libproxmox-acme-perl: 1.4.6 libproxmox-backup-qemu0: 1.4.0 libproxmox-rs-perl: 0.3.1 libpve-access-control: 8.0.5 libpve-apiclient-perl: 3.3.1 libpve-common-perl: 8.0.8 libpve-guest-common-perl: 5.0.4 libpve-http-server-perl: 5.0.4 libpve-rs-perl: 0.8.5 libpve-storage-perl: 8.0.2 libspice-server1: 0.15.1-1 lvm2: 2.03.16-2 lxc-pve: 5.0.2-4 lxcfs: 5.0.3-pve3 novnc-pve: 1.4.0-2 proxmox-backup-client: 3.0.2-1 proxmox-backup-file-restore: 3.0.2-1 proxmox-kernel-helper: 8.0.3 proxmox-mail-forward: 0.2.0 proxmox-mini-journalreader: 1.4.0 proxmox-offline-mirror-helper: 0.6.2 proxmox-widget-toolkit: 4.0.6 pve-cluster: 8.0.3 pve-container: 5.0.4 pve-docs: 8.0.4 pve-edk2-firmware: 3.20230228-4 pve-firewall: 5.0.3 pve-firmware: 3.8-2 pve-ha-manager: 4.0.2 pve-i18n: 3.0.5 pve-qemu-kvm: 8.0.2-5 pve-xtermjs: 4.16.0-3 qemu-server: 8.0.7 smartmontools: 7.3-pve1 spiceterm: 3.3.0 swtpm: 0.8.0+pve1 vncterm: 1.8.0 zfsutils-linux: 2.1.12-pve1

Here is the anonymized jobs.cfg file when it works...
XXXXXXXX-YYYY are real and unique numbers

vzdump: backup-XXXXXXXX-YYYY schedule Sat 00:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-1 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} repeat-missed 1 storage Cluster-CephFS vzdump: backup-XXXXXXXX-YYYY schedule Sat 06:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-2 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} repeat-missed 1 storage Cluster-CephFS vzdump: backup-XXXXXXXX-YYYY schedule Sat 12:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-3 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} repeat-missed 1 storage Cluster-CephFS vzdump: backup-XXXXXXXX-YYYY schedule Sat 18:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-4 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} repeat-missed 1 storage Cluster-CephFS vzdump: backup-XXXXXXXX-YYYY schedule Sun 00:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-1 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Syn-NAS-5 vzdump: backup-b2a43e22-46d0 schedule Sun 00:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-2 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Syn-NAS-4 vzdump: backup-XXXXXXXX-YYYY schedule Sun 00:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-3 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Syn-NAS-5 vzdump: backup-XXXXXXXX-YYYY schedule Sun 00:00 all 1 compress zstd enabled 1 mailnotification always mailto my_name@my_domain.com mode suspend node Cluster-4 notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Syn-NAS-4 vzdump: backup-XXXXXXXX-YYYY schedule mon..fri 19:00 all 1 enabled 1 mailnotification always mailto my_name@my_domain.com mode snapshot notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Eve.pbs.my_domain.com vzdump: backup-XXXXXXXX-YYYY schedule mon..fri 12:15 all 1 enabled 1 mailnotification always mailto my_name@my_domain.com mode snapshot notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Mid.pbs.my_domain.com vzdump: backup-XXXXXXXX-YYYY schedule mon..fri 06:00 all 1 enabled 1 mailnotification always mailto my_name@my_domain.com mode snapshot notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Mor.pbs.my_domain.com vzdump: backup-XXXXXXXX-YYYY schedule mon 00:00 all 1 enabled 1 mailnotification always mailto my_name@my_domain.com mode snapshot notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Mor.pbs.my_domain.com vzdump: backup-XXXXXXXX-YYYY schedule mon 02:00 all 1 enabled 1 mailnotification always mailto my_name@my_domain.com mode snapshot notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Mid.pbs.my_domain.com vzdump: backup-XXXXXXXX-YYYY schedule mon 04:00 all 1 enabled 1 mailnotification always mailto my_name@my_domain.com mode snapshot notes-template {{cluster}}, {{guestname}}, {{node}}, {{vmid}} storage Eve.pbs.my_domain.com

Please let me know what you think.

Kind regards,
YAGA
 
Last edited:
Hello Lukas,

Please let me know if you need more info or if I can do any other tests.

Regards,
YAGA
 
Do you see anything odd in the system logs? If you happen to know the time frame when this happened, you can limit the log ouptut with --since and --until:
journalctl --since YYYY-MM-DD --until YYYY-MM-DD

If you share any log output with us, please double check that you do not leak any sensitive information.
 
Hello Lukas,

Here is an extract of journalctl when error occurs.

Sep 25 21:54:01 Cluster-1 cron[3056]: (*system*vzdump) RELOAD (/etc/cron.d/vzdump) Sep 25 21:54:02 Cluster-1 pvescheduler[3736687]: 400 Parameter verification failed. vmid: property is missing Sep 25 21:54:02 Cluster-1 pvescheduler[3736687]: jobs: encountered object '400 Parameter verification failed. vmid: property is missing ', but neither allow_blessed, convert_blessed nor allow_tags settings are enabled (or TO_JSON/FREEZE method missing) at /usr/share/perl5/PVE/Jobs.pm line 228.

After the /etc/cron.d/vzdump reload I get these errors.

Sep 25 21:54:02 Cluster-1 pvescheduler[3736687]: 400 Parameter verification failed. vmid: property is missing Sep 25 21:54:02 Cluster-1 pvescheduler[3736687]: jobs: encountered object '400 Parameter verification failed. vmid: property is missing ', but neither allow_blessed, convert_blessed nor allow_tags settings are enabled (or TO_JSON/FREEZE method missing) at /usr/share/perl5/PVE/Jobs.pm line 228.

I get the same errors every time the backup is scheduled.

These errors may be the consequence and not the cause.

Regards,
YAGA
 
YAGA said:
These errors may be the consequence and not the cause.
Click to expand...
yup, this appears to be the consequence of the already missing configuration keys.

How old is this cluster? Do you happen to know which Version of Proxmox VE was used for the installation?

Could you share /etc/cron.d/vzdump with me?
 
Last edited:
  • Like
Reactions: YAGA and Lukas Wagner
Lukas Wagner said:
yup, this appears to be the consequence of the already missing configuration keys.

How old is this cluster? Do you happen to know which Version of Proxmox VE was used for the installation?

Could you share /etc/cron.d/vzdump with me?
Click to expand...
Hi Lukas,

Cluster is up and running since PVE 7. My first install was with PVE 7 but I don't know exactly the subversion.

/etc/cron.d/vzdump
Click to expand...

# cluster wide vzdump cron schedule # Automatically generated file - do not edit PATH="/usr/sbin:/usr/bin:/sbin:/bin"

Kind regards,
 
fiona said:
Hi,
did you ever run qm destroy <ID> --purge from the CLI? There was a bug in libpve-common-perl <= 8.0.8 which led to certain configuration keys to be dropped. It was fixed by https://git.proxmox.com/?p=pve-common.git;a=commit;h=d2a6411cab710d1e7b77feb134866e2c0d99b920 in version 8.0.9, currently available on the pvetest repository. EDIT: Now also on the no-subscription repository.
Click to expand...

Hi Fiona, Hi Lukas,

@fiona Very good point, I run qm destroy <ID> --purge from the CLI very frequently.

The bug actually happens after a qm destroy <ID> --purge

I will update PVE by the end of the week but I am confident that updating libpve-common-perl to 8.0.9 will resolve the issue.

Thanks a lot,
Regards
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back