Scheduled Backups Didn't Run (After 8.2.5 upgrade)

[tjh]

Platform: Standalone Proxmox 8.2.5 using only the community (Free) repo:

root@max:/var/log# pveversion
pve-manager/8.2.5/12c0a59769080547 (running kernel: 6.8.12-1-pve)

I got an alert this morning that my backups of my Proxmox guests were older than 24 hours.
Given that I have a job that runs every morning at 1am my time, that didn't seem right, especially as I haven't touched it in months.
I went to check the job was still there, it is:



So I checked my filesystem, and sure enough the backups I was expecting weren't there.

I checked the log in the GUI, the backup job isn't there (nothing shown for today the 20th Sept at ~1:30)



Nothing has changed on my server except that I installed updates yesterday when I saw them. These were the updates that were installed:

2024-09-19 09:40:54 upgrade libnvpair3linux:amd64 2.2.4-pve1 2.2.6-pve1
2024-09-19 09:40:54 upgrade libproxmox-rs-perl:amd64 0.3.3 0.3.4
2024-09-19 09:40:54 upgrade libpve-rs-perl:amd64 0.8.9 0.8.10
2024-09-19 09:40:55 upgrade libpve-storage-perl:all 8.2.3 8.2.4
2024-09-19 09:40:55 upgrade libuutil3linux:amd64 2.2.4-pve1 2.2.6-pve1
2024-09-19 09:40:55 upgrade libzfs4linux:amd64 2.2.4-pve1 2.2.6-pve1
2024-09-19 09:40:55 upgrade libzpool5linux:amd64 2.2.4-pve1 2.2.6-pve1
2024-09-19 09:40:55 upgrade novnc-pve:all 1.4.0-3 1.4.0-4
2024-09-19 09:40:56 upgrade pve-firmware:all 3.13-1 3.13-2
2024-09-19 09:41:05 upgrade proxmox-kernel-6.8:all 6.8.12-1 6.8.12-2
2024-09-19 09:41:05 upgrade proxmox-offline-mirror-docs:all 0.6.6 0.6.7
2024-09-19 09:41:05 upgrade proxmox-offline-mirror-helper:amd64 0.6.6 0.6.7
2024-09-19 09:41:05 upgrade pve-container:all 5.1.12 5.2.0
2024-09-19 09:41:05 upgrade pve-esxi-import-tools:amd64 0.7.1 0.7.2
2024-09-19 09:41:06 upgrade pve-i18n:all 3.2.2 3.2.3
2024-09-19 09:41:06 upgrade pve-manager:amd64 8.2.4 8.2.5
2024-09-19 09:41:06 upgrade pve-qemu-kvm:amd64 9.0.2-2 9.0.2-3
2024-09-19 09:41:10 upgrade spl:all 2.2.4-pve1 2.2.6-pve1
2024-09-19 09:41:10 upgrade zfs-initramfs:all 2.2.4-pve1 2.2.6-pve1
2024-09-19 09:41:10 upgrade zfsutils-linux:amd64 2.2.4-pve1 2.2.6-pve1
2024-09-19 09:41:10 upgrade zfs-zed:amd64 2.2.4-pve1 2.2.6-pve1

These upgrades moved my Promox PVE from 8.2.4 to 8.2.5. I haven't rebooted into the newer kernel it provides yet.

I have 2 VMs I backup and 2 LXC containers. Storage is just local-lvm on the same SSD the node boots from, and backups are written to an external USB drive that's mounted on boot:

/dev/sdc1 on /mnt/usb_backup type ext4 (rw,relatime,errors=remount-ro)

(Once backups have completed, restic then ships them off to remote storage)

Has anyone else experienced this issue since the 8.2.4->8.2.5 upgrade ~yesterday? Is there anything I can test/check to see why the scheduled backups didn't run? Is there any other information/logfiles I can provide or shoud be examining to determine why the scheduled backup didn't run?

The one thing this has done is proven to me my Zabbix Monitoring of "last backup writes" works corectly to alert me to the condition.

Thank you very much.

 

[tjh]

Replying to myself, but I have just found this in syslog.
Sadly I don't really know what it means nor how to fix it.

2024-09-20T01:00:06.896391+12:00 max pvescheduler[3248904]: 400 Parameter verification failed.#012job-id: invalid format - invalid configuration ID '5e8548a1be291e6b716
9e88156579566cdd9872d:1'#012
2024-09-20T01:00:06.896541+12:00 max pvescheduler[3248904]: jobs: encountered object '400 Parameter verification failed.#012job-id: invalid format - invalid configurati
on ID '5e8548a1be291e6b7169e88156579566cdd9872d:1'#012#012', but neither allow_blessed, convert_blessed nor allow_tags settings are enabled (or TO_JSON/FREEZE method mi
ssing) at /usr/share/perl5/PVE/Jobs.pm line 228.

 

[MrJosh]

I updated yesterday and also found that my system did not backup last night.
Reviewing what you have posted, I have the same issue and error message in the system log for my backup jobs.

root@pve:~# pveversion
pve-manager/8.2.5/12c0a59769080547 (running kernel: 6.8.12-2-pve)

 

[tjh]

Thanks MrJosh appreciate you commenting - good to know it's not happening to just me.
Hopefully a small error that can be fixed quickly.
Cheers!

 

[jshuf]

I've been experiencing the same symptoms since upgrading to 8.2.5. 3 of the 6 nodes in my cluster failed to run their nightly backup jobs.

Digging into this, I noticed that in /etc/pve/jobs.cfg, nodes that successfully ran backups had headers of the form:

vzdump: backup-XXXXXXXX-YYYY

While nodes that failed had headers of the form:

vzdump: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ:1

I also noticed that the "job-id" in the syslog message had this latter form. Based on that, and the 'invalid-format' error, and the fact that the affected nodes are the longest running Proxmox nodes in my cluster, perhaps it's something to do with an older format for jobs in the configuration file? If that's the case, then perhaps deleting the affected backup jobs, recreating them, and then restarting the scheduler with

# systemctl restart pvescheduler.service

might be a workaround.

And that seems to be the case...after recreating the jobs, they now have the backup-XXXXXXXX-YYYY form in /etc/pve/jobs.cfg, and after restarting the scheduler, the backups ran. Going to wait for the full set of overnight backups to run again before declaring victory, but I'm optimistic.

 

[MrJosh]

Looks like there is a fix in the testing repo:

pve-manager (8.2.6) bookworm; urgency=medium

* fix #5731: vzdump jobs: fix execution of converted jobs

-- Proxmox Support Team <support@proxmox.com>  Fri, 20 Sep 2024 17:47:17 +0200

 

[tjh]

Hi @jshuf Thanks for the hint - I was going to recreate my job myself as it didn't run again last night (to be clear, I didn't expect it would)

I have recreated it and restarted the scheduler. Thanks for your post.

 

[kohly]

very nice!
this helps!

 

[jshuf]

Confirming that (at least for me), the workaround was successful - all backup jobs on all nodes ran successfully overnight.

 

[Kevo]

I also have this problem. I think I'm just going to do manual backups for a few days to see if the fix comes out. I only have 2 machines running the latest patches, so only a handful of backups need to be done.

 

[Kevo]

The 8.2.6 update fixed the issue for me. No problem with the backups running early this morning.