Boot pve starts skipped backups after long powerdown

[Kodey]

Whenever I power down pve say to relocate it, if it's been down for a while, when I boot the backups start so the vm which I normally use can't boot and I have to wait for the backups to finish before booting my regular vm. The backup is a snapshot but all the missed backups seem to be kicked off and a list of backups waiting for a lock is shown.
It doesn't seem to make sense as the last backup when the pve was last power on is usually sufficient and I really just want to get on with my work but it takes hours.
I'm using pve 7.1-10 with the pbs installed on the same node.

Why does it do this and how can I prevent it from trying to catch up on skipped backups when booting after a long power off?

 

[fabian]

you can set your guests to start on boot, that will happen before the backup scheduler is started on current PVE systems. also, starting a VM while a backup is running should be possible (for containers you have to wait for the backup of that container to be done before starting it).

 

[Kodey]

Thanks Fabian. Yes, that guest is set to start at boot but it doesn't when this happens. I'm not sure why but it has the disk logo on it from the web interface and doesn't start automatically even though it is supposed to and it normally does when it's just a quick reboot.
I didn't try to start it while the backup was running this time, but it started normally when I ran it manually after the backup was finished.
All the containers that normally startup at boot, did start even though they had snapshot backups running too.
It doesn't happen often and it's difficult to reproduce since I don't usually have a lot of downtime, so I'm not sure how best to find out what's going on.
I thought what you said too, so you can imagine how surprised I was when it didn't run that way.
What do you suggest?

 

[fabian]

the fix for ordering the (new) job scheduler that replaced cron a while ago after the on-boot guests is in pve-manager 7.1-11 - maybe you need to update?

 

[bogo22]

@fabian: Today I experienced the same, I guess what @Kodey meant was the following (described on my case):

My vm/ct's are backuped once a week on Monday to a external PBS server.
My last successful backup was on 18.04.22.
I turned off my PVE server from 20.04.22 to 29.04.22 (I started the PVE server on 29.04.22 - 10:16 am).
My assumption: Due to the shutdown of the PVE server the "next" regular backup cycle (25.04.22) is skipped and the next but one backup is on 02.05.22.
My observation: Immediately after boot/startup of the PVE server (29.04.22 - 10:16 am) the backup job was started, which is curious because there is not cron/backup job on around 10 am or so.
So it seams like the missed backup job from 25.04.22 is issued right after boot - is this the intended behavior of PVE/PBS?
It would be nice if the user can specify if missed backup jobs should be catched up (a similar option exists on systemd timers with persistent=, see LINK).

cat /etc/pve/jobs.cfg
(NOTE: In my example the backup storage type is "pbs", not sure if this behavior is the same with storage type=dir)
[...] vzdump: backup-[...] schedule mon 03:00:00 compress zstd enabled 1 mailnotification always mailto root mode stop node PVE-1 prune-backups keep-all=1 storage [...] vmid [...] [...]

pveversion -v |grep manager
pve-manager: 7.1-12 (running version: 7.1-12/b3c09de3) pve-ha-manager: 3.3-3

(NOTE: See time of start of the backup job below... 3min after boot..)
cat /var/log/syslog |grep pvescheduler
Apr 29 10:19:42 PVE-1 pvescheduler[57977]: starting server Apr 29 10:19:42 PVE-1 pvescheduler[57979]: <root@pam> starting task UPID:[...]:vzdump::root@pam: Apr 29 10:19:42 PVE-1 pvescheduler[57980]: INFO: starting new backup job: vzdump [...] --storage [...] --mailnotification always --compress zstd --quiet 1 --mailto root --prune-backups 'keep-all=1' --node PVE-1 --mode stop [...] Apr 29 10:31:22 PVE-1 pvescheduler[57980]: INFO: Backup job finished successfully

 

[fabian]

yes, it will do that: https://git.proxmox.com/?p=pve-mana...51fa81b4880be11a948;hb=refs/heads/master#l233

something similar to persistent in systemd-timers should be implementable, feel free to file a feature request at https://bugzilla.proxmox.com

 

[bogo22]

yes, it will do that: https://git.proxmox.com/?p=pve-mana...51fa81b4880be11a948;hb=refs/heads/master#l233

something similar to persistent in systemd-timers should be implementable, feel free to file a feature request at https://bugzilla.proxmox.com

Much thanks, for reference feature request: https://bugzilla.proxmox.com/show_bug.cgi?id=4026