Boot pve starts skipped backups after long powerdown

Kodey

Member
Oct 26, 2021
35
0
6
21
Whenever I power down pve say to relocate it, if it's been down for a while, when I boot the backups start so the vm which I normally use can't boot and I have to wait for the backups to finish before booting my regular vm. The backup is a snapshot but all the missed backups seem to be kicked off and a list of backups waiting for a lock is shown.
It doesn't seem to make sense as the last backup when the pve was last power on is usually sufficient and I really just want to get on with my work but it takes hours.
I'm using pve 7.1-10 with the pbs installed on the same node.

Why does it do this and how can I prevent it from trying to catch up on skipped backups when booting after a long power off?
 
Last edited:

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,483
1,396
164
you can set your guests to start on boot, that will happen before the backup scheduler is started on current PVE systems. also, starting a VM while a backup is running should be possible (for containers you have to wait for the backup of that container to be done before starting it).
 

Kodey

Member
Oct 26, 2021
35
0
6
21
Thanks Fabian. Yes, that guest is set to start at boot but it doesn't when this happens. I'm not sure why but it has the disk logo on it from the web interface and doesn't start automatically even though it is supposed to and it normally does when it's just a quick reboot.
I didn't try to start it while the backup was running this time, but it started normally when I ran it manually after the backup was finished.
All the containers that normally startup at boot, did start even though they had snapshot backups running too.
It doesn't happen often and it's difficult to reproduce since I don't usually have a lot of downtime, so I'm not sure how best to find out what's going on.
I thought what you said too, so you can imagine how surprised I was when it didn't run that way.
What do you suggest?
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,483
1,396
164
the fix for ordering the (new) job scheduler that replaced cron a while ago after the on-boot guests is in pve-manager 7.1-11 - maybe you need to update? ;)
 

bogo22

Active Member
Nov 4, 2016
51
3
28
@fabian: Today I experienced the same, I guess what @Kodey meant was the following (described on my case):

My vm/ct's are backuped once a week on Monday to a external PBS server.
My last successful backup was on 18.04.22.
I turned off my PVE server from 20.04.22 to 29.04.22 (I started the PVE server on 29.04.22 - 10:16 am).
My assumption: Due to the shutdown of the PVE server the "next" regular backup cycle (25.04.22) is skipped and the next but one backup is on 02.05.22.
My observation: Immediately after boot/startup of the PVE server (29.04.22 - 10:16 am) the backup job was started, which is curious because there is not cron/backup job on around 10 am or so.
So it seams like the missed backup job from 25.04.22 is issued right after boot - is this the intended behavior of PVE/PBS?
It would be nice if the user can specify if missed backup jobs should be catched up (a similar option exists on systemd timers with persistent=, see LINK).

cat /etc/pve/jobs.cfg
(NOTE: In my example the backup storage type is "pbs", not sure if this behavior is the same with storage type=dir)
[...] vzdump: backup-[...] schedule mon 03:00:00 compress zstd enabled 1 mailnotification always mailto root mode stop node PVE-1 prune-backups keep-all=1 storage [...] vmid [...] [...]

pveversion -v |grep manager
pve-manager: 7.1-12 (running version: 7.1-12/b3c09de3) pve-ha-manager: 3.3-3

(NOTE: See time of start of the backup job below... 3min after boot..)
cat /var/log/syslog |grep pvescheduler
Apr 29 10:19:42 PVE-1 pvescheduler[57977]: starting server Apr 29 10:19:42 PVE-1 pvescheduler[57979]: <root@pam> starting task UPID:[...]:vzdump::root@pam: Apr 29 10:19:42 PVE-1 pvescheduler[57980]: INFO: starting new backup job: vzdump [...] --storage [...] --mailnotification always --compress zstd --quiet 1 --mailto root --prune-backups 'keep-all=1' --node PVE-1 --mode stop [...] Apr 29 10:31:22 PVE-1 pvescheduler[57980]: INFO: Backup job finished successfully
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!