PVE runs a bunch of backups after unscheduled power cycle

N

NextPancake401

Member
Dec 17, 2021
21
2
8
Ohio
I have a 3 node cluster with HA off. It runs pretty well and does as expected most of the time except when power goes out. Anytime the cluster has an unexpected power cycle or outage (like losing power to the office), when the systems come back online, Proxmox VE starts a surge of backups of a bunch of VMs and containers. Even if the cluster was only off for a few minutes (and those minutes weren't during a time backup was meant to happen).
Is there any reason for this to happen? What might be causing it?
 
Was this ever solved ?
I have 3 backup jobs set to run during the night.
All of them have "Repeat Missed" unchecked and yet every time power goes off and on during the day - all backups jobs are started.

proxmox backup job.jpg
 
Last edited:
could you post the full journal of that boot ("journalctl -b")?
 
I lost power again and when I booted up - it didn't make another backup.
I guess that journalctl -b will show details only of that last boot ?
Any way to extract previous boot details ?
 
Blonov said:
journalctl -b -1

Phil
Click to expand...
I found it in the meantime - but thanks :)
Attached log I cleaned from lines that contained any of these words (as they were repeating so many times the log was too huge - I must work on them as I had no idea). I also removed the domain. Let me know if you need the full version.

nut-driver@ups
nut-monitor
pbs1 (my Proxmox Backup Server's datastore which is offline most of the day but Proxmox still scans it)
pbs1_USB_NVME (same as above, second datastore)
status update time
sshd-session
 

Attachments

I think I know what happened there..

on startup of the scheduler, it will set the "last run" to now if repeat-missed is not enabled (basically pretending all jobs ran while the system was down). your system lost the system clock, so it set the last run to September. and then when the clock was synchronized again, the last run was obviously quite a bit in the past, so an execution was scheduled ;)

if you go to the point in the log where the time got synchronized from Sep 9th 11:07 to Nov 18th 13:50, you will see that the backup tasks are scheduled exactly on the next full minute when pve-scheduler does its next cycle..
 
  • Like
Reactions: dariuszszyc and UdoB
Makes sense, and my thought were around same idea initially.
But how do I fix that ? It's not the 1st time my system wasn't showing the right date&time and it couldn't sync as internet was not up yet.

Does this mean I have low BIOS battery power or the issue is somewhere else ?
 
this probably means that your BIOS battery is empty, or your RTC is broken, yes.
 
  • Like
Reactions: dariuszszyc
You must log in or register to reply here.
Top Bottom