[SOLVED] Continuous Backup Loop

[cgirlamo]

Hello,

I seem to be caught in a continuous backup loop. I'm running a datacluster with 11 nodes. I have nightly backups going for important VM's on each node, but over the last day backup jobs have been running and failing continuously, severely slowing down our services.

so far, I have been going through each of the nodes and running the following commands:
pkill -9 vzdump
rm -f /var/run/vzdump.lock
rm -f /var/log/pve/tasks/active
systemctl restart pvedaemon.service pveproxy.service

These commands stop all the currently running backup jobs, but then within 30 minutes or so the backups continue again.

Any help would be much appreciated

 

[niteshadow]

I guess it depends on how you scheduled the backup.
Maybe you have "Repeat missed" on ?
What type of back-end is the backup storage ? NFS ? Is it near full ?

 

[cgirlamo]

Just went and double checked the configs for my backups, "Repeat missed" was not on.

the back-end storage is NFS, and there's ~500TB of free space on there.

 

[niteshadow]

Do you have one nightly backup job or multiple jobs ?
Can you show the schedule ?

 

[cgirlamo]

 

[niteshadow]

You last job is not nightly at midnight (should show 00:00 for midnight)
24:00 is same as just 24, meaning once per hour at 24 minutes after the hour (triggers 24 times per day !)
Always use the schedule simulator to verify scheduling.

 

[cgirlamo]

That makes sense with what we've been experiencing! Thank you for your help