Hidden backup jobs

[ProxmoxFan]

Something strange happening here. I have some backup jobs running that shouldn't be and I can't find where the command is that is telling them to run.
I originally setup my cluster back on Promox 6 with PBS1. (Now upgraded to 7.2.7 and PBS 2.2.5) At that time, the GUI couldn't handle all of the intricate details of scheduling certain backup jobs, so I set them up using the command line in /etc/pve/vzdump.

As you can see from attached image, there are currently 5 scheduled jobs for VMs 100, 104, 106, 109 and 105. It has always worked fine.

However, the past week or so, I am getting an email report for a couple of VMs that aren't even on that list! One fails backup with "broken pipe" - 106 - BUT it's not the same 106 as listed above. The 106 listed in the vzdump file is a database server, and this 106 is not it (has a different name on it). The other completes successfully (108) but 108 is not an ID included in the vzdump above, so how are they running? Why is 106 the wrong ID and how is 108 being told to run a backup job? Where can I find the command that's telling these VMs to backup, if they're not included in the vzdump listed above?

Thanks
John

 

[Dunuin]

The webUI changed between PVE 6.X and PVE 7.X. You shouldn`t edit backups jobs without clearing the browser cache first (CTRL+F5). Otherwise backups jobs will be created that won't work and some backup jobs will be missing in the webUI.

 

[ProxmoxFan]

Hi Dunuin - I know that 6 & 7 are different, but all of my backup jobs were created in 6 before the upgrade and have been working fine for over a year. However now that I started adding / deleting some new VMs as I test various programs etc... there seems to have been a disconnect somewhere along the line. I currently have 0 backups visible in the UI (never did, as they disappeared from 6 when I had to edit the cron file and they never came back even in 7). However my question here is what is the command / where is it running from, that causes these "hidden" backups to run (for the incorrectly identified 106 and the non-existant 108)? There must be a script somewhere that is being called that references these VMs and I need to delete it.

Thanks

 

[Dunuin]

PVE changed the scheduler. PVE 6.X used "/etc/cron.d/vzdump" to schedule backups but PVE7.X will use calender events like described here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_calendar_events

When upgrading from PVE 6.X to 7.X all old backup jobs should be converted from the old to the new format. If you didn't cleared your browser cache you are using an old cached PVE 6.X webUI with a modern PVE 7.X running, so the webUI can't show backups using the new format and when editing backup jobs these will be wrong because you try to create PVE 7.X backup jobs with a PVE 6.X webUI. So always clear all your browser caches after an PVE upgrade, so its really using the PVE 7.X webUI with a PVE 7.X server. Atleast here that helped with missing backups jobs after upgrading from PVE 6.X to 7.X.

 

[ProxmoxFan]

In a web browser that I typically don't use, and just to be sure I deleted all history/cache/everything from it first, then closed and reopened the browser and then logged into proxmox. I don't see anything different than my normal browser. On the Datacenter tab, under the new Backup tab, it is blank and telling me that none of my VMs are covered by a backup - even though they are, and run several times a day.

You mentioned converting the backup jobs to the new format. Do you have instructions for how to do that? Your calendar link above only talks about the options for scheduling. Not the details of where and how. It also doesn't reference where I can find whatever command is making those 2 "hidden" jobs run and stop it.

Thanks

 

[Neobin]

With the addition of pvescheduler - the new backup schedules are no longer configured in the cluster-wide /etc/pve/vzdump.cron file. While this file is still read and will remain supported for the foreseeable future, changing an existing backup schedule will cause the schedule to be stored in the new config file /etc/pve/jobs.cfg.

This is not problematic, as long as not all nodes in a cluster have been upgraded to the new version, since backups will only run on nodes who read the new config file.
To avoid any problems, simply upgrade all nodes in your cluster before making any changes to scheduled backups.

https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_7.1

How does your: /etc/pve/jobs.cfg look like?

 

[ProxmoxFan]

https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_7.1

How does your: /etc/pve/jobs.cfg look like?

Hi - apparently I don't have a jobs.cfg file. (see attached screenshot of /etc/pve folder)

 

[Neobin]

The following is all without deeper knowledge and experience with such a problem, so take this with a grain of salt, but what I would do is:

1. Create a test backup job over the GUI and see if there now is a: /etc/pve/jobs.cfg.
2. Check if the test backup job runs successful as expected.
3. If yes: Remove all jobs from the deprecated: /etc/pve/vzdump.cron.
4. Make sure that: /etc/pve/vzdump.cron is empty on all nodes in the cluster.
5. Create all wanted backup jobs over the GUI from scratch.

With this at least you should now have your wanted backup jobs in a clean state utilizing the pvescheduler.

Again: I do not know how PVE handles all this in the background! Maybe you want to wait for a more qualified answer.
Normally, if you had setup your backup jobs over the GUI before PVE 7.1, after the upgrade to PVE 7.1, the migration is done automatically when a backup job is changed over the GUI.

For the ghost backup jobs, did you already check on all nodes, if something regarding backup jobs is in: /etc/cron.d/? /etc/cron.d/vzdump should be a symlink to: /etc/pve/vzdump.cron.

Otherwise I'm out of ideas, sorry.

 

[ProxmoxFan]

Thanks everyone for your ideas. I figured it out. Apparently the "hidden" jobs weren't hidden at all. They were coming from a test server that I use for playing with VMs before using in Production, and I had forgotten about these backups. So the good news is there aren't any problems with production cluster