[SOLVED] Update: After upgrade to PVE 7.4 scheduled backups don't run on 2 nodes

M_Allen

Member
Jul 7, 2022
10
3
8
There are 2 updates below. tldr;
- Update 1: Remove ibev-perl and restart pvescheduler
- Update 2: Restarting pvescheduler may not be enough. Instead stop pvescheduler and then start it again.

Original Problem:

We have two clusters, one has 3 nodes and the other has 2. Both clusters are kept up to date, use PBS and had working scheduled backups before the upgrade to PVE 7.4.

After upgrading to PVE 7.4 the 2 node cluster is working (and backing up) fine. On the 3 node cluster only guests from one of the nodes are being backed up. There are no errors - anywhere. There is nothing about trying to backup the guests from the other two nodes. They seem to just be ignored.

I have attached some files:
- Pve03.txt has the file versions of the PVE nodes.
- All 3 nodes have the same versions except for the old kernel versions (they are all running the same kernel.)
- Pbs.txt has the file versions of the backup server. The backup server is running as a VM on the node that the backups are working on.
- jobs.cfg.txt are the backup job files from /etc/pve - as one would expect the file contents match.
- email-log.txt is the text of a scheduled backup email that shows all of the guests that were backed up and those that should have been backed up. No errors are shown in this log either.
- vzdump.cron only has a PATH statement:
PATH="/usr/sbin:/usr/bin:/sbin:/bin"

If I create a new duplicate backup schedule only the guests on Pve03 are backed up.
- If I manually run the backup schedule all selected guests on all 3 nodes are backed up.

If I create separate backups for each node then the two problem nodes backups do "not" run on schedule. No errors, no logs. Just ignored. (They are in jobs.cfg.txt)
- They will run if started manually.

If it matters the two nodes where the backups do no run were created first. They were added to a 2 node cluster. They were running PVE 5x. Then the third node was added at a later date under PVE 6x (I think.) The third node is the only node that scheduled backups run on.

Thanks for a great set of products!

Update 1:

Using what nevakee posted below I removed `libev-perl` from my 2 problem nodes and restarted pvescheduler. Because I have "Repeat Missed" checked in my backup settings the guests on each of the nodes started backing up immediately.
- Both nodes have been running PVE since at least 5.x
- I verified that the other "newer" nodes did not have `libev-perl` installed.

Note: Updating this post caused the while thread to be embargoed until it was approved. That's why it went missing for a period of time.

Update 2 - March 30:

After some updates to PVE the newest node stopped performing scheduled backups. This is the node that always worked.
- Restarting pvescheduler did not fix the problem.
- After the second round of updates and an update to Proxmox backup the problem persisted.

It was found (at least on this node) that restarting pvescheduler won't work. The pvescheduler service has to be stopped and then started. This fixes its problem.
 

Attachments

  • pbs.txt
    640 bytes · Views: 5
  • Pve03.txt
    1.5 KB · Views: 2
  • jobs.cfg.txt
    1.2 KB · Views: 2
  • email-log.txt
    13.3 KB · Views: 1
Last edited:
Man I thought I broke something myself. Glad to see that its affecting more than just me. I've tried restarting all hosts, resetting all backups but nothing. I have 3 nodes in my cluster and all of them are not backing up any more to the PBS we have.

Hopefully this gets resolved quick.
 
  • Like
Reactions: thusband
My backups are every Sunday at 01:00 so I thought, initially, that it had to do with the time change here in the UK. Manual backups work just fine.
 
My backups are every Sunday at 01:00 so I thought, initially, that it had to do with the time change here in the UK. Manual backups work just fine.
I run backups every 15 minutes so it's a bit of an issue if something does go wrong.

I've seen on the bug here https://bugzilla.proxmox.com/show_bug.cgi?id=4615 that some users have had success by removing libev-perl. I've tried removing libev-perl and restarting the pvescheduler but not sure if it's working yet, the tasks haven't fired and I'm running a manual backup. Will see what happens after the manual backup.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!