[SOLVED] Scheduled backups stopped running

[egrueda]

Using proxmox 7.4-3 in all of my servers.
All of them using PBS client to make daily backups with no errors.
A few days ago, one of my servers stopped doing the scheduled backups.
There was no error I could found, scheduled task just didn't run.
Manually running the backup still runs without problem.

I created a second backup task just to check, but it's not running automatically either.

Why isn't the schedule task running?
Can you gime me any help or any hint?
Thanks!

/etc/pve/vzdump.cron

# cluster wide vzdump cron schedule
# Automatically generated file - do not edit


PATH="/usr/sbin:/usr/bin:/sbin:/bin"

pveversion -v

proxmox-ve: 7.4-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.13: 7.1-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
pve-zsync: 2.2.3
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

 

[Chris]

Using proxmox 7.4-3 in all of my servers.
All of them using PBS client to make daily backups with no errors.
A few days ago, one of my servers stopped doing the scheduled backups.
There was no error I could found, scheduled task just didn't run.
Manually running the backup still runs without problem.

I created a second backup task just to check, but it's not running automatically either.

Why isn't the schedule task running?
Can you gime me any help or any hint?
Thanks!

/etc/pve/vzdump.cron

# cluster wide vzdump cron schedule
# Automatically generated file - do not edit


PATH="/usr/sbin:/usr/bin:/sbin:/bin"

pveversion -v

proxmox-ve: 7.4-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.13: 7.1-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
pve-zsync: 2.2.3
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

Hi,
please post the output of systemctl status pvescheduler.service as well as ps auxwf. You most likely run into this bug, which is already fixed, see https://bugzilla.proxmox.com/show_bug.cgi?id=4615

 

[egrueda]

Thanks, Chris.

I see service pvescheduler is enabled, active and running.
If ths bug was the issue, should the service be stopped?

This started happening last week after I updated the server on march 27th
I have updated it again couple of hours ago

Also:
libev-perl: 4.33-1+b1
libpve-common-perl: 7.3-4

Should I uninstall libev-perl?

Edit: sorry, I forgot about "ps auxwf" output

 

[HeJohn]

same here, so i restarted the pvescheduler under server/system - it was on status running.
no error message when restarting the pvescheduler.

automatic backup still did not work !

on the console i saw with "ps -ef | grep pvescheduler"
[pvescheduler] <defunct>

so i restarted pvescheduler again via the web-gui, but the process kept the status.
with "service pvescheduler restart" the service became active again.

restarting pvescheduler again via the web-gui, but the process kept the original start-time ?!.
a stop and start of pvescheduler worked.

the automatic backup worked again.

 

[Chris]

Thanks, Chris.

I see service pvescheduler is enabled, active and running.
If ths bug was the issue, should the service be stopped?

This started happening last week after I updated the server on march 27th
I have updated it again couple of hours ago

Also:
libev-perl: 4.33-1+b1
libpve-common-perl: 7.3-4

Should I uninstall libev-perl?

You can either send a kill signal to the defunct processes and restart the service or reboot the host. Both should get rid of the zombie processes. See here for further reference https://forum.proxmox.com/threads/p...-not-run-at-schedule-time.124926/#post-544752

 

[egrueda]

Got it!

Thanks @Chris and @HeJohn for your help.
I saw those "defunct" zombie processes
I just stopped service, verified no zombie processes were there and started the service again.

A scheduled backup has just started, so I can confirm issue is solved.
Big thanks, guys!