[SOLVED] Backups not running after upgrade to 7.4

[mjtbrady]

I believe that this is the exact same problem reported here https://forum.proxmox.com/threads/pvescheduler-defunct.124781/ in the German forum.

root@pve001:/var/log# ps fauxwww | grep pvescheduler
root      110315  0.0  0.0   6244   712 pts/0    S+   10:35   0:00  |                   \_ grep pvescheduler
root      110086  0.0  0.6 336408 109192 ?       Ss   10:35   0:00 pvescheduler
root      110088  0.0  0.0      0     0 ?        Z    10:35   0:00  \_ [pvescheduler] <defunct>

The stops and starts here were all me trying stuff.

root@pve001:/var/log# journalctl -f -u pvescheduler
-- Journal begins at Mon 2022-11-28 18:46:54 NZDT. --
Mar 26 10:30:05 pve001 systemd[1]: Stopping Proxmox VE scheduler...
Mar 26 10:30:05 pve001 pvescheduler[102391]: received signal TERM
Mar 26 10:30:05 pve001 pvescheduler[102391]: got shutdown request, signal running jobs to stop
Mar 26 10:30:08 pve001 pvescheduler[102391]: server stopped
Mar 26 10:30:09 pve001 systemd[1]: pvescheduler.service: Succeeded.
Mar 26 10:30:09 pve001 systemd[1]: Stopped Proxmox VE scheduler.
Mar 26 10:30:09 pve001 systemd[1]: pvescheduler.service: Consumed 1.642s CPU time.
Mar 26 10:35:17 pve001 systemd[1]: Starting Proxmox VE scheduler...
Mar 26 10:35:17 pve001 pvescheduler[110086]: starting server
Mar 26 10:35:17 pve001 systemd[1]: Started Proxmox VE scheduler.
^C

The update was done off the pve-no-subscription repo. Note that pve-cluster still says 7.3. Is that correct?

root@pve001:/var/log# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.3-3
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-3
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.3
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20221111-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.11-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

root@pve001:/var/log# systemctl status pvescheduler
● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-03-26 10:35:17 NZDT; 5min ago
    Process: 110085 ExecStart=/usr/bin/pvescheduler start (code=exited, status=0/SUCCESS)
   Main PID: 110086 (pvescheduler)
      Tasks: 3 (limit: 18908)
     Memory: 101.4M
        CPU: 838ms
     CGroup: /system.slice/pvescheduler.service
             └─110086 pvescheduler

Mar 26 10:35:17 pve001 systemd[1]: Starting Proxmox VE scheduler...
Mar 26 10:35:17 pve001 pvescheduler[110086]: starting server
Mar 26 10:35:17 pve001 systemd[1]: Started Proxmox VE scheduler.

root@pve001:/var/log# cat /etc/pve/jobs.cfg 
vzdump: backup-98564b4b-9816
    schedule 01:00
    compress zstd
    enabled 1
    mailnotification always
    mailto a.b@c.d.e.f
    mode snapshot
    node pve001
    notes-template {{guestname}}
    prune-backups keep-daily=6,keep-monthly=6,keep-weekly=3
    repeat-missed 1
    storage nas02_nfs
    vmid 100

The backup job (I only have the one) runs without issue when run manually from the GUI and ran fine on 7.3 for a couple of months.

 

[mjtbrady]

https://bugzilla.proxmox.com/show_bug.cgi?id=4615 is tracking this issue.

I did have libev-perl installed. I have removed it and restarted pvescheduler. pveschedule and the backups are now running again.

My system was a new 7.3 install, but I think that it was an install on top of Debian due to issues with the Proxmox installer on my hardware.

 

[JRNetwork]

https://bugzilla.proxmox.com/show_bug.cgi?id=4615 is tracking this issue.

I did have libev-perl installed. I have removed it and restarted pvescheduler. pveschedule and the backups are now running again.

My system was a new 7.3 install, but I think that it was an install on top of Debian due to issues with the Proxmox installer on my hardware.

Did you do anything other than that as I have done this and it seems to still not be working. My servers backup every 15 minutes and haven't done since the upgrade. Manual backups do work

 

[mjtbrady]

Yes that is all I did. From my command history on that host

apt remove libev-perl
systemctl restart pvescheduler
systemctl status pvescheduler

That was last night. The backup then ran as normal over night at 1am.

 

[JRNetwork]

Confirmed this works, I removed as above but instead of systemctl I used service pvescheduler restart.

Thanks for the confirmation on what you used!

 

[Bob der Baumeister]

Yes that is all I did. From my command history on that host

apt remove libev-perl
systemctl restart pvescheduler
systemctl status pvescheduler

That was last night. The backup then ran as normal over night at 1am.

for me:
the named action resulted in a network crash (systemd-networkd: "...assertion 'a' failed...").
Cluster-Node had to be rebootet, thereafter all worked properly again....

 

[dcsapak]

note that there is already a fix packaged on pve-no-subscription. with that it's not necessary to remove the libev-perl pacakge anymore