[PVE 7.4-3] Backups jobs not run at schedule time

D

duckpage

New Member
Mar 28, 2023
4
0
1
Hi, I made a fresh install of proxmox 7.4 and I configurated backups jobs under
Code: 
Datacenter->Backups
in pve-manager. After that I checked that in the file
Code: 
/etc/pve/jobs.cfg
that all is correct

Code: 
# The content of my file
vzdump: backup-63450d34-7597
    schedule 21:00
    compress zstd
    enabled 1
    mailnotification always
    mailto webmaster@f...a.eu
    mode snapshot
    notes-template {{guestname}}
    repeat-missed 0
    storage pve-nfs-backups
    vmid 254,250

But at 21:00 nothing happend. I tried to delete and recreate the config, create a new one with a new schedule time but the same result: backups dose'nt start. I need to manual start backup jobs with the button "Run now".

What could be the problem?

Thanks
 
Last edited:
Hi,
please check the status of the scheduler service by running systemctl status pvescheduler.service
 
duckpage said:
Is active. I already tried to restart it but the same.
Click to expand...
Could you post the journal related to the scheduler service, journalctl -b -u pvescheduler.service
 
Some minute a go restart the server and the output of the command is:

Code: 
-- Journal begins at Thu 2023-03-16 09:43:35 CET, ends at Tue 2023-03-28 15:10:51 CEST. --
mar 28 14:12:35 pts00 systemd[1]: Starting Proxmox VE scheduler...
mar 28 14:12:35 pts00 pvescheduler[55355]: starting server
mar 28 14:12:35 pts00 systemd[1]: Started Proxmox VE scheduler.

I can try to set a new backup and wait for the schedule time and send again the output.
 
duckpage said:
Some minute a go restart the server and the output of the command is:

Code: 
-- Journal begins at Thu 2023-03-16 09:43:35 CET, ends at Tue 2023-03-28 15:10:51 CEST. --
mar 28 14:12:35 pts00 systemd[1]: Starting Proxmox VE scheduler...
mar 28 14:12:35 pts00 pvescheduler[55355]: starting server
mar 28 14:12:35 pts00 systemd[1]: Started Proxmox VE scheduler.

I can try to set a new backup and wait for the schedule time and send again the output.
Click to expand...
I mean, from what I see you rebootet the node, so maybe get also the output from before the reboot, e.g. journalctl --since 2023-03-26 -u pvescheduler.service
 
Can report the same here / tried to sanitize the exports a bit
Backups are set to daily schedules but not running since the last upgrade

root@pve~ # uptime
12:09:12 up 260 days, 19:21, 3 users, load average: 3.86, 3.78, 3.43

root@twist ~ # pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.3-3
pve-kernel-5.4: 6.4-18
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.4.189-2-pve: 5.4.189-2
pve-kernel-4.4.67-1-pve: 4.4.67-92
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-3
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.3
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20221111-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.11-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1


root@pve ~ # systemctl status pvescheduler.service
● pvescheduler.service - Proxmox VE scheduler
Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-07-10 16:49:42 -03; 8 months 17 days ago
Process: 3706630 ExecReload=/usr/bin/pvescheduler restart (code=exited, status=0/SUCCESS)
Main PID: 2250 (pvescheduler)
Tasks: 3 (limit: 154382)
Memory: 113.7M
CPU: 2w 16h 25min 17.254s
CGroup: /system.slice/pvescheduler.service
└─2250 pvescheduler

Mar 26 01:53:57 pve pvescheduler[3499020]: INFO: Finished Backup of VM 210 (00:09:33)
Mar 26 01:53:57 pve pvescheduler[3499020]: INFO: Backup job finished successfully
Mar 26 01:53:58 pve sendmail[3543753]: 32Q4rwdJ3543753: from=proxmox-pve@company.com, size=207652, class=0, nrcpts=1, msgid=<202303260453.32Q4rwdJ3543753@pve.assekuransa.com>, bodytype=8BITMIME, relay=root@localhost
Mar 26 01:53:58 pve sendmail[3543753]: 32Q4rwdJ3543753: to=sysadmins@company.com, ctladdr=proxmox-pve@assekuransa.com (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=237652, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (32Q4rw443543760 Message accepted for delivery)
Mar 26 11:53:31 pve systemd[1]: Reloading Proxmox VE scheduler.
Mar 26 11:53:31 pve pvescheduler[3706630]: send HUP to 2250
Mar 26 11:53:31 pve systemd[1]: Reloaded Proxmox VE scheduler.
Mar 26 11:53:32 pve pvescheduler[2250]: received signal HUP
Mar 26 11:53:32 pve pvescheduler[2250]: server shutdown (restar


root@pve ~ # journalctl -b -u pvescheduler.service | tail -n50
Mar 25 01:18:44 pve pvescheduler[3103049]: INFO: Starting Backup of VM 115 (qemu)
Mar 25 01:18:46 pve pvescheduler[3103049]: INFO: Finished Backup of VM 115 (00:00:02)
Mar 25 01:18:46 pve pvescheduler[3103049]: INFO: Starting Backup of VM 130 (qemu)
Mar 25 01:22:38 pve pvescheduler[3103049]: INFO: Finished Backup of VM 130 (00:03:52)
Mar 25 01:22:38 pve pvescheduler[3103049]: INFO: Starting Backup of VM 140 (qemu)
Mar 25 01:31:52 pve pvescheduler[3103049]: INFO: Finished Backup of VM 140 (00:09:14)
Mar 25 01:31:52 pve pvescheduler[3103049]: INFO: Starting Backup of VM 150 (qemu)
Mar 25 01:34:23 pve pvescheduler[3103049]: INFO: Finished Backup of VM 150 (00:02:31)
Mar 25 01:34:23 pve pvescheduler[3103049]: INFO: Starting Backup of VM 201 (qemu)
Mar 25 01:42:09 pve pvescheduler[3103049]: INFO: Finished Backup of VM 201 (00:07:46)
Mar 25 01:42:09 pve pvescheduler[3103049]: INFO: Starting Backup of VM 205 (qemu)
Mar 25 01:45:02 pve pvescheduler[3103049]: INFO: Finished Backup of VM 205 (00:02:53)
Mar 25 01:45:03 pve pvescheduler[3103049]: INFO: Starting Backup of VM 210 (qemu)
Mar 25 01:53:31 pve pvescheduler[3103049]: INFO: Finished Backup of VM 210 (00:08:28)
Mar 25 01:53:31 pve pvescheduler[3103049]: INFO: Backup job finished successfully
Mar 25 01:53:32 pve sendmail[3147538]: 32P4rWu73147538: from=proxmox-pve@mycompany.com, size=207952, class=0, nrcpts=1, msgid=<202303250453.32P4rWu73147538@pve.mycompany.com>, bodytype=8BITMIME, relay=root@localhost
Mar 25 01:53:32 pve sendmail[3147538]: 32P4rWu73147538: to=sysadmins@mycompany.com, ctladdr=proxmox-pve@mycompany.com (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=237952, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (32P4rWpU3147539 Message accepted for delivery)
Mar 26 00:00:07 pve pvescheduler[3498982]: <root@pam> starting task UPID:pve:0035640C:8505380B:641FB535:vzdump::root@pam:
Mar 26 00:00:10 pve pvescheduler[3499020]: INFO: starting new backup job: vzdump 100 102 106 107 115 130 140 150 201 205 210 --node pve --mailto sysadmins@mycompany.com --compress zstd --storage Backuppve --notes-template '{{guestname}}, {{node}}, {{vmid}}' --mode snapshot --mailnotification always --quiet 1
Mar 26 00:00:11 pve pvescheduler[3499020]: INFO: Starting Backup of VM 100 (qemu)
Mar 26 00:11:57 pve pvescheduler[3499020]: INFO: Finished Backup of VM 100 (00:11:47)
Mar 26 00:11:57 pve pvescheduler[3499020]: INFO: Starting Backup of VM 102 (qemu)
Mar 26 00:44:09 pve pvescheduler[3499020]: INFO: Finished Backup of VM 102 (00:32:12)
Mar 26 00:44:09 pve pvescheduler[3499020]: INFO: Starting Backup of VM 106 (qemu)
Mar 26 00:50:57 pve pvescheduler[3499020]: INFO: Finished Backup of VM 106 (00:06:48)
Mar 26 00:50:57 pve pvescheduler[3499020]: INFO: Starting Backup of VM 107 (qemu)
Mar 26 01:18:28 pve pvescheduler[3499020]: INFO: Finished Backup of VM 107 (00:27:31)
Mar 26 01:18:28 pve pvescheduler[3499020]: INFO: Starting Backup of VM 115 (qemu)
Mar 26 01:18:30 pve pvescheduler[3499020]: INFO: Finished Backup of VM 115 (00:00:02)
Mar 26 01:18:30 pve pvescheduler[3499020]: INFO: Starting Backup of VM 130 (qemu)
Mar 26 01:21:50 pve pvescheduler[3499020]: INFO: Finished Backup of VM 130 (00:03:20)
Mar 26 01:21:50 pve pvescheduler[3499020]: INFO: Starting Backup of VM 140 (qemu)
Mar 26 01:31:03 pve pvescheduler[3499020]: INFO: Finished Backup of VM 140 (00:09:13)
Mar 26 01:31:03 pve pvescheduler[3499020]: INFO: Starting Backup of VM 150 (qemu)
Mar 26 01:33:41 pve pvescheduler[3499020]: INFO: Finished Backup of VM 150 (00:02:38)
Mar 26 01:33:41 pve pvescheduler[3499020]: INFO: Starting Backup of VM 201 (qemu)
Mar 26 01:41:33 pve pvescheduler[3499020]: INFO: Finished Backup of VM 201 (00:07:52)
Mar 26 01:41:33 pve pvescheduler[3499020]: INFO: Starting Backup of VM 205 (qemu)
Mar 26 01:44:24 pve pvescheduler[3499020]: INFO: Finished Backup of VM 205 (00:02:51)
Mar 26 01:44:24 pve pvescheduler[3499020]: INFO: Starting Backup of VM 210 (qemu)
Mar 26 01:53:57 pve pvescheduler[3499020]: INFO: Finished Backup of VM 210 (00:09:33)
Mar 26 01:53:57 pve pvescheduler[3499020]: INFO: Backup job finished successfully
Mar 26 01:53:58 pve sendmail[3543753]: 32Q4rwdJ3543753: from=proxmox-pve@mycompany.com, size=207652, class=0, nrcpts=1, msgid=<202303260453.32Q4rwdJ3543753@pve.mycompany.com>, bodytype=8BITMIME, relay=root@localhost
Mar 26 01:53:58 pve sendmail[3543753]: 32Q4rwdJ3543753: to=sysadmins@mycompany.com, ctladdr=proxmox-pve@mycompany.com (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=237652, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (32Q4rw443543760 Message accepted for delivery)
Mar 26 11:53:31 pve systemd[1]: Reloading Proxmox VE scheduler.
Mar 26 11:53:31 pve pvescheduler[3706630]: send HUP to 2250
Mar 26 11:53:31 pve systemd[1]: Reloaded Proxmox VE scheduler.
Mar 26 11:53:32 pve pvescheduler[2250]: received signal HUP
Mar 26 11:53:32 pve pvescheduler[2250]: server shutdown (restart)
Mar 26 11:53:33 pve pvescheduler[2250]: restarting server


root@pve ~ # cat /etc/vzdump.conf
# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#performance: max-workers=N
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#prune-backups: keep-INTERVAL=N[,...]
#script: FILENAME
#exclude-path: PATHLIST
#pigz: N
#notes-template: {{guestname}}
root@pve~#cat /etc/pve/jobs.cfg
vzdump: 9eeb2f40a9781b3fcc93725baf6dc1522aa6a2fe:1
schedule 00:00
compress zstd
enabled 1
mailnotification always
mailto sysadmins@comapny.com
mode snapshot
node pve
notes-template {{guestname}}, {{node}}, {{vmid}}
quiet 1
storage Backuppve
vmid 100,102,106,107,115,130,140,150,201,205,210

vzdump: 4da4e9735de64d91fe3e474d5bf1ea6c15521a64:1
schedule 00:00
all 1
compress zstd
enabled 1
mailnotification always
mailto sysadmins@comapny.com
mode snapshot
node pve02
notes-template {{guestname}}, {{node}}, {{vmid}}
quiet 1
storage Backuppve02
 
Last edited:
arturobaldo said:
Mar 26 11:53:32 pve pvescheduler[2250]: received signal HUP
Mar 26 11:53:32 pve pvescheduler[2250]: server shutdown (restart)
Mar 26 11:53:33 pve pvescheduler[2250]: restarting server
Click to expand...
HI, from these logs it seems like the server never manages to restart. Could you provide the output of ps auxwf and strace -yyttT -f -s 512 -p 2250 as well as the full journal around that time, so journalctl --since '2023-03-26 11:00' --until '2023-03-26 13:00'
 
arturobaldo said:
Sure thing, here they are
Click to expand...
So the reason why pvescheduler is not restarting is that there are 2 zombie processes with PID 3706701 and 3711164 which are not clean up. You can try to kill them with kill -9 3711164 3706701. Where there any backups running at that time? Could you also provide the task log for the last backup task?
 
I killed them and then was able to restart the service sucessfully.
I Can confirm that fixed the issue. I tried by changing the scheduled start time to a couple minutes ago and the job was sucessfully triggered.
Thanks a lot for your help
 
Awesome, many thanks, I ensured to apply the update on all my nodes.
I guess the OP can mark this as FIXED for future reference
 
kill -9 ... not works.
But for me works:
killall -9 pvescheduler
systemctl start pvescheduler.service
 
treki said:
kill -9 ... not works.
Click to expand...
Of course not, you will have to send the kill signal to the right process id, which might be different on your system.
 
Of course, that's what I did.
But the tasks remained.
killall... helped then.
 
Seams a bug in a upgraded package... Same here on three nodes... reboot fixed it.

Code: 
root@host03:~# systemctl status pvescheduler.service
● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-02-05 22:23:47 CET; 6 months 22 days ago
    Process: 939925 ExecReload=/usr/bin/pvescheduler restart (code=exited, status=0/SUCCESS)
   Main PID: 1370 (pvescheduler)
      Tasks: 3 (limit: 154317)
     Memory: 12.3G
        CPU: 6h 41min 17.285s
     CGroup: /system.slice/pvescheduler.service
             └─1370 pvescheduler

Jul 31 13:21:37 host03 systemd[1]: Reloading Proxmox VE scheduler.
Jul 31 13:21:38 host03 pvescheduler[939925]: send HUP to 1370
Jul 31 13:21:38 host03 pvescheduler[1370]: received signal HUP
Jul 31 13:21:38 host03 pvescheduler[1370]: server shutdown (restart)
Jul 31 13:21:38 host03 systemd[1]: Reloaded Proxmox VE scheduler.
Jul 31 13:21:39 host03 pvescheduler[1370]: restarting server
Warning: journal has been rotated since unit was started, output may be incomplete.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom