Problem with pvescheduler

mztiq · May 11, 2023

Hey there,

I configured few backup tasks (Datacenter --> Backup) and they worked fine for months but since a few weeks those backup tasks are not running automatically at all.
When I manually start them they work fine.

I'm not exactly sure when these problems started but I had a hardware failure of my NUC a few month ago where I just replaced the NUC and kept the SSD with the Proxmox installation, could that be the source of some problems?

I also noticed, that none of the Task Logs show information anymore (no matter what task it is, see attachment "task_log.png") and the `pvescheduler` service is generating weird syslog entries too.

Code:

● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-11 14:01:30 CEST; 9min ago
    Process: 1829565 ExecStart=/usr/bin/pvescheduler start (code=exited, status=0/SUCCESS)
   Main PID: 1829593 (pvescheduler)
      Tasks: 1 (limit: 38087)
     Memory: 101.0M
        CPU: 656ms
     CGroup: /system.slice/pvescheduler.service
             └─1829593 pvescheduler

May 11 14:09:00 proxmox pvescheduler[1838190]: could not update job state, skipping - no such task
May 11 14:09:00 proxmox pvescheduler[1838190]: could not update job state, skipping - no such task
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task

Code:

[...]
May 11 14:06:00 proxmox pvescheduler[1834793]: could not update job state, skipping - no such task#012
May 11 14:06:00 proxmox pvescheduler[1834793]: could not update job state, skipping - no such task#012
May 11 14:06:00 proxmox pvescheduler[1834793]: could not update job state, skipping - no such task#012
May 11 14:06:00 proxmox pvescheduler[1834793]: could not update job state, skipping - no such task#012
May 11 14:07:00 proxmox pvescheduler[1835914]: could not update job state, skipping - no such task#012
May 11 14:07:00 proxmox pvescheduler[1835914]: could not update job state, skipping - no such task#012
May 11 14:07:00 proxmox pvescheduler[1835914]: could not update job state, skipping - no such task#012
May 11 14:07:00 proxmox pvescheduler[1835914]: could not update job state, skipping - no such task#012
May 11 14:08:00 proxmox pvescheduler[1837048]: could not update job state, skipping - no such task#012
May 11 14:08:00 proxmox pvescheduler[1837048]: could not update job state, skipping - no such task#012
May 11 14:08:00 proxmox pvescheduler[1837048]: could not update job state, skipping - no such task#012
May 11 14:08:00 proxmox pvescheduler[1837048]: could not update job state, skipping - no such task#012
May 11 14:09:00 proxmox pvescheduler[1838190]: could not update job state, skipping - no such task#012
May 11 14:09:00 proxmox pvescheduler[1838190]: could not update job state, skipping - no such task#012
May 11 14:09:00 proxmox pvescheduler[1838190]: could not update job state, skipping - no such task#012
May 11 14:09:00 proxmox pvescheduler[1838190]: could not update job state, skipping - no such task#012
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task#012
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task#012
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task#012
May 11 14:10:00 proxmox pvescheduler[1839370]: could not update job state, skipping - no such task#012
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task#012
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task#012
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task#012
May 11 14:11:00 proxmox pvescheduler[1840487]: could not update job state, skipping - no such task#012
May 11 14:12:00 proxmox pvescheduler[1841611]: could not update job state, skipping - no such task#012
May 11 14:12:00 proxmox pvescheduler[1841611]: could not update job state, skipping - no such task#012
May 11 14:12:00 proxmox pvescheduler[1841611]: could not update job state, skipping - no such task#012
May 11 14:12:00 proxmox pvescheduler[1841611]: could not update job state, skipping - no such task#012
May 11 14:13:00 proxmox pvescheduler[1842717]: could not update job state, skipping - no such task#012
May 11 14:13:00 proxmox pvescheduler[1842717]: could not update job state, skipping - no such task#012
May 11 14:13:00 proxmox pvescheduler[1842717]: could not update job state, skipping - no such task#012
May 11 14:13:00 proxmox pvescheduler[1842717]: could not update job state, skipping - no such task#012
May 11 14:14:00 proxmox pvescheduler[1843837]: could not update job state, skipping - no such task#012
May 11 14:14:00 proxmox pvescheduler[1843837]: could not update job state, skipping - no such task#012
May 11 14:14:00 proxmox pvescheduler[1843837]: could not update job state, skipping - no such task#012
May 11 14:14:00 proxmox pvescheduler[1843837]: could not update job state, skipping - no such task#012
May 11 14:15:00 proxmox pvescheduler[1844984]: could not update job state, skipping - no such task#012
May 11 14:15:00 proxmox pvescheduler[1844984]: could not update job state, skipping - no such task#012
May 11 14:15:00 proxmox pvescheduler[1844984]: could not update job state, skipping - no such task#012
May 11 14:15:00 proxmox pvescheduler[1844984]: could not update job state, skipping - no such task#012
May 11 14:16:00 proxmox pvescheduler[1846154]: could not update job state, skipping - no such task#012
May 11 14:16:00 proxmox pvescheduler[1846154]: could not update job state, skipping - no such task#012
May 11 14:16:00 proxmox pvescheduler[1846154]: could not update job state, skipping - no such task#012
May 11 14:16:00 proxmox pvescheduler[1846154]: could not update job state, skipping - no such task#012

I even deleted all the backup tasks (Datacenter --> Backup) which temporarily fixed the problem (the tasks were running automatically after their schedule for one or two days) but the syslog entry `could not update job state, skipping - no such task#012` stayed the same.

Code:

proxmox-ve: 7.4-1 (running kernel: 5.15.107-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.107-1-pve: 5.15.107-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-1
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.6
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

Any hints on how to fix this would be highly appreciated.

Chris · May 11, 2023

Hi,
state files for jobs are stored in /var/lib/pve-manager/jobs. What contents do you have there? Seems like a job file is still present but no tasks for it can be found.

mztiq · May 11, 2023

Hi Chris,

here's the output:

vzdump-backup-047c80bc-dbcb.json vzdump-backup-212be15d-2dfa.json vzdump-backup-46ca95ab-174b.json vzdump-backup-5e423457-2249.json vzdump-backup-98700bff-aefd.json

I've set up 5 Proxmox backup jobs and there are 5 *.json-files so this seems about right, doens't it?
If I'd delete these *.json-files would the corresponding backup job in Datacenter --> Backup disappear too and do you have any idea why the Task Logs show no output too? (see attached screenshot in the original post).

Chris · May 11, 2023

mztiq said:
Hi Chris,

here's the output:

vzdump-backup-047c80bc-dbcb.json vzdump-backup-212be15d-2dfa.json vzdump-backup-46ca95ab-174b.json vzdump-backup-5e423457-2249.json vzdump-backup-98700bff-aefd.json

I've set up 5 Proxmox backup jobs and there are 5 *.json-files so this seems about right, doens't it?
If I'd delete these *.json-files would the corresponding backup job in Datacenter --> Backup disappear too and do you have any idea why the Task Logs show no output too? (see attached screenshot in the original post).

The jobs state files should be fine, the corresponding config for the scheduled tasks is located at /etc/pve/jobs.cfg.

But you are probably missing the task files/directories and logs under /var/log/pve/tasks. What is the output of ls -la /var/log/pve/tasks?

mztiq · May 11, 2023

Hey Chris,

here's the Output of ls -la /var/log/pve/tasks:

Code:

total 340
drwxr-xr-x 18 root root  4096 May 11 14:05 .
drwxr-xr-x  3 root root  4096 Oct 23  2021 ..
drwxr-xr-x  2 root root 16384 May 10 15:42 0
drwxr-xr-x  2 root root 16384 May 10 15:42 1
drwxr-xr-x  2 root root 16384 May 10 15:42 2
drwxr-xr-x  2 root root 16384 May 10 15:42 3
drwxr-xr-x  2 root root 16384 May 10 15:42 4
drwxr-xr-x  2 root root 16384 May 11 05:43 5
drwxr-xr-x  2 root root 12288 May  9 01:38 6
drwxr-xr-x  2 root root 12288 May  5 05:15 7
drwxr-xr-x  2 root root 12288 Apr 29 03:57 8
drwxr-xr-x  2 root root 16384 May  8 04:54 9
drwxr-xr-x  2 root root 20480 May 11 13:50 A
-rw-r--r--  1 root root  1922 May 11 14:05 active
-rw-r--r--  1 root root     0 Oct 23  2021 .active.lock
drwxr-xr-x  2 root root 20480 May 10 15:42 B
drwxr-xr-x  2 root root 12288 May 10 15:42 C
drwxr-xr-x  2 root root 16384 May  8 04:54 D
drwxr-xr-x  2 root root 12288 May  8 04:54 E
drwxr-xr-x  2 root root 12288 May  5 05:15 F
-rw-r--r--  1 root root 19271 May 11 14:05 index
-rw-r--r--  1 root root 50135 Apr  5 21:53 index.1

Chris · May 12, 2023

Okay, so far all seems fine, the json state file probably reference incorrect/nonexisting UPID task files. So by running

Bash:

rm /var/lib/pve-manager/jobs/*.json

you remove these and they will be recreated.

Note that old tasks will still have no tasklog but new ones should.

mztiq · May 12, 2023

Chris said:
Okay, so far all seems fine, the json state file probably reference incorrect/nonexisting UPID task files. So by running

Bash:

rm /var/lib/pve-manager/jobs/*.json

you remove these and they will be recreated.

Note that old tasks will still have no tasklog but new ones should.

Hey Chris,

seems like this removed the could not update job state, skipping - no such task#012 syslog errors, I'll keep an eye on it and check if future tasks have a tasklog again.

I still have no idea why my scheduled backup tasks are not running like they're scheduled.
Like I already mentioned I can run them manually just fine but they're not running automatically.
As you can see in the attached screenshots, the "Next Run" date is correct but the tasks just don't automatically start as they used to.

I really appreciate your help.

Chris · May 12, 2023

mztiq said:
Hey Chris,

seems like this removed the could not update job state, skipping - no such task#012 syslog errors, I'll keep an eye on it and check if future tasks have a tasklog again.

I still have no idea why my scheduled backup tasks are not running like they're scheduled.
Like I already mentioned I can run them manually just fine but they're not running automatically.
As you can see in the attached screenshots, the "Next Run" date is correct but the tasks just don't automatically start as they used to.

I really appreciate your help.

Oaky, what's the output of

Bash:

systemctl status pve-scheduler.service
ps aux | grep pvescheduler

mztiq · May 12, 2023

Chris said:
Oaky, what's the output of

Bash:

systemctl status pve-scheduler.service ps aux | grep pvescheduler

Are you sure you meant pve-scheduler.service and not pvescheduler.service?

Here's the output of pve-scheduler.service:

Bash:

Unit pve-scheduler.service could not be found.

Here's pvescheduler.service:

Bash:

● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-11 14:01:30 CEST; 1 day 1h ago
    Process: 1829565 ExecStart=/usr/bin/pvescheduler start (code=exited, status=0/SUCCESS)
   Main PID: 1829593 (pvescheduler)
      Tasks: 1 (limit: 38087)
     Memory: 80.1M
        CPU: 21.327s
     CGroup: /system.slice/pvescheduler.service
             └─1829593 pvescheduler

And ps aux | grep pvescheduler:

Bash:

root     1829593  0.0  0.2 334076 80024 ?        Ss   May11   0:05 pvescheduler
root     3585579  0.0  0.0   6240  2404 pts/0    S+   15:49   0:00 grep pvescheduler

Chris · May 12, 2023

mztiq said:

Are you sure you meant pve-scheduler.service and not pvescheduler.service?

Here's the output of pve-scheduler.service:

Bash:

Unit pve-scheduler.service could not be found.

Here's pvescheduler.service:

Bash:

● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-11 14:01:30 CEST; 1 day 1h ago
    Process: 1829565 ExecStart=/usr/bin/pvescheduler start (code=exited, status=0/SUCCESS)
   Main PID: 1829593 (pvescheduler)
      Tasks: 1 (limit: 38087)
     Memory: 80.1M
        CPU: 21.327s
     CGroup: /system.slice/pvescheduler.service
             └─1829593 pvescheduler

And ps aux | grep pvescheduler:

Bash:

root     1829593  0.0  0.2 334076 80024 ?        Ss   May11   0:05 pvescheduler
root     3585579  0.0  0.0   6240  2404 pts/0    S+   15:49   0:00 grep pvescheduler

Ah yes,
sorry that was a typo... Okay so the scheduler is fine, but all of the jobs have a next run in the future.

Please create a test job which should run within a reasonable time to see if the cleanup of the job state files also fixed the issue with the jobs not executing as required.

mztiq · May 12, 2023

Chris said:
Ah yes,
sorry that was a typo... Okay so the scheduler is fine, but all of the jobs have a next run in the future.

Please create a test job which should run within a reasonable time to see if the cleanup of the job state files also fixed the issue with the jobs not executing as required.

The test job I just created started fine, I'll check if the existing tasks (some of them should run this night) work again too.
Thanks again so much for your help!

mztiq · May 15, 2023

After keeping my eye on the scheduled backup tasks since friday I can tell, that the tasks start automatically again but the task log still seems to disappear a few hours or maybe a day after the task was running.

After a task ended successfully I can see the task log for a few hours but the next day the log always disappears and I'm getting the message unable to open file - No such file or directory (see Screenshot of my initial post.

Oh and sadly the syslog error could not update job state, skipping - no such task#012 is back again too.
Seems like the fix chris suggested was not permanent, any Ideas?

Chris · May 16, 2023

mztiq said:
After keeping my eye on the scheduled backup tasks since friday I can tell, that the tasks start automatically again but the task log still seems to disappear a few hours or maybe a day after the task was running.

After a task ended successfully I can see the task log for a few hours but the next day the log always disappears and I'm getting the message unable to open file - No such file or directory (see Screenshot of my initial post.

Oh and sadly the syslog error could not update job state, skipping - no such task#012 is back again too.
Seems like the fix chris suggested was not permanent, any Ideas?

Then something external is interfering with your task logs. That is also what causes the no such task error. Do you have some sort of log rotation configured? Is the /var/log directory a mount point?

mztiq · May 16, 2023

Hey Chris,

to summarize the situation right now:

Task from this morning at 2:54 am shows unable to open file - No such file or directory
Tasks from this morning at 3:12 am, 4:57 am and 5:42 am show the task log correctly (not sure for how long tho)

/var/log is not a mount point and I dont have any special log rotation configured (at least not that I know of).
Here are some "default values" for the log rotation, not sure if any of this could be interesting in this context:

Code:

logrotate state -- version 2
"/var/log/syslog" 2023-5-14-0:0:34
"/var/log/dpkg.log" 2023-5-1-0:0:3
"/var/log/auth.log" 2023-5-14-0:0:34
"/var/log/corosync/corosync.log" 2022-10-26-0:0:0
"/var/log/apt/term.log" 2023-5-1-0:0:3
"/var/log/apt/history.log" 2023-5-1-0:0:3
"/var/log/alternatives.log" 2023-5-1-0:0:3
"/var/log/debug" 2023-5-12-0:0:34
"/var/log/mail.log" 2023-5-14-0:0:34
"/var/log/glusterfs/*.log" 2022-10-26-0:0:0
"/var/log/kern.log" 2023-5-14-0:0:34
"/var/log/chrony/*.log" 2022-10-26-0:0:0
"/var/log/wtmp" 2021-10-24-0:0:0
"/var/log/daemon.log" 2023-5-14-0:0:34
"/var/log/mail.warn" 2022-10-20-0:0:28
"/var/log/pve-firewall.log" 2023-5-16-0:0:34
"/var/log/pveproxy/access.log" 2023-5-16-0:0:34
"/var/log/btmp" 2023-5-1-0:0:3
"/var/log/lpr.log" 2022-10-26-0:0:0
"/var/log/mail.err" 2022-10-20-0:0:28
"/var/log/ceph/*.log" 2022-10-26-0:0:0
"/var/log/user.log" 2023-4-16-0:0:2
"/var/log/mail.info" 2023-5-14-0:0:34
"/var/log/glusterfs/bricks/*.log" 2022-10-26-0:0:0
"/var/log/cron.log" 2022-10-26-0:0:0
"/var/log/messages" 2023-5-14-0:0:34

Bash:

# see "man logrotate" for details

# global options do not affect preceding include directives

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may also be configured here.

Code:

/var/log/pveproxy/access.log {
        rotate 7
        daily
        missingok
        compress
        delaycompress
        notifempty
        create 640 www-data www-data
        sharedscripts
        postrotate
                /bin/systemctl try-reload-or-restart pveproxy.service
                /bin/systemctl try-reload-or-restart spiceproxy.service
        endscript
}

Bash:

/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript
}

I hope any of this helps, let me know if I can provide any further information.

yo-ga · 2024-10-21T21:41:16+0200

You proxmox is solo node or part of cluster ?
If you found a solution, share it with us

P.S. I have a similar problem, but on a cluster of 2 nodes, on a slave node

Search

Search

Problem with pvescheduler

mztiq

New Member

Attachments

Chris

Proxmox Staff Member

mztiq

New Member

Chris

Proxmox Staff Member

mztiq

New Member

Chris

Proxmox Staff Member

mztiq

New Member

Attachments

Chris

Proxmox Staff Member

mztiq

New Member

Chris

Proxmox Staff Member

mztiq

New Member

mztiq

New Member

Chris

Proxmox Staff Member

mztiq

New Member

yo-ga

New Member