Healthchecks.io - Backup Hook Script

dj_arbz

New Member
Dec 26, 2021
11
1
3
31
I am working on a hook script to integrate with a self-hosted version of Healthchecks.io when backing up my VM/LXCs to PBS.

I have almost everything how I want it, but need a way during the Job-End phase to determine if there were any errors during the backup process.
I was just going to parse the log file for the task in /var/log/pve/tasks but it appears that it is not written there yet.

Is there a way to discover the overall job status in the Job-End phase?
Either that, or where can I find the running log that I can parse?
 
When the task finishes, you can find the log with all the errors in /var/log/pve/tasks. For example:
Code:
INFO: starting new backup job: vzdump 104 --quiet 1 --notes-template '{{guestname}}' --mode snapshot --mailnotification always --storage pbs-test --node luna
ERROR: Backup of VM 104 failed - unable to find VM '104'
INFO: Failed at 2023-08-04 10:00:01
INFO: Backup job finished with errors
TASK ERROR: job errors
in file (for example):
/var/log/pve/tasks/1/1/UPID:xxx:00004AE2:00060190:64CCB001:vzdump:104:xxx@pam:
 
Last edited:
In my testing, this file does not exist until after the job has fully completed, aka after my hook script is called for the job-end phase.
 
Yes, the log file will only be available in the log-end phase of the hook script. But the errors are also written to syslog and you can see them with sudo journalctl -u pvescheduler for example.
 
Thanks, while not the same output, I think it could serve my purpose well.
Looking at the journal output below, I think I can grab what I want with this command.
Bash:
job_log="$(journalctl --no-pager -u pvescheduler _CMDLINE="task ${TASK_ID}" | tail --bytes=100000)"

Code:
journalctl -n19 --no-pager -o verbose -u pvescheduler
-- Journal begins at Wed 2022-08-17 17:33:59 CDT, ends at Thu 2023-08-10 14:59:56 CDT. --
Thu 2023-08-10 01:00:01.126439 CDT [s=c5332958b48440889321a5215fe4911c;i=46a87a;b=71c384b4be0345c2850633b3b269bcdc;m=57a09868051;t=6028b4e82283e;x=ad8f34062cb2fec9]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _COMM=pvescheduler
    _CMDLINE=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    SYSLOG_PID=2946975
    SYSLOG_TIMESTAMP=Aug 10 01:00:01
    MESSAGE=<root@pam> starting task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_RAW=[123B blob data]
    _PID=2946975
    _SOURCE_REALTIME_TIMESTAMP=1691647201126439
Thu 2023-08-10 01:00:08.465396 CDT [s=c5332958b48440889321a5215fe4911c;i=46a881;b=71c384b4be0345c2850633b3b269bcdc;m=57a09f67c21;t=6028b4ef2240e;x=e74d7e9ee74f13a0]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    SYSLOG_PID=2946976
    SYSLOG_TIMESTAMP=Aug 10 01:00:08
    MESSAGE=INFO: starting new backup job: vzdump --exclude 3201 --mode snapshot --mailnotification failure --all 1 --quiet 1 --storage pbs --prune-backups 'keep-all=1' --script /usr/local/bin/vzdump-hook-script --mailto root@localhost --notes-template '{{guestname}} ({{vmid}}.{{node}}.{{cluster}})'
    SYSLOG_RAW=[335B blob data]
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    _SOURCE_REALTIME_TIMESTAMP=1691647208465396
Thu 2023-08-10 01:52:08.656550 CDT [s=c5332958b48440889321a5215fe4911c;i=46b538;b=71c384b4be0345c2850633b3b269bcdc;m=57ac3f0d2f4;t=6028c08ec7ae1;x=1300eba8f8446d0b]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    MESSAGE=INFO: Starting Backup of VM 3001 (lxc)
    SYSLOG_PID=2946976
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_TIMESTAMP=Aug 10 01:52:08
    SYSLOG_RAW=[83B blob data]
    _SOURCE_REALTIME_TIMESTAMP=1691650328656550
Thu 2023-08-10 01:55:10.423966 CDT [s=c5332958b48440889321a5215fe4911c;i=46b603;b=71c384b4be0345c2850633b3b269bcdc;m=57acec65fc9;t=6028c13c207b6;x=707b78cdf166c836]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    SYSLOG_PID=2946976
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_TIMESTAMP=Aug 10 01:55:10
    MESSAGE=INFO: Finished Backup of VM 3001 (00:03:02)
    SYSLOG_RAW=[88B blob data]
    _SOURCE_REALTIME_TIMESTAMP=1691650510423966
Thu 2023-08-10 01:55:21.068909 CDT [s=c5332958b48440889321a5215fe4911c;i=46b60f;b=71c384b4be0345c2850633b3b269bcdc;m=57acf68cd98;t=6028c14647585;x=72323d449caf0ce8]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    MESSAGE=INFO: Backup job finished successfully
    SYSLOG_PID=2946976
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_TIMESTAMP=Aug 10 01:55:21
    SYSLOG_RAW=[83B blob data]
    _SOURCE_REALTIME_TIMESTAMP=1691650521068909
 
  • Like
Reactions: ggoller

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!