Healthchecks.io - Backup Hook Script

[dj_arbz]

I am working on a hook script to integrate with a self-hosted version of Healthchecks.io when backing up my VM/LXCs to PBS.

I have almost everything how I want it, but need a way during the Job-End phase to determine if there were any errors during the backup process.
I was just going to parse the log file for the task in /var/log/pve/tasks but it appears that it is not written there yet.

Is there a way to discover the overall job status in the Job-End phase?
Either that, or where can I find the running log that I can parse?

 

[ggoller]

When the task finishes, you can find the log with all the errors in /var/log/pve/tasks. For example:

INFO: starting new backup job: vzdump 104 --quiet 1 --notes-template '{{guestname}}' --mode snapshot --mailnotification always --storage pbs-test --node luna
ERROR: Backup of VM 104 failed - unable to find VM '104'
INFO: Failed at 2023-08-04 10:00:01
INFO: Backup job finished with errors
TASK ERROR: job errors

in file (for example):
/var/log/pve/tasks/1/1/UPID:xxx:00004AE2:00060190:64CCB001:vzdump:104:xxx@pam:

 

[dj_arbz]

In my testing, this file does not exist until after the job has fully completed, aka after my hook script is called for the job-end phase.

 

[ggoller]

Yes, the log file will only be available in the log-end phase of the hook script. But the errors are also written to syslog and you can see them with sudo journalctl -u pvescheduler for example.

 

[dj_arbz]

Thanks, while not the same output, I think it could serve my purpose well.
Looking at the journal output below, I think I can grab what I want with this command.

job_log="$(journalctl --no-pager -u pvescheduler _CMDLINE="task ${TASK_ID}" | tail --bytes=100000)"

journalctl -n19 --no-pager -o verbose -u pvescheduler
-- Journal begins at Wed 2022-08-17 17:33:59 CDT, ends at Thu 2023-08-10 14:59:56 CDT. --
Thu 2023-08-10 01:00:01.126439 CDT [s=c5332958b48440889321a5215fe4911c;i=46a87a;b=71c384b4be0345c2850633b3b269bcdc;m=57a09868051;t=6028b4e82283e;x=ad8f34062cb2fec9]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _COMM=pvescheduler
    _CMDLINE=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    SYSLOG_PID=2946975
    SYSLOG_TIMESTAMP=Aug 10 01:00:01
    MESSAGE=<root@pam> starting task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_RAW=[123B blob data]
    _PID=2946975
    _SOURCE_REALTIME_TIMESTAMP=1691647201126439
Thu 2023-08-10 01:00:08.465396 CDT [s=c5332958b48440889321a5215fe4911c;i=46a881;b=71c384b4be0345c2850633b3b269bcdc;m=57a09f67c21;t=6028b4ef2240e;x=e74d7e9ee74f13a0]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    SYSLOG_PID=2946976
    SYSLOG_TIMESTAMP=Aug 10 01:00:08
    MESSAGE=INFO: starting new backup job: vzdump --exclude 3201 --mode snapshot --mailnotification failure --all 1 --quiet 1 --storage pbs --prune-backups 'keep-all=1' --script /usr/local/bin/vzdump-hook-script --mailto root@localhost --notes-template '{{guestname}} ({{vmid}}.{{node}}.{{cluster}})'
    SYSLOG_RAW=[335B blob data]
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    _SOURCE_REALTIME_TIMESTAMP=1691647208465396
Thu 2023-08-10 01:52:08.656550 CDT [s=c5332958b48440889321a5215fe4911c;i=46b538;b=71c384b4be0345c2850633b3b269bcdc;m=57ac3f0d2f4;t=6028c08ec7ae1;x=1300eba8f8446d0b]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    MESSAGE=INFO: Starting Backup of VM 3001 (lxc)
    SYSLOG_PID=2946976
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_TIMESTAMP=Aug 10 01:52:08
    SYSLOG_RAW=[83B blob data]
    _SOURCE_REALTIME_TIMESTAMP=1691650328656550
Thu 2023-08-10 01:55:10.423966 CDT [s=c5332958b48440889321a5215fe4911c;i=46b603;b=71c384b4be0345c2850633b3b269bcdc;m=57acec65fc9;t=6028c13c207b6;x=707b78cdf166c836]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    SYSLOG_PID=2946976
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_TIMESTAMP=Aug 10 01:55:10
    MESSAGE=INFO: Finished Backup of VM 3001 (00:03:02)
    SYSLOG_RAW=[88B blob data]
    _SOURCE_REALTIME_TIMESTAMP=1691650510423966
Thu 2023-08-10 01:55:21.068909 CDT [s=c5332958b48440889321a5215fe4911c;i=46b60f;b=71c384b4be0345c2850633b3b269bcdc;m=57acf68cd98;t=6028c14647585;x=72323d449caf0ce8]
    PRIORITY=6
    SYSLOG_FACILITY=3
    _UID=0
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=71c384b4be0345c2850633b3b269bcdc
    _MACHINE_ID=ea33a24ad9d742c69e342d9f9b5a8f8a
    _HOSTNAME=pve
    _TRANSPORT=syslog
    _EXE=/usr/bin/perl
    SYSLOG_IDENTIFIER=pvescheduler
    _SYSTEMD_CGROUP=/system.slice/pvescheduler.service
    _SYSTEMD_UNIT=pvescheduler.service
    _SYSTEMD_INVOCATION_ID=a6318426bc074b3bbc6680c986d27e92
    _COMM=task UPID:pve:0
    MESSAGE=INFO: Backup job finished successfully
    SYSLOG_PID=2946976
    _PID=2946976
    _CMDLINE=task UPID:pve:002CF7A0:23E4641B:64D47CE1:vzdump::root@pam:
    SYSLOG_TIMESTAMP=Aug 10 01:55:21
    SYSLOG_RAW=[83B blob data]
    _SOURCE_REALTIME_TIMESTAMP=1691650521068909