How-to: alert on failed or missed vzdump backups with a hook script (job-end runs even when guests fail)

[Watchgoose]

Hey,

Full disclosure, I'm the founder of Watchgoose.com (cron and heartbeat monitoring platform), so I have an interest here.

I wanted a vzdump hook script that reports start, success and failure to a ping-URL service and I think I found one thing that trips people up (it came up in this thread): job-end is also called when some guests failed. If one guest backup fails vzdump calls backup-abort for that guest, carries on with the rest and still finishes with job-end. The job-abort is only used when the whole job dies so a hook that sends "success" on job-end reports success for a partly failed job.

The script below remembers every backup-abort and sends a failure at job-end instead. It works with any service that has start, success and fail URLs (the example uses Watchgoose URLs naturally).

/usr/local/bin/watchgoose-vzdump-hook.sh:

#!/bin/sh
# vzdump hook script: report backup jobs to Watchgoose.
# Install: copy to /usr/local/bin/, chmod +x, then add
#   script: /usr/local/bin/watchgoose-vzdump-hook.sh
# to /etc/vzdump.conf (all jobs) or to a single job in /etc/pve/jobs.cfg.

# vzdump clears the environment before calling hooks, so set PATH ourselves.
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH

URL="https://watchgoose.com/ping/your-check-uuid"
STATE="/run/watchgoose-vzdump.failed"

ping_wg() {
    # Never let a monitoring problem fail the backup.
    curl -fsS -m 10 --retry 3 -o /dev/null "$@" || true
}

case "$1" in
    job-init|job-start)
        # job-init only exists on newer PVE versions; a second start ping is harmless.
        rm -f "$STATE"
        ping_wg "$URL/start"
        ;;
    backup-abort)
        # $2 = mode, $3 = guest ID. vzdump carries on with the next guest.
        echo "Backup of $VMTYPE $3 failed on $(hostname)" >> "$STATE"
        ;;
    job-end)
        # job-end also runs when some guests failed, so check our own record.
        if [ -s "$STATE" ]; then
            ping_wg --data-binary @"$STATE" "$URL/fail"
        else
            ping_wg --data-raw "vzdump job finished on $(hostname)" "$URL"
        fi
        rm -f "$STATE"
        ;;
    job-abort)
        ping_wg --data-raw "vzdump job aborted on $(hostname)" "$URL/fail"
        rm -f "$STATE"
        ;;
esac

exit 0

Make it executable:

chmod +x /usr/local/bin/watchgoose-vzdump-hook.sh

Then add it for all jobs in /etc/vzdump.conf:

script: /usr/local/bin/watchgoose-vzdump-hook.sh

or add the same script: line to a single job in /etc/pve/jobs.cfg (vzdump also takes --script on the command line).

A few things I learned on the way:

• the hook gets almost no environment (vzdump clears it), so the script sets PATH itself. That is the usual cause of "curl: not found" in hook scripts
• it always exits 0, because a non-zero exit in a job phase fails the whole backup job
• in a cluster, each node runs its own part of the job and calls the hook itself, so use one check (one URL) per node
• vzdump runs one job at a time per node, so a single state file in /run is enough
• if vzdump never runs (timer problem, node down and so on) nothing arrives and the check alerts when it is overdue. That is the main reason to use a heartbeat rather than email alone

How I tested it: I don't have a PVE box available for this at the moment, so I ran the script with the exact arguments and environment that PVE/VZDump.pm passes (job-init, job-start, backup-start/backup-end/backup-abort/log-end with mode and VMID, job-end, job-abort; STOREID, DUMPDIR, VMTYPE, HOSTNAME, TARGET, LOGFILE; empty PATH). I checked what arrived for a clean run, one failed guest, an aborted job, and an older version without job-init. If you run it on a real node, I would love to hear how it went, especially with PBS storage and cluster jobs.

Also, Watchgoose has a support plan for Open Source maintainers (Open Source Program - Watchgoose).

I am happy to answer any questions.

Cheers!