How-to: alert on failed or missed vzdump backups with a hook script (job-end runs even when guests fail)

[Watchgoose]

I'm the founder of Watchgoose (cron and heartbeat monitoring), so I have an interest here.

I wanted a vzdump hook script that reports start, success and failure to a ping-URL service, and found one thing that trips people up (it came up in this thread): job-end is also called when some guests failed. If one guest backup fails, vzdump calls backup-abort for that guest, carries on with the rest, and still finishes with job-end. job-abort is only used when the whole job dies. So a hook that sends "success" on job-end reports success for a partly failed job.

The script below remembers every backup-abort and sends a failure at job-end instead. It works with any service that has start, success and fail URLs; the example uses Watchgoose URLs.

/usr/local/bin/watchgoose-vzdump-nightly.sh (one copy per backup job, each with its own check URL):

#!/bin/sh
# vzdump hook script: report one backup job to Watchgoose.
# Use one copy of this script, with its own check URL, per backup job.

# vzdump runs hooks with an almost empty environment: set PATH explicitly
# instead of relying on the shell's default.
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH

URL="https://watchgoose.com/ping/your-check-uuid"
STATE="/run/$(basename "$0").failed"

ping_wg() {
    # Never let a monitoring problem fail the backup.
    curl -fsS -m 10 --retry 3 -o /dev/null "$@" || true
}

case "$1" in
    job-start)
        # job-start runs once vzdump holds the node's backup lock. Do not reset
        # on job-init: that phase runs before the lock, so a queued job would
        # wipe the record of a job that is still running.
        rm -f "$STATE"
        ping_wg "$URL/start"
        ;;
    backup-abort)
        # $2 = mode, $3 = guest ID. vzdump carries on with the next guest.
        echo "Backup of $VMTYPE $3 failed on $(hostname)" >> "$STATE"
        ;;
    job-end)
        # job-end also runs when some guests failed, so check our own record.
        if [ -s "$STATE" ]; then
            ping_wg --data-binary @"$STATE" "$URL/fail"
        else
            ping_wg --data-raw "vzdump job finished on $(hostname)" "$URL"
        fi
        rm -f "$STATE"
        ;;
    job-abort)
        ping_wg --data-raw "vzdump job aborted on $(hostname)" "$URL/fail"
        rm -f "$STATE"
        ;;
esac

exit 0

Make it executable:

chmod +x /usr/local/bin/watchgoose-vzdump-nightly.sh

Then set it on the backup job as root (pvesh get /cluster/backup lists the job IDs):

pvesh set /cluster/backup/backup-1a2b3c4d-5e6f --script /usr/local/bin/watchgoose-vzdump-nightly.sh

In /etc/pve/jobs.cfg that becomes an indented script property, without a colon:

vzdump: backup-1a2b3c4d-5e6f
	schedule 01:00
	script /usr/local/bin/watchgoose-vzdump-nightly.sh

script: /path (with a colon) belongs in /etc/vzdump.conf, but that applies the script to every backup on the node, so only use it if the node runs a single backup job. For another job, make another copy of the script with that job's own check URL.

Run the job once by hand (Datacenter > Backup > Run now) and check that the check receives a start and a success ping. A brand-new check only starts watching for missed runs after its first success ping.

A few things I learned on the way:

• The hook gets an almost empty environment. Debian's /bin/sh falls back to a default PATH, but setting it explicitly keeps the script independent of that. If you still see "curl: not found", check that curl is installed.
• It always exits 0, because a non-zero exit in a job phase fails the backup job.
• Use one check per backup job and per node. If two jobs share a check, one finishing can hide the other never running. In a cluster, install the script at the same path on every node, each copy with that node's own URL.
• The record starts at job-start, not job-init: vzdump runs job-init before it takes the node's backup lock, so a queued second job could otherwise wipe the first job's failures.
• If vzdump never runs (timer problem, node down), nothing arrives and the check alerts when it is overdue. That is the main reason to use a heartbeat rather than email alone.

How I tested it: I don't have a PVE box available for this, so I ran the script with the exact arguments and environment that PVE/VZDump.pm passes (job-init, job-start, backup-start/backup-end/backup-abort/log-end with mode and VMID, job-end, job-abort; STOREID, DUMPDIR, VMTYPE, HOSTNAME, TARGET, LOGFILE; empty PATH). I checked what arrived for a clean run, one failed guest, an aborted job, a second job's job-init arriving while the first job was still running, and an older version without job-init. If you run it on a real node, I'd be glad to hear how it went, especially with PBS storage and cluster jobs.

Watchgoose has a support plan for Open Source maintainers (Open Source Program - Watchgoose). I am happy to answer any questions.

Cheers!

 

[Watchgoose]

Correction to the script above, thanks to a careful review. I have updated the original post.

• The script now resets its state at job-start instead of job-init. vzdump runs job-init before it takes the node's backup lock so a second job waiting for the lock could wipe the first job failure record and turn a failed run into a success ping
• For a single job in /etc/pve/jobs.cfg, the property is an indented script /path without a colon (set it with pvesh set /cluster/backup/<job-id> --script /path). The script: /path form is only for /etc/vzdump.conf
• Use one check per backup job and node otherwise one job finishing can hide another that never ran

Cheers!