Monitoring of backup jobs and alerting on actual issues

[sbarmen]

Hey all, I have been playing with the best way of monitoring my backup jobs. The main problem is that I want to reduce the noise, not get "alerts" for "everything is fine". I hade email reports set up, all fine and well, but it gets very noisy. I even had a local LLM running to get it to alert me when I needed to take action, works OK, but I do not trust it fully.

After discussing and researching I ended up using Uptime Kuma to get a "only notify me when something is wrong" approach. More or less it will tell me if a job failed or partially failed (single VM or container failed). This works pretty good and I get notified on Signal when something needs my attention. All well and done, but is this the best way? My approach has been to hook a script using /etc/vzdump.conf on every node. Uptime Kuma goes amber/red if no alert is received in X minutes, and if it receives a push then the script adds a status (up/down/problem). This gives me great visibility of all backups running in the cluster, but needs to run on all my nodes.


Full setup instructions here: https://www.sbarmen.no/posts/uptime-kuma-monitor-proxmox-backup/

The problem; this works only for the cluster, not for other hosts. I also use PBS to backup some physical machines, and also some config on the cluster nodes themselves. Is there a way to hook into all jobs on PBS with a similar approach with a script that could run a push notification on events in the same way? Any thoughts or ideas? Is there a best practice for this that I have missed?

 

[lubosr]

Short answer: PBS won't tell you about backups the way you want, but it does know about them. Just not in the notification system.

Everything below is from my lab, so swap in your own host, user and datastore. The calls need a login ticket first:

PBS='https://pbs.example.com:8007'
read -rsp 'password: ' PW; echo
RESP=$(curl -sk -d 'username=backup@pbs' -d "password=$PW" \
  "$PBS/api2/json/access/ticket")
TICKET=$(jq -r .data.ticket <<<"$RESP")
CSRF=$(jq -r .data.CSRFPreventionToken <<<"$RESP")
echo "${TICKET:0:20}"

Ticket goes in a cookie, CSRF token in a header. GETs only need the cookie, so the ticket is enough for everything here. They last about two hours.



Have a look at what the notification matchers can match on:

curl -sk -b "PBSAuthCookie=$TICKET" "$PBS/api2/json/config/notifications/matcher-field-values" | jq

On 4.2.5 there's no backup in the type list. PBS reports on the maintenance it runs against backups, but not on a backup arriving, so there's nothing to send to a webhook.

The task list is different. Every backup that starts shows up there:

curl -sk -b "PBSAuthCookie=$TICKET" \
  "$PBS/api2/json/nodes/localhost/tasks?typefilter=backup&errors=1" \
| jq -r '"ERRORS only: \(.total)", (.data[] | "\(.worker_id)\t\(.status)")'

I stopped a VM backup halfway through to see what a failure looks like. It came out as backup ended but finished state is not set. and errors=1 still returned it, so the message text doesn't matter, anything that isn't OK counts:



Add since=<epoch> and you have your poller: ask for errors since the last check, push to Kuma if anything comes back. Your physical machines are in there too, proxmox-backup-client jobs land as backup tasks like everything else. You see your own tasks by default, so whatever token takes the backups can read them; Sys.Audit on /system/tasks if you want everyone's.

One thing I ran into. If a backup is refused before it starts, there's no task at all. I changed a group's ownership so the client would be rejected, ran a backup, and the count on the server stayed the same. The client reported the error, the server recorded nothing.

I only tested the ownership case, but the pattern is clear enough: the task list tells you what failed, not what never arrived. A revoked token, a client that can't reach the server, a host nobody scheduled, all silent.

So I wouldn't replace what you've got. Task API for PBS-side failures including the non-PVE hosts, and keep a heartbeat for whether anything arrived at all, which your Kuma setup already does on the cluster. It's polling rather than a push on the event, which works for Kuma but isn't the same as your vzdump hook.

Hope this helps
Lubos