Hey all, I have been playing with the best way of monitoring my backup jobs. The main problem is that I want to reduce the noise, not get "alerts" for "everything is fine". I hade email reports set up, all fine and well, but it gets very noisy. I even had a local LLM running to get it to alert me when I needed to take action, works OK, but I do not trust it fully.
After discussing and researching I ended up using Uptime Kuma to get a "only notify me when something is wrong" approach. More or less it will tell me if a job failed or partially failed (single VM or container failed). This works pretty good and I get notified on Signal when something needs my attention. All well and done, but is this the best way? My approach has been to hook a script using
Full setup instructions here: https://www.sbarmen.no/posts/uptime-kuma-monitor-proxmox-backup/
The problem; this works only for the cluster, not for other hosts. I also use PBS to backup some physical machines, and also some config on the cluster nodes themselves. Is there a way to hook into all jobs on PBS with a similar approach with a script that could run a push notification on events in the same way? Any thoughts or ideas? Is there a best practice for this that I have missed?
After discussing and researching I ended up using Uptime Kuma to get a "only notify me when something is wrong" approach. More or less it will tell me if a job failed or partially failed (single VM or container failed). This works pretty good and I get notified on Signal when something needs my attention. All well and done, but is this the best way? My approach has been to hook a script using
/etc/vzdump.conf on every node. Uptime Kuma goes amber/red if no alert is received in X minutes, and if it receives a push then the script adds a status (up/down/problem). This gives me great visibility of all backups running in the cluster, but needs to run on all my nodes. Full setup instructions here: https://www.sbarmen.no/posts/uptime-kuma-monitor-proxmox-backup/
The problem; this works only for the cluster, not for other hosts. I also use PBS to backup some physical machines, and also some config on the cluster nodes themselves. Is there a way to hook into all jobs on PBS with a similar approach with a script that could run a push notification on events in the same way? Any thoughts or ideas? Is there a best practice for this that I have missed?
