Monitoring backups

[mailinglists]

Hi.

How do you guys monitor backup system (with Nagios compatible plugins)?

Now that we are switching to PBS, I monitor for failed backups by looking at the failed backup tasks with pvesh get /nodes/XXX/tasks --typefilter vzdump --errors or by parsing system logs for erros on PVE hosts.

Is there a better way? Do you have any plugins you want to share?

 

[roadrunner_rad]

May be quick and dirty looking on the PBS for failed backups like:

grep ":backup:" /var/log/proxmox-backup/tasks |grep -v " OK"

 

[mailinglists]

Tnx for suggeston, but it is basically just another way to get pvesh get /nodes/XXX/tasks --typefilter vzdump --errors and we already do that.

 

[stalker]

datestatus=`pvesh get /nodes/$HOSTNAME/tasks/ --typefilter vzdump --output-format json| jq -r '.[] | "\(.endtime) \(.status)"'|head -n1`

TS=$(echo "$datestatus" | ( read TS STATUS; echo $TS ) );
STATUS=$(echo "$datestatus" | ( read TS STATUS; echo $STATUS ) );

LIMIT=$( date +%s -d '28 hours ago' );
if [ $TS -lt $LIMIT ]; then
    echo "WARNING - backup is too old"
    exit 1
elif [ $STATUS != "OK" ]; then
    echo "CRITICAL - status failed"
    exit 2
else
  echo "OK"
  exit 0
fi

 

[mailinglists]

This is a nice script, but this will notify only after 28 hours of no successful backups and for some VMs we don't even use more than a few hours old backups. It will also fail to notify you about backups manually started via GUI, becayse there typefilter would be pvedaemon in that case.

 

[t.lamprecht]

Now that we are switching to PBS, I monitor for failed backups by looking at the failed backup tasks with pvesh get /nodes/XXX/tasks --typefilter vzdump --errors or by parsing system logs for erros on PVE hosts.

Maybe I'm missing something, but you could use mail notifications on error and map/inject that somehow™ into your monitoring system.

Related to that, you could go a similar way a user did when getting some rudimentary "gotify" support, using the mail .forward file:
https://forum.proxmox.com/threads/support-for-gotify.69719/#post-313131

I know it's a bit hacky, but just mentioning it as possible workarounds - a advantage of using that is, that it's more event based and does not need constant polling.

 

[DerDanilo]

https://forum.proxmox.com/threads/notification-center-and-multiple-notification-transports.76568/

I opened a feature request for a notification center. But I guess that this might be a feature of the central management board for all Proxmox products only.
If a central entity knows about all clusters it can easily check for errors and report them properly.