Help with webhook notifications

[SimHat]

I've had email notifications working for a long time with my backups, but I'm trying to transition to webhook notifications. I have my webhook set up and when I hit test, it is working and I get the "test notification".

The notification target is enabled. When my backups run I'm not getting any notification. I feel like this should be working, but I must be missing something simple.

 

[TDex96]

Hey!

Same problem here with slack. The test notification works fine, but i didn't got any notification after the backups.

 

[kamao]

Have the same issue with backup notifications to Discord.

The test comes through OK, yet when a backup job runs.. nothing. To confirm, on the backup job:



On Notification matcher, currently 'All':

 

[SimHat]

I set up my Proxmox Backup server notifications the same way, and they do work, so either I don't really understand how the matcher rules work or something isn't working correctly with PVE notifications.

 

[Lukas Wagner]

Anything in the backup job task logs? If the HTTP call is rejected you should see it there, right at the end of the log output.

With Discord there seems to be the problem that they have a limit on the amount of text that you can send at once via a webhook. So this can be the reason why a 'Test target' works, while a backup job which backs up a couple of VMs does not.
Maybe Slack imposes similar limits.

A solution on the side of Proxmox VE could be to provide a way to switch to 'more compact' notification templates (e.g. skipping logs for backup jobs), but I haven't really looked into this yet.
The templates that are used to render the notification text are stored in /usr/share/pve-manager/templates/default - if it is indeed a problem with too much text, you could try editing the template (note that your changes might be overwritten by future package updates)

 

[SimHat]

Anything in the backup job task logs? If the HTTP call is rejected you should see it there, right at the end of the log output.

With Discord there seems to be the problem that they have a limit on the amount of text that you can send at once via a webhook. So this can be the reason why a 'Test target' works, while a backup job which backs up a couple of VMs does not.
Maybe Slack imposes similar limits.

A solution on the side of Proxmox VE could be to provide a way to switch to 'more compact' notification templates (e.g. skipping logs for backup jobs), but I haven't really looked into this yet.
The templates that are used to render the notification text are stored in /usr/share/pve-manager/templates/default - if it is indeed a problem with too much text, you could try editing the template (note that your changes might be overwritten by future package updates)

Nothing in the logs

INFO: Duration: 31.78s   
INFO: End Time: Wed Dec  4 02:09:06 2024   
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=5
INFO: running 'proxmox-backup-client prune' for 'ct/110'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-110-disk-0_vzdump" successfully removed.
INFO: Finished Backup of VM 110 (00:00:33)
INFO: Backup finished at 2024-12-04 02:09:07
INFO: Backup job finished successfully
TASK OK

 

[Lukas Wagner]

Nothing in the logs

INFO: Duration: 31.78s  
INFO: End Time: Wed Dec  4 02:09:06 2024  
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=5
INFO: running 'proxmox-backup-client prune' for 'ct/110'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-110-disk-0_vzdump" successfully removed.
INFO: Finished Backup of VM 110 (00:00:33)
INFO: Backup finished at 2024-12-04 02:09:07
INFO: Backup job finished successfully
TASK OK

Oh, there seems to be an issue at the moment where messages logged by the notification system do not show up in the task logs. At the moment you only see them in the system logs. You should see messages like "notified via target ...." or "could not notify via target ... ".

 

[kamao]

Status code 400 here (this is with one VM selected):

 

[Lukas Wagner]

Status code 400 here (this is with one VM selected):

View attachment 78688

Could you share your webhook configuration (e.g. via a screenshot of the UI) and the task log for this backup job?

 

[kamao]

Thanks Lukas. To confirm, the test with this config comes through OK (and works from PBS [e.g. garbage collection], same as @SimHat )

 

[SimHat]

This is a length issue. I was able to confirm that it is working backing up a single VM, but the message is far too long when backing up multiple.
See this thread. https://forum.proxmox.com/threads/v...notification-to-discord-webhook.157823/page-2

 

[TakeusV1]

According to the discord documentation, the maximum number of characters is 2000.

 

[lingfish]

There should be an option to send line by line -- wouldn't help so much going to things like Slack, but I'm attempting to use it to send to Graylog.