Webhook notifications stopped working after update

F

FuXo

New Member
Aug 15, 2024
7
0
1
Hi,
My backup notifications stopped working after update from kernel 6.2.16-20-pve to 6.8.12-11-pve.
Proxmox VE 8.4.1

I'm using webhook ( ntfy.sh )
Body:
=======

Proxmox

{{ title }}
{{ message }}
=======

After backup job I get error:
ERROR: could not notify via target `ntfy`: http://192.168.0.242/Proxmox: status code 400

Even if i set only {{ message }} I get this error, but test notifications works fine.

Before update everything worked fine.
Can someone help me with this problem ?
 

Attachments

  • Zrzut ekranu 2025-06-12 202906.png
    Zrzut ekranu 2025-06-12 202906.png
    165 KB · Views: 12
  • Zrzut ekranu 2025-06-12 203120.png
    Zrzut ekranu 2025-06-12 203120.png
    29.4 KB · Views: 12
Hey,

I strongly doubt that the updated kernel has any effect the webhhook target.

Could you share the output of pveversion -v for both nodes? This command lists a lot more relevant package versions. I'm particularly interested in the versions of libproxmox-rs-perl and libpve-rs-perl, these contain the implementation of the notification stack.

Also could you share the full config of your ntfy webhhok target from /etc/pve/notifications.cfg? Please make sure to mask out any sensitive values.
How many VMs are backed up by the backup job you mention? Something that we've seen a couple of times already is that some webhook endpoints have pretty low limits for the amount of text that you can send - so if the number of guests backed up by the job is high, the logs can exceed that limit.

Thanks!
 
Node that does'nt work:

proxmox-ve: 8.4.0 (running kernel: 6.8.12-11-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8: 6.8.12-11
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
ceph-fuse: 17.2.8-pve2
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
frr-pythontools: 10.2.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.11
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-4
pve-ha-manager: 4.0.7
pve-i18n: 3.4.4
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2

webhook: ntfy
body PT09PT09PQoKUHJveG1veAoKe3sgdGl0bGUgfX0Ke3sgbWVzc2FnZSB9fQo9PT09PT09
method post
url http://192.168.0.242/Proxmox

matcher: ntfy-matcher
mode all
target ntfy

Backup job - only 1 VM

Node that does work:
proxmox-ve: residual config (running kernel: 6.2.16-20-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
pve-kernel-6.2.16-5-pve: 6.2.16-6
ceph-fuse: 17.2.7-pve3
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2

webhook: Ntfy
body PT09PT09PQoKUHJveG1veAoKe3sgdGl0bGUgfX0Ke3sgbWVzc2FnZSB9fQo9PT09PT09
method post
url http://192.168.77.49/Proxmox

matcher: ntfy-matcher
mode all
target Ntfy

Backup job - only 1 VM
 
Code: 
diff a.txt b.txt
1c1
< proxmox-ve: 8.4.0 (running kernel: 6.8.12-11-pve)
---
> proxmox-ve: residual config (running kernel: 6.2.16-20-pve)
4,7c4,6
< proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
< proxmox-kernel-6.8: 6.8.12-11
< proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
< ceph-fuse: 17.2.8-pve2
---
> proxmox-kernel-6.2.16-20-pve: 6.2.16-20
> pve-kernel-6.2.16-5-pve: 6.2.16-6
> ceph-fuse: 17.2.7-pve3
10d8
< frr-pythontools: 10.2.2-1+pve1
41c39
< proxmox-widget-toolkit: 4.3.11
---
> proxmox-widget-toolkit: 4.3.10
48c46
< pve-firmware: 3.15-4
---
> pve-firmware: 3.15-3
50c48
< pve-i18n: 3.4.4
---
> pve-i18n: 3.4.2

This is the diff between the package versions of both nodes. Unfortunately there is nothing that really explains your problem.

I've tried the exact same body against the hosted version of nfty.sh, for me it works without any issues on a fully up-to-date Proxmox VE node.

It seems like the IP for the ntfy server is different in both nodes. Are they trying to talk to the same ntfy server via two different IPs? Or do you have two separate instances of ntfy? If yes, which version of ntfy are these running, is it the same across both instances? Are there any differences in configuration, in case these are two instances?
 
There are 2 separate environments.

Both ntfy images version are the same:

Container Repository Tag Image Id Size
-----------------------------------------------------------------
ntfy binwiederhier/ntfy latest f87d2a1253c2 72.61 MB

docker-compose file:
version: "3"

services:
ntfy:
image: binwiederhier/ntfy
container_name: ntfy
command:
- serve
environment:
NTFY_UPSTREAM_BASE_URL: "https://ntfy.sh"
NTFY_BASE_URL: http://192.168.77.49
volumes:
- /var/cache/ntfy:/var/cache/ntfy
- /etc/ntfy:/etc/ntfy
ports:
- 80:80
restart: unless-stopped

For the second istance the only diffrence is:
NTFY_BASE_URL: http://192.168.0.242
 
ntfy | 2025/06/13 08:34:09 DEBUG Rate limiters reset for visitor (visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=0, visitor_messages_limit=17280, visitor_messages_remaining=17280, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2025-06-13T08:34:09.633Z)
ntfy | 2025/06/13 08:34:09 DEBUG HTTP request started (http_method=POST, http_path=/Proxmox, tag=http, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=0, visitor_messages_limit=17280, visitor_messages_remaining=17280, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2025-06-13T08:34:09.633Z)
ntfy | 2025/06/13 08:34:09 DEBUG Connection closed with HTTP 400 (ntfy error 40014) (error=invalid request: attachments not allowed, error_code=40014, http_method=POST, http_path=/Proxmox, http_status=400, message_body_size=0, message_event=message, message_id=sX0nJQ1uMnwF, message_sender=192.168.0.250, message_time=1749803649, tag=http, topic=Proxmox, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0000913052, visitor_seen=2025-06-13T08:34:09.633Z)
ntfy | 2025/06/13 08:34:09 DEBUG HTTP request finished (http_method=POST, http_path=/Proxmox, tag=http, time_taken_ms=1, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0001879212, visitor_seen=2025-06-13T08:34:09.633Z)
 
Log from working enviroment

ntfy | 2025/06/13 08:38:39 DEBUG Rate limiters reset for visitor (visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.77.17, visitor_ip=192.168.77.17, visitor_messages=0, visitor_messages_limit=17280, visitor_messages_remaining=17280, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2025-06-13T08:38:39.202Z)
ntfy | 2025/06/13 08:38:39 DEBUG HTTP request started (http_method=POST, http_path=/Proxmox, tag=http, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.77.17, visitor_ip=192.168.77.17, visitor_messages=0, visitor_messages_limit=17280, visitor_messages_remaining=17280, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2025-06-13T08:38:39.202Z)
ntfy | 2025/06/13 08:38:39 DEBUG Received message (http_method=POST, http_path=/Proxmox, message_body_size=2817, message_call=, message_delayed=false, message_email=, message_event=message, message_firebase=true, message_id=oKEElzrj5lVX, message_sender=192.168.77.17, message_time=1749803919, message_unifiedpush=false, tag=publish, topic=Proxmox, topic_last_access=2025-06-13T08:38:39.202Z, topic_subscribers=0, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.77.17, visitor_ip=192.168.77.17, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.000026, visitor_seen=2025-06-13T08:38:39.202Z)
ntfy | 2025/06/13 08:38:39 DEBUG Adding message to cache (http_method=POST, http_path=/Proxmox, message_body_size=2817, message_event=message, message_id=oKEElzrj5lVX, message_sender=192.168.77.17, message_time=1749803919, tag=publish, topic=Proxmox, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.77.17, visitor_ip=192.168.77.17, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0000342562, visitor_seen=2025-06-13T08:38:39.202Z)
ntfy | 2025/06/13 08:38:39 DEBUG Wrote 1 message(s) in 175.453µs (tag=message_cache)
ntfy | 2025/06/13 08:38:39 DEBUG HTTP request finished (http_method=POST, http_path=/Proxmox, tag=http, time_taken_ms=0, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.77.17, visitor_ip=192.168.77.17, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.000115589, visitor_seen=2025-06-13T08:38:39.202Z)
ntfy | 2025/06/13 08:38:39 DEBUG Publishing poll request to https://ntfy.sh/396520cd080927175d4fb50f489e64dc0a0314406298faedfbf2443a2bafd09c (message_body_size=2817, message_event=message, message_id=oKEElzrj5lVX, message_sender=192.168.77.17, message_time=1749803919, topic=Proxmox, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.77.17, visitor_ip=192.168.77.17, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.000142165, visitor_seen=2025-06-13T08:38:39.202Z)
 
I think i found the problem.
I've tried to backup new VM and the notification worked.
The diffrence is that this message is shoreter than the other one:

ntfy | 2025/06/13 08:56:32 TRACE HTTP request started (http_method=POST, http_path=/Proxmox, http_request=POST /Proxmox HTTP/1.1
ntfy | Accept-Encoding: gzip
ntfy | User-Agent: proxmox-sync-http-client/0.9.4
ntfy | Accept: */*
ntfy | Content-Length: 1864
ntfy |
ntfy | =======
ntfy |
ntfy | Proxmox
ntfy |
ntfy | vzdump backup status (proxmox.local.pl): backup successful
ntfy |
ntfy | Details
ntfy | =======
ntfy | VMID Name Status Time Size Filename
ntfy | 110 test ok 10s 424.393 KiB /var/lib/vz/dump/vzdump-qemu-110-2025_06_13-10_56_22.vma.zst
ntfy |
ntfy | Total running time: 10s
ntfy | Total size: 424.393 KiB
ntfy |
ntfy | Logs
ntfy | ====
ntfy | vzdump 110 --node proxmox --all 0 --mode snapshot --storage local --compress zstd --notes-template '{{guestname}}' --fleecing 0 --notification-mode notification-system
ntfy |
ntfy |
ntfy | 110: 2025-06-13 10:56:22 INFO: Starting Backup of VM 110 (qemu)
ntfy | 110: 2025-06-13 10:56:22 INFO: status = stopped
ntfy | 110: 2025-06-13 10:56:22 INFO: backup mode: stop
ntfy | 110: 2025-06-13 10:56:22 INFO: ionice priority: 7
ntfy | 110: 2025-06-13 10:56:22 INFO: VM Name: test
ntfy | 110: 2025-06-13 10:56:22 INFO: include disk 'scsi0' 'local-lvm:vm-110-disk-0' 10G
ntfy | 110: 2025-06-13 10:56:22 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-110-2025_06_13-10_56_22.vma.zst'
ntfy | 110: 2025-06-13 10:56:22 INFO: starting kvm to execute backup task
ntfy | 110: 2025-06-13 10:56:24 INFO: started backup task '6e96cf6f-4f90-4954-aedb-9639633653d1'
ntfy | 110: 2025-06-13 10:56:27 INFO: 47% (4.8 GiB of 10.0 GiB) in 3s, read: 1.6 GiB/s, write: 0 B/s
ntfy | 110: 2025-06-13 10:56:30 INFO: 94% (9.4 GiB of 10.0 GiB) in 6s, read: 1.6 GiB/s, write: 0 B/s
ntfy | 110: 2025-06-13 10:56:31 INFO: 100% (10.0 GiB of 10.0 GiB) in 7s, read: 564.9 MiB/s, write: 0 B/s
ntfy | 110: 2025-06-13 10:56:31 INFO: backup is sparse: 10.00 GiB (100%) total zero data
ntfy | 110: 2025-06-13 10:56:31 INFO: transferred 10.00 GiB in 7 seconds (1.4 GiB/s)
ntfy | 110: 2025-06-13 10:56:31 INFO: stopping kvm after backup task
ntfy | 110: 2025-06-13 10:56:32 INFO: archive file size: 424KB
ntfy | 110: 2025-06-13 10:56:32 INFO: adding notes to backup
ntfy | 110: 2025-06-13 10:56:32 INFO: Finished Backup of VM 110 (00:00:10)
ntfy |
ntfy |
ntfy |
ntfy | =======, tag=http, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=0, visitor_messages_limit=17280, visitor_messages_remaining=17280, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2025-06-13T08:56:32.096Z)
ntfy | 2025/06/13 08:56:32 TRACE Received message (http_method=POST, http_path=/Proxmox, message_body={
ntfy | "id": "LIj5ge5aMBGE",
ntfy | "time": 1749804992,
ntfy | "expires": 1749848192,
ntfy | "event": "message",
ntfy | "topic": "Proxmox",
ntfy | "message": "=======\n\nProxmox\n\nvzdump backup status (proxmox.local.pl): backup successful \n\nDetails\n=======\nVMID Name Status Time Size Filename \n110 test ok 10s 424.393 KiB /var/lib/vz/dump/vzdump-qemu-110-2025_06_13-10_56_22.vma.zst \n\nTotal running time: 10s\nTotal size: 424.393 KiB\n\nLogs\n====\nvzdump 110 --node proxmox --all 0 --mode snapshot --storage local --compress zstd --notes-template '{{guestname}}' --fleecing 0 --notification-mode notification-system\n\n\n110: 2025-06-13 10:56:22 INFO: Starting Backup of VM 110 (qemu)\n110: 2025-06-13 10:56:22 INFO: status = stopped\n110: 2025-06-13 10:56:22 INFO: backup mode: stop\n110: 2025-06-13 10:56:22 INFO: ionice priority: 7\n110: 2025-06-13 10:56:22 INFO: VM Name: test\n110: 2025-06-13 10:56:22 INFO: include disk 'scsi0' 'local-lvm:vm-110-disk-0' 10G\n110: 2025-06-13 10:56:22 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-110-2025_06_13-10_56_22.vma.zst'\n110: 2025-06-13 10:56:22 INFO: starting kvm to execute backup task\n110: 2025-06-13 10:56:24 INFO: started backup task '6e96cf6f-4f90-4954-aedb-9639633653d1'\n110: 2025-06-13 10:56:27 INFO: 47% (4.8 GiB of 10.0 GiB) in 3s, read: 1.6 GiB/s, write: 0 B/s\n110: 2025-06-13 10:56:30 INFO: 94% (9.4 GiB of 10.0 GiB) in 6s, read: 1.6 GiB/s, write: 0 B/s\n110: 2025-06-13 10:56:31 INFO: 100% (10.0 GiB of 10.0 GiB) in 7s, read: 564.9 MiB/s, write: 0 B/s\n110: 2025-06-13 10:56:31 INFO: backup is sparse: 10.00 GiB (100%) total zero data\n110: 2025-06-13 10:56:31 INFO: transferred 10.00 GiB in 7 seconds (1.4 GiB/s)\n110: 2025-06-13 10:56:31 INFO: stopping kvm after backup task\n110: 2025-06-13 10:56:32 INFO: archive file size: 424KB\n110: 2025-06-13 10:56:32 INFO: adding notes to backup\n110: 2025-06-13 10:56:32 INFO: Finished Backup of VM 110 (00:00:10)\n\n\n\n======="
ntfy | }, message_body_size=1864, message_call=, message_delayed=false, message_email=, message_event=message, message_firebase=true, message_id=LIj5ge5aMBGE, message_sender=192.168.0.250, message_time=1749804992, message_unifiedpush=false, tag=publish, topic=Proxmox, topic_last_access=2025-06-13T08:56:32.097Z, topic_subscribers=0, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.000699274, visitor_seen=2025-06-13T08:56:32.096Z)
ntfy | 2025/06/13 08:56:32 DEBUG Adding message to cache (http_method=POST, http_path=/Proxmox, message_body_size=1864, message_event=message, message_id=LIj5ge5aMBGE, message_sender=192.168.0.250, message_time=1749804992, tag=publish, topic=Proxmox, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0014119164, visitor_seen=2025-06-13T08:56:32.096Z)
ntfy | 2025/06/13 08:56:32 DEBUG Wrote 1 message(s) in 1.929868ms (tag=message_cache)
ntfy | 2025/06/13 08:56:32 TRACE No stream or WebSocket subscribers, not forwarding (message_body_size=1864, message_event=message, message_id=LIj5ge5aMBGE, message_sender=192.168.0.250, message_time=1749804992, tag=publish, topic=Proxmox, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0029265168, visitor_seen=2025-06-13T08:56:32.096Z)
ntfy | 2025/06/13 08:56:32 DEBUG Publishing poll request to https://ntfy.sh/3ce16c1476724fa71a5f05a66728411f3e0e036bd75f33afbb23c2a26dd35cf1 (message_body_size=1864, message_event=message, message_id=LIj5ge5aMBGE, message_sender=192.168.0.250, message_time=1749804992, topic=Proxmox, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0031602066, visitor_seen=2025-06-13T08:56:32.096Z)
ntfy | 2025/06/13 08:56:32 DEBUG HTTP request finished (http_method=POST, http_path=/Proxmox, tag=http, time_taken_ms=18, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0036722288, visitor_seen=2025-06-13T08:56:32.096Z)
ntfy | 2025/06/13 08:56:59 DEBUG Deleted 0 stale visitor(s) (stale_visitors=0, tag=manager, time_taken_ms=0)


The other one:

ntfy | 2025/06/13 09:01:29 TRACE HTTP request started (http_method=POST, http_path=/Proxmox, http_request=POST /Proxmox HTTP/1.1
ntfy | Content-Length: 8795
ntfy | Accept-Encoding: gzip
ntfy | User-Agent: proxmox-sync-http-client/0.9.4
ntfy | Accept: */*
ntfy |
ntfy | =======
ntfy |
ntfy | Proxmox
ntfy |
ntfy | vzdump backup status (proxmox.local.pl): backup successful
ntfy |
ntfy | Details
ntfy | =======
ntfy | VMID Name Status Time Size Filename
ntfy | 104 npm ok 4m 29s 7.676 GiB /mnt/pve/Synology/dump/vzdump-qemu-104-2025_06_13-10_57_00.vma.zst
ntfy |
ntfy | Total running time: 4m 29s
ntfy | Total size: 7.676 GiB
ntfy |
ntfy | Logs
ntfy | ====
ntfy | vzdump 104 --prune-backups 'keep-last=2' --node proxmox --all 0 --mode snapshot --storage Synology --compress zstd --fleecing 0 --notification-mode notification-system
ntfy |
ntfy |
ntfy | 104: 2025-06-13 10:57:00 INFO: Starting Backup of VM 104 (qemu)
ntfy | 104: 2025-06-13 10:57:00 INFO: status = running
ntfy | 104: 2025-06-13 10:57:00 INFO: VM Name: npm
ntfy | 104: 2025-06-13 10:57:00 INFO: include disk 'scsi0' 'local-lvm:vm-104-disk-0' 32G
ntfy | 104: 2025-06-13 10:57:00 INFO: backup mode: snapshot
ntfy | 104: 2025-06-13 10:57:00 INFO: ionice priority: 7
ntfy | 104: 2025-06-13 10:57:00 INFO: creating vzdump archive '/mnt/pve/Synology/dump/vzdump-qemu-104-2025_06_13-10_57_00.vma.zst'
ntfy | 104: 2025-06-13 10:57:00 INFO: started backup task '34f4b252-76ba-4dcc-870c-682c70dd3b10'
ntfy | 104: 2025-06-13 10:57:00 INFO: resuming VM again
ntfy | 104: 2025-06-13 10:57:03 INFO: 1% (612.6 MiB of 32.0 GiB) in 3s, read: 204.2 MiB/s, write: 154.5 MiB/s
ntfy | 104: 2025-06-13 10:57:06 INFO: 2% (978.8 MiB of 32.0 GiB) in 6s, read: 122.0 MiB/s, write: 113.5 MiB/s
ntfy | 104: 2025-06-13 10:57:09 INFO: 7% (2.3 GiB of 32.0 GiB) in 9s, read: 445.9 MiB/s, write: 93.2 MiB/s
ntfy | 104: 2025-06-13 10:57:15 INFO: 8% (2.6 GiB of 32.0 GiB) in 15s, read: 57.4 MiB/s, write: 54.6 MiB/s
ntfy | 104: 2025-06-13 10:57:18 INFO: 9% (3.0 GiB of 32.0 GiB) in 18s, read: 123.2 MiB/s, write: 119.8 MiB/s
ntfy | 104: 2025-06-13 10:57:24 INFO: 10% (3.2 GiB of 32.0 GiB) in 24s, read: 47.1 MiB/s, write: 45.0 MiB/s
ntfy | 104: 2025-06-13 10:57:27 INFO: 11% (3.5 GiB of 32.0 GiB) in 27s, read: 102.7 MiB/s, write: 99.3 MiB/s
ntfy | 104: 2025-06-13 10:57:30 INFO: 16% (5.4 GiB of 32.0 GiB) in 30s, read: 640.4 MiB/s, write: 60.6 MiB/s
ntfy | 104: 2025-06-13 10:57:33 INFO: 17% (5.7 GiB of 32.0 GiB) in 33s, read: 85.8 MiB/s, write: 81.0 MiB/s
ntfy | 104: 2025-06-13 10:57:36 INFO: 18% (5.8 GiB of 32.0 GiB) in 36s, read: 41.1 MiB/s, write: 39.2 MiB/s
ntfy | 104: 2025-06-13 10:57:40 INFO: 19% (6.2 GiB of 32.0 GiB) in 40s, read: 100.6 MiB/s, write: 70.1 MiB/s
ntfy | 104: 2025-06-13 10:57:44 INFO: 20% (6.4 GiB of 32.0 GiB) in 44s, read: 68.9 MiB/s, write: 68.0 MiB/s
ntfy | 104: 2025-06-13 10:57:49 INFO: 21% (6.8 GiB of 32.0 GiB) in 49s, read: 66.0 MiB/s, write: 60.9 MiB/s
ntfy | 104: 2025-06-13 10:57:55 INFO: 22% (7.1 GiB of 32.0 GiB) in 55s, read: 56.2 MiB/s, write: 55.9 MiB/s
ntfy | 104: 2025-06-13 10:57:59 INFO: 23% (7.4 GiB of 32.0 GiB) in 59s, read: 68.0 MiB/s, write: 66.6 MiB/s
ntfy | 104: 2025-06-13 10:58:03 INFO: 24% (7.7 GiB of 32.0 GiB) in 1m 3s, read: 86.1 MiB/s, write: 85.5 MiB/s
ntfy | 104: 2025-06-13 10:58:06 INFO: 25% (8.0 GiB of 32.0 GiB) in 1m 6s, read: 118.4 MiB/s, write: 109.7 MiB/s
ntfy | 104: 2025-06-13 10:58:09 INFO: 26% (8.4 GiB of 32.0 GiB) in 1m 9s, read: 113.5 MiB/s, write: 90.3 MiB/s
ntfy | 104: 2025-06-13 10:58:15 INFO: 27% (8.7 GiB of 32.0 GiB) in 1m 15s, read: 60.5 MiB/s, write: 60.1 MiB/s
ntfy | 104: 2025-06-13 10:58:19 INFO: 28% (9.0 GiB of 32.0 GiB) in 1m 19s, read: 62.5 MiB/s, write: 61.4 MiB/s
ntfy | 104: 2025-06-13 10:58:25 INFO: 29% (9.3 GiB of 32.0 GiB) in 1m 25s, read: 62.2 MiB/s, write: 61.7 MiB/s
ntfy | 104: 2025-06-13 10:58:29 INFO: 30% (9.7 GiB of 32.0 GiB) in 1m 29s, read: 83.4 MiB/s, write: 76.1 MiB/s
ntfy | 104: 2025-06-13 10:58:33 INFO: 31% (10.0 GiB of 32.0 GiB) in 1m 33s, read: 77.9 MiB/s, write: 77.7 MiB/s
ntfy | 104: 2025-06-13 10:58:36 INFO: 32% (10.3 GiB of 32.0 GiB) in 1m 36s, read: 97.9 MiB/s, write: 58.1 MiB/s
ntfy | 104: 2025-06-13 10:58:40 INFO: 33% (10.6 GiB of 32.0 GiB) in 1m 40s, read: 82.5 MiB/s, write: 76.0 MiB/s
ntfy | 104: 2025-06-13 10:58:45 INFO: 34% (10.9 GiB of 32.0 GiB) in 1m 45s, read: 63.1 MiB/s, write: 63.0 MiB/s
ntfy | 104: 2025-06-13 10:58:50 INFO: 35% (11.2 GiB of 32.0 GiB) in 1m 50s, read: 67.4 MiB/s, write: 67.1 MiB/s
ntfy | 104: 2025-06-13 10:58:54 INFO: 36% (11.6 GiB of 32.0 GiB) in 1m 54s, read: 90.6 MiB/s, write: 90.2 MiB/s
ntfy | 104: 2025-06-1 ... (peeked 4096 bytes), tag=http, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2025-06-13T09:01:29.862Z)
ntfy | 2025/06/13 09:01:29 DEBUG Connection closed with HTTP 400 (ntfy error 40014) (error=invalid request: attachments not allowed, error_code=40014, http_method=POST, http_path=/Proxmox, http_status=400, message_body_size=0, message_event=message, message_id=3m5W0IZnX4Eg, message_sender=192.168.0.250, message_time=1749805289, tag=http, topic=Proxmox, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=2, visitor_messages_limit=17280, visitor_messages_remaining=17278, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0000295942, visitor_seen=2025-06-13T09:01:29.862Z)
ntfy | 2025/06/13 09:01:29 DEBUG HTTP request finished (http_method=POST, http_path=/Proxmox, tag=http, time_taken_ms=0, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:192.168.0.250, visitor_ip=192.168.0.250, visitor_messages=2, visitor_messages_limit=17280, visitor_messages_remaining=17278, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.000051642, visitor_seen=2025-06-13T09:01:29.862Z)



Is there a way to send only:
ntfy | Details
ntfy | =======
ntfy | VMID Name Status Time Size Filename
ntfy | 104 npm ok 4m 29s 7.676 GiB /mnt/pve/Synology/dump/vzdump-qemu-104-2025_06_13-10_57_00.vma.zst
ntfy |
ntfy | Total running time: 4m 29s
ntfy | Total size: 7.676 GiB
ntfy |

without:
ntfy | Logs
ntfy | ====
ntfy | vzdump 104 --prune-backups 'keep-last=2' --node proxmox --all 0 --mode snapshot --storage Synology --compress zstd --fleecing 0 --notification-mode notification-system
ntfy |
.
.
.
 
Yeah, that seems to be it. I dug through the ntfy documentation, seems like any message longer than 4096 bytes will be sent as an attachment, and apparently the default self-hosted version of ntfy does not allow attachments [1].

FuXo said:
Is there a way to send only:
ntfy | Details
ntfy | =======
ntfy | VMID Name Status Time Size Filename
ntfy | 104 npm ok 4m 29s 7.676 GiB /mnt/pve/Synology/dump/vzdump-qemu-104-2025_06_13-10_57_00.vma.zst
ntfy |
ntfy | Total running time: 4m 29s
ntfy | Total size: 7.676 GiB
ntfy |

without:
ntfy | Logs
ntfy | ====
ntfy | vzdump 104 --prune-backups 'keep-last=2' --node proxmox --all 0 --mode snapshot --storage Synology --compress zstd --fleecing 0 --notification-mode notification-system
ntfy |
Click to expand...

Yes, actually there is, starting with Proxmox VE 8.4, which added a way to override the template which is used to render the notification.
The documentation for it is not deployed yet, but you can access it in via our Git repo [2].

First, create the override directory and copy the base template to it:

Code: 
mkdir -p /etc/pve/notification-templates/default
cp /usr/share/pve-manager/templates/default/vzdump-body.txt.hbs /etc/pve/notification-templates/default/

You can then edit /etc/pve/notification-templates/default/vzdump-body.txt.hbs and remove the part where it produces the logs, e.g. like this:

Code: 
{{error}}
Details
=======
{{table guest-table}}
Total running time: {{duration total-time}}
Total size: {{human-bytes total-size}}

Note that this will affect all notification targets which produce a plain-text notification (plain-text part in emails, webhook, gotify).

Hope this helps!

[1] https://docs.ntfy.sh/publish/#limitations
[2] https://git.proxmox.com/?p=pve-docs...03097dc6bd17eced2ec4143510161060;hb=HEAD#l467
 
  • Like
Reactions: proxuserglider230, zotac, paolog89 and 1 other person
Lukas Wagner said:
Yeah, that seems to be it. I dug through the ntfy documentation, seems like any message longer than 4096 bytes will be sent as an attachment, and apparently the default self-hosted version of ntfy does not allow attachments [1].



Yes, actually there is, starting with Proxmox VE 8.4, which added a way to override the template which is used to render the notification.
The documentation for it is not deployed yet, but you can access it in via our Git repo [2].

First, create the override directory and copy the base template to it:

Code: 
mkdir -p /etc/pve/notification-templates/default
cp /usr/share/pve-manager/templates/default/vzdump-body.txt.hbs /etc/pve/notification-templates/default/

You can then edit /etc/pve/notification-templates/default/vzdump-body.txt.hbs and remove the part where it produces the logs, e.g. like this:

Code: 
{{error}}
Details
=======
{{table guest-table}}
Total running time: {{duration total-time}}
Total size: {{human-bytes total-size}}

Note that this will affect all notification targets which produce a plain-text notification (plain-text part in emails, webhook, gotify).

Hope this helps!

[1] https://docs.ntfy.sh/publish/#limitations
[2] https://git.proxmox.com/?p=pve-docs...03097dc6bd17eced2ec4143510161060;hb=HEAD#l467
Click to expand...
Is there any documentation around what are the possible handlebar objects I could call / reference in these template files?
 
proxuserglider230 said:
Is there any documentation around what are the possible handlebar objects I could call / reference in these template files?
Click to expand...
A colleague is working on documentation for this, but I cannot provide an ETA at this moment. But in general, most of the variables/helpers that are available are already used by the existing templates, so the easiest option is just to use them as a starting point.
 
You must log in or register to reply here.
Top Bottom