Not getting Notifications for Verify and Garbage Collect Jobs

Darkbotic

Member
Jul 10, 2024
68
1
8
Hello everyone,

I have a question about notifications.

I have my PBS notifications properly set up.
Every day I have these jobs for my Datastore in the following order:

1) Verify Job
2) Prune Job
3) Garbage Collect Job
4) Sync Job

The Verify Job had an error but a notification was not sent.
I checked the logs and no attempt for sending a notification was made.

Code:
2025-05-06T10:00:36-04:00: verify Backups:vm/102/2025-05-06T12:00:50Z
2025-05-06T10:00:36-04:00:   check qemu-server.conf.blob
2025-05-06T10:00:36-04:00:   check drive-scsi0.img.fidx
2025-05-06T10:00:37-04:00: can't verify chunk, load failed - store 'Backups', unable to load chunk '179e9f443c30a3114f6c84cdb20ad1be42fb949ff18e74a45403943905544480' - Data blob has wrong CRC checksum.
2025-05-06T10:00:37-04:00: corrupted chunk renamed to "/mnt/datastore/Backups/.chunks/179e/179e9f443c30a3114f6c84cdb20ad1be42fb949ff18e74a45403943905544480.0.bad"
2025-05-06T10:00:54-04:00: can't verify chunk, load failed - store 'Backups', unable to load chunk 'e7b4ac26f24f5d2fcc3a9d1a9a4dfaa35ce36543c10364c0cb88b7fc055d95d4' - Data blob has wrong CRC checksum.
2025-05-06T10:00:54-04:00: corrupted chunk renamed to "/mnt/datastore/Backups/.chunks/e7b4/e7b4ac26f24f5d2fcc3a9d1a9a4dfaa35ce36543c10364c0cb88b7fc055d95d4.0.bad"
2025-05-06T10:00:54-04:00: can't verify chunk, load failed - store 'Backups', unable to load chunk 'fceea976d94ee4621265cef64ddb27d929f8379acac77c2d4c3eff8a870cf5e2' - Data blob has wrong CRC checksum.
2025-05-06T10:00:54-04:00: corrupted chunk renamed to "/mnt/datastore/Backups/.chunks/fcee/fceea976d94ee4621265cef64ddb27d929f8379acac77c2d4c3eff8a870cf5e2.0.bad"
2025-05-06T10:00:55-04:00:   verified 10469.16/19264.00 MiB in 18.86 seconds, speed 555.13/1021.47 MiB/s (3 errors)


May 06 10:00:37 PBS proxmox-backup-proxy[583]: can't verify chunk, load failed - store 'Backups', unable to load chunk '179e9f443c30a3114f6c84cdb20ad1be42fb949ff18e74a45403943905544480' - Data blob has wrong CRC checksum.
May 06 10:00:54 PBS proxmox-backup-proxy[583]: can't verify chunk, load failed - store 'Backups', unable to load chunk 'e7b4ac26f24f5d2fcc3a9d1a9a4dfaa35ce36543c10364c0cb88b7fc055d95d4' - Data blob has wrong CRC checksum.
May 06 10:00:54 PBS proxmox-backup-proxy[583]: can't verify chunk, load failed - store 'Backups', unable to load chunk 'fceea976d94ee4621265cef64ddb27d929f8379acac77c2d4c3eff8a870cf5e2' - Data blob has wrong CRC checksum.
May 06 10:02:04 PBS proxmox-backup-proxy[583]: Failed to verify the following snapshots/groups:
May 06 10:02:04 PBS proxmox-backup-proxy[583]:     ns/Proxmox/vm/102/2025-05-06T12:00:50Z
May 06 10:02:04 PBS proxmox-backup-proxy[583]: TASK ERROR: verification failed - please check the log for details

Then the Prune Job went through fine.

Then the Garbage Collect Job had 3 warnings but no notification attempt was made:

Code:
2025-05-06T12:30:01-04:00: warning: unable to access non-existent chunk 179e9f443c30a3114f6c84cdb20ad1be42fb949ff18e74a45403943905544480, required by "/mnt/datastore/Backups/ns/Proxmox/vm/102/2025-05-06T12:00:50Z/drive-scsi0.img.fidx"
2025-05-06T12:30:01-04:00: warning: unable to access non-existent chunk e7b4ac26f24f5d2fcc3a9d1a9a4dfaa35ce36543c10364c0cb88b7fc055d95d4, required by "/mnt/datastore/Backups/ns/Proxmox/vm/102/2025-05-06T12:00:50Z/drive-scsi0.img.fidx"
2025-05-06T12:30:01-04:00: warning: unable to access non-existent chunk fceea976d94ee4621265cef64ddb27d929f8379acac77c2d4c3eff8a870cf5e2, required by "/mnt/datastore/Backups/ns/Proxmox/vm/102/2025-05-06T12:00:50Z/drive-scsi0.img.fidx"

2025-05-06T12:30:04-04:00: TASK WARNINGS: 3

Then the Sync Job went through with an error:

Code:
2025-05-06T13:09:01-04:00: Encountered errors: No such file or directory (os error 2)
2025-05-06T13:09:01-04:00: Failed to push group vm/102 to remote!

2025-05-06T13:09:28-04:00: TASK ERROR: Sync failed with some errors!


Syslog

May 06 13:09:28 PBS proxmox-backup-proxy[583]: TASK ERROR: Sync failed with some errors!
May 06 13:09:28 PBS postfix/pickup[2204]: 4092323D10: uid=34 from=<PBS@redacted.com>
May 06 13:09:28 PBS postfix/cleanup[2220]: 4092323D10: message-id=<redacted@PBS.redacted.com>
May 06 13:09:28 PBS postfix/qmgr[701]: 4092323D10: from=<PBS@pepen.org>, size=1419, nrcpt=1 (queue active)
May 06 13:09:29 PBS postfix/smtp[2222]: 4092323D10: replace: header From: Proxmox Backup Server <PBS@redacted.com>: From: PBS <PBS@redacted.com>
May 06 13:09:30 PBS postfix/smtp[2222]: 4092323D10: to=<redacted@gmail.com>, relay=smtp.redacted.com[redacted]:465, delay=1.8, delays=0.03/0.04/0.86/0.87, dsn=2.0.0, status=sent (250 Message received)
May 06 13:09:30 PBS postfix/qmgr[701]: 4092323D10: removed

Here are my notification settings for that Datastore:

1746553255624.png

Why am I not getting the notifications for Verify and Prune Jobs?

Thanks in advance!
 
Local PBS

Code:
proxmox-backup                      3.4.0         running kernel: 6.8.12-10-pve
proxmox-backup-server               3.4.1-1       running version: 3.4.1       
proxmox-kernel-helper               8.1.1                                     
proxmox-kernel-6.8.12-10-pve-signed 6.8.12-10                                 
proxmox-kernel-6.8                  6.8.12-10                                 
proxmox-kernel-6.8.12-9-pve-signed  6.8.12-9                                   
ifupdown2                           3.2.0-1+pmx11                             
libjs-extjs                         7.0.0-5                                   
proxmox-backup-docs                 3.4.1-1                                   
proxmox-backup-client               3.4.1-1                                   
proxmox-mail-forward                0.3.2                                     
proxmox-mini-journalreader          1.4.0                                     
proxmox-offline-mirror-helper       0.6.7                                     
proxmox-widget-toolkit              4.3.10                                     
pve-xtermjs                         5.5.0-2                                   
smartmontools                       7.3-pve1                                   
zfsutils-linux                      2.2.7-pve2

Remote PBS

Code:
proxmox-backup                      3.4.0         running kernel: 6.8.12-10-pve
proxmox-backup-server               3.4.1-1       running version: 3.4.1       
proxmox-kernel-helper               8.1.1                                     
proxmox-kernel-6.8.12-10-pve-signed 6.8.12-10                                 
proxmox-kernel-6.8                  6.8.12-10                                 
proxmox-kernel-6.8.12-9-pve-signed  6.8.12-9                                   
ifupdown2                           3.2.0-1+pmx11                             
libjs-extjs                         7.0.0-5                                   
proxmox-backup-docs                 3.4.1-1                                   
proxmox-backup-client               3.4.1-1                                   
proxmox-mail-forward                0.3.2                                     
proxmox-mini-journalreader          1.4.0                                     
proxmox-offline-mirror-helper       0.6.7                                     
proxmox-widget-toolkit              4.3.10                                     
pve-xtermjs                         5.5.0-2                                   
smartmontools                       7.3-pve1                                   
zfsutils-linux                      2.2.7-pve2