Receiving hourly job failure notifications for snapshots that don't exist

acheslow

New Member
Jan 7, 2025
4
0
1
I am receiving these notifications every hour:
Code:
Job ID:    v-1cb0e751-6bbe
Datastore: synology

Verification failed on these snapshots/groups:


    ct/10443/2025-07-11T08:30:13Z
    ct/11443/2025-07-09T09:14:19Z


Please visit the web interface for further details:

<https://proxmox-backup.lan:8007/#pbsServerAdministration:tasks>

However, those snapshots do not exist in my datastore and I can't find anywhere to remove the associated verification job. The error in the task shows:
Code:
verify synology:ct/11443/2025-07-09T09:14:19Z - manifest load error: unable to load blob '"/mnt/synology/ct/11443/2025-07-09T09:14:19Z/index.json.blob"' - No such file or directory (os error 2)

I have tried removing the group from the datastore and doing a new backup of those containers.

Any tips on how I can stop these notifications from firing? All of my other backup and verification jobs are working as expected.

As an aside, I had renamed the hostname from proxmox-backup.lan to pbs.lan recently, but the notifications are still referencing the old hostname.

Code:
proxmox-backup-manager report
## FILES

### General System Info

`$ cat '/etc/hostname'`
```
pbs
```

`$ cat '/etc/hosts'`
```
127.0.0.1 localhost.localdomain localhost
192.168.1.52 pbs.lan pbs

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
```

`$ cat '/etc/network/interfaces'`
```
auto lo
iface lo inet loopback

auto ens3
iface ens3 inet static
        address 192.168.1.52/24
        gateway 192.168.1.1

source /etc/network/interfaces.d/*

auto ens4
iface ens4 inet static
        address 192.168.10.200/24
```

`$ cat '/etc/apt/sources.list'`
```
deb http://deb.debian.org/debian bookworm main contrib
deb http://deb.debian.org/debian bookworm-updates main contrib
deb http://security.debian.org/debian-security bookworm-security main contrib
```

`$ cat '/etc/apt/sources.list.d/pbs-install-repo.list'`
```
deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription
```

`$ cat '/etc/apt/sources.list.d/pbs-enterprise.list'`
```
# deb https://enterprise.proxmox.com/debian/pbs bookworm pbs-enterprise
```

`$ cat '/etc/apt/sources.list.d/pbstest-for-beta.list'`
```
# deb http://download.proxmox.com/debian/pbs bookworm pbstest
```


`$ cat '/proc/pressure/io'`
```
some avg10=0.00 avg60=0.00 avg300=0.00 total=387753224
full avg10=0.00 avg60=0.00 avg300=0.00 total=309316832
```

`$ cat '/proc/pressure/cpu'`
```
some avg10=0.00 avg60=0.00 avg300=0.00 total=247349780
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
```

`$ cat '/proc/pressure/memory'`
```
some avg10=0.00 avg60=0.00 avg300=0.00 total=20674010
full avg10=0.00 avg60=0.00 avg300=0.00 total=14608258
```


### Datastores & Remotes

`$ cat '/etc/proxmox-backup/datastore.cfg'`
```
datastore: synology
        comment Synology PBS mount
        gc-schedule 04:00
        notification-mode notification-system
        path /mnt/synology
        verify-new true
```

### User & Access

`$ cat '/etc/proxmox-backup/user.cfg'`
```
user: root@pam
        email {redacted}

user: backup@pbs
        enable true
        expire 0

user: api@pam

token: api@pam!homepage
        enable true
        expire 0
```

`$ cat '/etc/proxmox-backup/acl.cfg'`
```
acl:1:/:api@pam,api@pam!homepage:Audit
acl:1:/datastore:backup@pbs:DatastoreAdmin
```

### Remotes

`$ cat '/etc/proxmox-backup/remote.cfg'`
```
remote: pbs-office
        auth-id root@pam
        fingerprint {redacted}
        host pbs-office.lan
        password {redacted}
```

### Jobs

`$ cat '/etc/proxmox-backup/sync.cfg'`
```

```

`$ cat '/etc/proxmox-backup/prune.cfg'`
```
prune: s-ddfeda4b-7c11
        keep-last 5
        schedule 23:00
        store synology
```

`$ cat '/etc/proxmox-backup/verification.cfg'`
```
verification: v-46ba34a5-0094
        ignore-verified true
        ns 
        outdated-after 30
        schedule daily
        store synology
```

### Tape

`$ cat '/etc/proxmox-backup/tape.cfg'`
```
# file does not exist
```

`$ cat '/etc/proxmox-backup/media-pool.cfg'`
```
# file does not exist
```

### Others

`$ cat '/etc/proxmox-backup/node.cfg'`
```
# file does not exist
```

`$ cat '/etc/proxmox-backup/traffic-control.cfg'`
```
# file does not exist
```
## COMMANDS

$ `date -R`
```
Wed, 16 Jul 2025 09:56:23 -0700
```

$ `proxmox-backup-manager versions --verbose`
```
proxmox-backup                      3.4.0         running kernel: 6.8.12-11-pve
proxmox-backup-server               3.4.2-1       running version: 3.4.2       
proxmox-kernel-helper               8.1.1                                      
proxmox-kernel-6.8.12-11-pve-signed 6.8.12-11                                  
proxmox-kernel-6.8                  6.8.12-11                                  
proxmox-kernel-6.8.12-8-pve-signed  6.8.12-8                                   
proxmox-kernel-6.8.12-6-pve-signed  6.8.12-6                                   
proxmox-kernel-6.8.12-5-pve-signed  6.8.12-5                                   
proxmox-kernel-6.8.12-4-pve-signed  6.8.12-4                                   
ifupdown2                           3.2.0-1+pmx11                              
libjs-extjs                         7.0.0-5                                    
proxmox-backup-docs                 3.4.2-1                                    
proxmox-backup-client               3.4.2-1                                    
proxmox-mail-forward                0.3.3                                      
proxmox-mini-journalreader          1.5                                        
proxmox-offline-mirror-helper       0.6.7                                      
proxmox-widget-toolkit              4.3.11                                     
pve-xtermjs                         5.5.0-2                                    
smartmontools                       7.3-pve1                                   
zfsutils-linux                      2.2.7-pve2
```

$ `proxmox-backup-manager subscription get`
```
status   notfound                                                
message  There is no subscription key                            
serverid {redacted}                        
url      https://www.proxmox.com/en/proxmox-backup-server/pricing
```

$ `proxmox-backup-manager ldap list`
```

```

$ `proxmox-backup-manager openid list`
```

```

$ `proxmox-boot-tool status`
```
Re-executing '/sbin/proxmox-boot-tool' in new private mount namespace..
```
STDERR:
```
E: /etc/kernel/proxmox-boot-uuids does not exist.

```

$ `df -h`
```
Filesystem             Size  Used Avail Use% Mounted on
udev                   3.9G     0  3.9G   0% /dev
tmpfs                  795M  1.0M  794M   1% /run
/dev/mapper/pbs-root    28G  9.4G   17G  37% /
tmpfs                  3.9G     0  3.9G   0% /dev/shm
tmpfs                  5.0M     0  5.0M   0% /run/lock
/dev/sda2              511M   12M  500M   3% /boot/efi
synology:/volume3/pbs   18T  1.8T   16T  11% /mnt/synology
tmpfs                  795M     0  795M   0% /run/user/0
```

$ `lsblk --ascii -M -o +HOTPLUG,ROTA,PHY-SEC,FSTYPE,MODEL,TRAN`
```
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS HOTPLUG ROTA PHY-SEC FSTYPE      MODEL        TRAN
sda            8:0    0   64G  0 disk                   0    0     512             Storage      
|-sda1         8:1    0 1007K  0 part                   0    0     512                          
|-sda2         8:2    0  512M  0 part /boot/efi         0    0     512 vfat                     
`-sda3         8:3    0 31.5G  0 part                   0    0     512 LVM2_member              
  |-pbs-swap 252:0    0  3.8G  0 lvm  [SWAP]            0    0     512 swap                     
  `-pbs-root 252:1    0 27.7G  0 lvm  /                 0    0     512 ext4                     
sr0           11:0    1 1024M  0 rom                    1    1     512             QEMU DVD-ROM ata
sr1           11:1    1 1024M  0 rom                    1    1     512             QEMU DVD-ROM ata
```

$ `ls -l /dev/disk/by-id /dev/disk/by-path`
```
/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root  9 Jul 14 13:46 ata-QEMU_DVD-ROM_QM00001 -> ../../sr0
lrwxrwxrwx 1 root root  9 Jul 14 13:46 ata-QEMU_DVD-ROM_QM00002 -> ../../sr1
lrwxrwxrwx 1 root root 10 Jul 14 13:46 dm-name-pbs-root -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jul 14 13:46 dm-name-pbs-swap -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 14 13:46 dm-uuid-LVM-Rc0YxV3ZbATgBbr5ACN9ftnINjdjRbwK5CHFxw0LPnPeEiLGoeQSOyifEEzI23pf -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jul 14 13:46 dm-uuid-LVM-Rc0YxV3ZbATgBbr5ACN9ftnINjdjRbwKJRUdOaauIqf1Xws85Yvs97V2TzxBUgtx -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jul 14 13:46 lvm-pv-uuid-mPovw3-502X-N3dI-99qE-9kjD-XFwO-C2AAE9 -> ../../sda3
lrwxrwxrwx 1 root root  9 Jul 14 13:46 scsi-3600140536c664a3d9f15d4ef6da151d4 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 14 13:46 scsi-3600140536c664a3d9f15d4ef6da151d4-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 14 13:46 scsi-3600140536c664a3d9f15d4ef6da151d4-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 14 13:46 scsi-3600140536c664a3d9f15d4ef6da151d4-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Jul 14 13:46 wwn-0x600140536c664a3d9f15d4ef6da151d4 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 14 13:46 wwn-0x600140536c664a3d9f15d4ef6da151d4-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 14 13:46 wwn-0x600140536c664a3d9f15d4ef6da151d4-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 14 13:46 wwn-0x600140536c664a3d9f15d4ef6da151d4-part3 -> ../../sda3

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root  9 Jul 14 13:46 pci-0000:00:01.1-ata-1 -> ../../sr1
lrwxrwxrwx 1 root root  9 Jul 14 13:46 pci-0000:00:01.1-ata-1.0 -> ../../sr0
lrwxrwxrwx 1 root root  9 Jul 14 13:46 pci-0000:00:01.1-ata-1.1 -> ../../sr1
lrwxrwxrwx 1 root root  9 Jul 14 13:46 pci-0000:00:0a.0-scsi-0:0:1:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 14 13:46 pci-0000:00:0a.0-scsi-0:0:1:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 14 13:46 pci-0000:00:0a.0-scsi-0:0:1:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 14 13:46 pci-0000:00:0a.0-scsi-0:0:1:0-part3 -> ../../sda3
```

$ `zpool status`
```

```
STDERR:
```
no pools available

```

$ `zfs list`
```

```
STDERR:
```
no datasets available

```

$ `arcstat `
```
    time  read  ddread  ddh%  dmread  dmh%  pread  ph%   size      c  avail
09:56:23     0       0     0       0     0      0    0    960   248M   6.1G
```
## FUNCTIONS

#### Datastores
```
synology
```


#### System Load & Uptime
$ `top -b -c -w512 -n 1 -o TIME`
```
top - 09:56:23 up 1 day, 20:09,  1 user,  load average: 0.07, 0.03, 0.00
Tasks: 133 total,   1 running, 132 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.0 us, 20.0 sy,  0.0 ni, 60.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   7944.0 total,    239.3 free,    591.3 used,   7412.9 buff/cache     
MiB Swap:   3912.0 total,   3911.5 free,      0.5 used.   7352.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    625 backup    20   0 2138368  63628  28908 S   0.0   0.8  78:43.92 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy
    575 root      20   0  790872  30632  25064 S   0.0   0.4  47:09.48 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-api
     17 root      20   0       0      0      0 I   0.0   0.0   0:54.61 [rcu_preempt]
     64 root      20   0       0      0      0 S   0.0   0.0   0:32.40 [kswapd0]
   4104 root      20   0       0      0      0 I   0.0   0.0   0:14.57 [kworker/3:0-events]
    573 coroqne+  20   0   17712  13184   6912 S   0.0   0.2   0:11.50 /usr/bin/corosync-qnetd -f
     47 root      20   0       0      0      0 S   0.0   0.0   0:09.95 [kcompactd0]
    234 root      20   0       0      0      0 S   0.0   0.0   0:08.96 [jbd2/dm-1-8]
    290 root      20   0   97848  63872  62976 S   0.0   0.8   0:05.86 /lib/systemd/systemd-journald
     83 root       0 -20       0      0      0 I   0.0   0.0   0:05.11 [kworker/3:1H-kblockd]
    317 root      39  19       0      0      0 S   0.0   0.0   0:05.08 [dbuf_evict]
    315 root      39  19       0      0      0 S   0.0   0.0   0:04.98 [arc_reap]
    314 root      20   0       0      0      0 S   0.0   0.0   0:04.97 [arc_evict]
    442 root      20   0       0      0      0 S   0.0   0.0   0:04.21 [l2arc_feed]
   6994 root      20   0       0      0      0 I   0.0   0.0   0:03.99 [kworker/2:1-mm_percpu_wq]
      1 root      20   0  168072  12112   8912 S   0.0   0.1   0:03.94 /sbin/init
   5181 root      20   0       0      0      0 I   0.0   0.0   0:03.74 [kworker/0:1-cgroup_destroy]
    781 root      20   0   42668   4628   4096 S   0.0   0.1   0:03.57 /usr/lib/postfix/sbin/master -w
     79 root       0 -20       0      0      0 I   0.0   0.0   0:03.54 [kworker/1:1H-kblockd]
    135 root       0 -20       0      0      0 I   0.0   0.0   0:03.41 [kworker/2:1H-kblockd]
   7546 root      20   0       0      0      0 I   0.0   0.0   0:02.73 [kworker/1:0-events]
     36 root      20   0       0      0      0 S   0.0   0.0   0:02.69 [ksoftirqd/3]
    783 postfix   20   0   43244   6912   6272 S   0.0   0.1   0:02.60 qmgr -l -t unix -u
```
 
/mnt/synology/ct/11443/2025-07-09T09:14:19Z/
Check the contents of this and the other folder mentioned in the sync job, they are probably empty. Not sure however how you did end up with these stray folders? Did you interact with the datastore contents directly at some point?

As an aside, I had renamed the hostname from proxmox-backup.lan to pbs.lan recently, but the notifications are still referencing the old hostname.
Where did you adapt the hostname?
 
Hi Chris, yes the folders are empty. I believe I did manually remove them in the hope that this would remove the verification job.

I changed the hostname by manually updating /etc/hostname and /etc/hosts and rebooting the server.
 
Try something like grep -sR "proxmox-backup.lan" /etc to find references to the old hostname.
 
Last edited:
Thanks. It was in /etc/alises.db and /etc/postfix/main.cf. I updated the latter and then ran postalias /etc/alises and now they're both cleared out.

BTW, I rebooted each of the nodes in the cluster and then rebooted PBS and so far I haven't received any additional notifications. Fingers crossed that took care of it.