PBS Sync jobs Failed

[emregozturk]

Hello,

I have two sites and PBS.
I have configured remotes then created a sync jobs. When I start the sync job it's getting following message

2022-12-05T15:22:44+03:00: Starting datastore sync job 'Equinix-PBSDBackupsDBackups::1000'
2022-12-05T15:22:44+03:00: sync datastore 'DDBackups' from 'Equinix-PBS/DDBackups'
2022-12-05T15:22:44+03:00: ----
2022-12-05T15:22:44+03:00: Syncing datastore DDBackups, root namespace into datastore DDBackups, root namespace
2022-12-05T15:22:44+03:00: found 1 groups to sync (out of 89 total)
2022-12-05T15:22:45+03:00: sync snapshot vm/101/2022-12-04T19:00:02Z
2022-12-05T15:22:45+03:00: sync archive qemu-server.conf.blob
2022-12-05T15:22:45+03:00: sync archive drive-virtio1.img.fidx
2022-12-05T15:22:45+03:00: percentage done: 100.00% (6/6 snapshots)
2022-12-05T15:22:45+03:00: sync group vm/101 failed - reading file "/mnt/datastore/DDBackups/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" failed: No such file or directory (os error 2)
2022-12-05T15:22:45+03:00: Finished syncing namespace , current progress: 0 groups, 6 snapshots
2022-12-05T15:22:45+03:00: TASK ERROR: sync failed with some errors.

and also when i check sync jobs i see Local no namespace accessible It is the same for both sites.

 

[dcsapak]

2022-12-05T15:22:45+03:00: sync group vm/101 failed - reading file "/mnt/datastore/DDBackups/.chunks/bb9f/bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8" failed: No such file or directory (os error 2)

it seems that specific chunk does not exist anymore? did you manually delete it?

 

[emregozturk]

it seems that specific chunk does not exist anymore? did you manually delete it?

Hello,

I didn't it. But you are right it does not exist and also I tried restoring from backup i get similar error. But daily backups are taken successfully.

 

[dcsapak]

i'd start with verifying the existing snapshots to check which snapshots are affected

 

[emregozturk]

i'd start with verifying the existing snapshots to check which snapshots are affected

I checked all of vms and snapshots i think all vms are affected because I can't do file restore and image restore its getting "proxmox-file-restore" failed: Error: Starting VM failed.

Why does this happen, How can I revive Backups?

 

[dcsapak]

i meant a pbs verify operation. this will go through all chunks of a backup and checks if its missing or corrupted. it will then mark the backups that are not ok

 

[emregozturk]

I verified them and I saw that a lot chunks corrupted so I removed and recreated all PBS configurations. What I think might be causing this is sync jobs. Is it possible?

 

[dcsapak]

Is it possible

not really

chunks are only deleted on a garbage collect and only for chunks that are not referenced anymore (for over 24 hours)
can you post your pbs versions? (proxmox-backup-manager versions --verbose)

 

[emregozturk]

not really

chunks are only deleted on a garbage collect and only for chunks that are not referenced anymore (for over 24 hours)

can you post your pbs versions? (proxmox-backup-manager versions --verbose)

proxmox-backup-manager versions --verbose
proxmox-backup 2.2-1 running kernel: 5.15.35-1-pve
proxmox-backup-server 2.2.1-1 running version: 2.2.1
pve-kernel-5.15 7.2-3
pve-kernel-helper 7.2-3
pve-kernel-5.15.35-1-pve 5.15.35-3
ifupdown2 3.1.0-1+pmx3
libjs-extjs 7.0.0-1
proxmox-backup-docs 2.2.1-1
proxmox-backup-client 2.2.1-1
proxmox-mini-journalreader 1.2-1
proxmox-widget-toolkit 3.5.1
pve-xtermjs 4.16.0-1
smartmontools 7.2-pve3
zfsutils-linux 2.1.4-pve1

 

[dcsapak]

ok it seems you did not upgrade for quite some time, so i'd recommend upgrading in any case before continuing