Missing chunks on new backups

[jacksonwashere]

Hey, I've been having issues with my backups since I deleted a datastore and created a new one and started afresh.

After creating the new datastore the data would backup as usual but as soon as the first GC runs then backups randomly become corrupted.
One instance has a bunch of verified images then goes failed and then goes back to verified. Another instance was good and then now is nothing but failed verifications.

I'm assuming for whatever reason pruning/gc seems to think certain chunks are no longer being referenced and decides to delete them?



After initially creating the new backup store I was questioning if the full backup was actually being done or not because the file size on disk was way smaller than the vm size (I'd just assume it's very compressed?) so I tried shutting down the node and then backing them up. Since seeing these invalid verifications I'm now starting to question if anything is working at all and what is the best thing to do next.

Does PVE have any knowledge of what's stored in the PBS datastore and is that my source of issues? assuming it thinks I'm still using the old store?

Is it possible to disable partial backups all together? or have some backups not be partial?

Should I look into wiping my instance and starting again? If so can I do this without wiping the whole os (I have packages installed/configured)? Would deleting /etc/proxmox-backup/datastore.cfg be enough?

Thanks!

 

[Dunuin]

GC needs "atime" or "relatime" enabled to decide what chunks to delete. Check that your storage is supporting this. A lot of tutorials tell you to disable atime for better performance and some NAS/SAN will even have it disable by default.
Unreliable SMB/NFS could also cause problems.

 

[jacksonwashere]

I've got atime and relatime both turned on - would it be ok to turn relatime off since it seems to only update access once every 24 hours without a write?

I guess my next question would be is if the pbs system knows there's missing chunks, why does it not fill in the missing chunks on the next backup? why does the pve instance backing up not know what the current state of the backup system is?

 

[Dunuin]

I've got atime and relatime both turned on - would it be ok to turn relatime off since it seems to only update access once every 24 hours without a write?

I would prefer relatime over atime for better performance. PBS is designed with these 24h in mind. A GC will only delete chunks if the prune was at least 24h and 5 min ago.

 

[Dunuin]

I guess my next question would be is if the pbs system knows there's missing chunks, why does it not fill in the missing chunks on the next backup?

PBS will only know that if you ran a re-verify task after the chunks got lost.
A normal verify job will skip backup snapshots that already got verified.
You will have to configure the verify job to re-verify old backup snaoshots after X days.
Once it knows that these chunks are missing or corrupted it will try to reupload them the next time you backup that guest (but this of cause will only work if the guest still contains the data of that missing chunk).

 

[fabian]

Hey, I've been having issues with my backups since I deleted a datastore and created a new one and started afresh.

how exactly did you do that? the symptoms you experience are usually caused by manually messing up the datastore directory..