[SOLVED] PBS DataStore consistency between backups

[sabiwww]

I installed PVE and PBS on the same server (I only maintain a single server in my home and I have a NAS). I am using the Backup server as a target of my backups. The backup job runs daily and the datastore is rsync'd onto the NAS daily witch a cron job:
rsync --update -avzP /hdd/backups /mnt/pve/NAS/proxmox-backups
(The NAS syncs the proxmox-backups to Backblaze B2).

My question is, will the backups work for an older version If the .chunks are not fully synced to the NAS/cloud yet?
With my setup, the backups could get out of sync in two places. Between the PBS and the NAS, and between the NAS and the cloud storage. The slowest is the cloud upload.
- If the PBS backup job is not done before the rsync starts or the rsync is not done before the PBS job starts.
- If the cloud upload is not done before rsync starts to change files
- Also, I guess GC could also mess up my backups

I could live with not being able to restore the last 1-2 daily backups, but will I be able to restore a VM at all if rsync is not fully synced the chunks to the NAS or the NAS to the cloud? Do previous versions work or the whole datastore becomes unusable if a few chunks are from a different backup version or missing?

It is not that easy to find answers for this question.
Thanks for the help!

 

[Matthias.]

As long as the chunks required for a backup are available, you can restore that backup.

Try it out by deleting some entries in the chunk folder - some backups will be destroyed, but the others will still work. Obviously, don't do this with any actual backups as it will irreparably destroy the backup.
Generally, I would recommend testing your backup and restore methods instead of trusting that they work : )

 

[sabiwww]

As long as the chunks required for a backup are available, you can restore that backup.

Try it out by deleting some entries in the chunk folder - some backups will be destroyed, but the others will still work. Obviously, don't do this with any actual backups as it will irreparably destroy the backup.
Generally, I would recommend testing your backup and restore methods instead of trusting that they work : )

Thanks. I am planning to test a restore from cloud, but will require some time and even if I test it, it is not guaranteed to cover cases like "what happens when garbage collection was running during rsync"..etc.
So you are saying as long as the current state is successfully copied, backups running during the process do not mess with previous backups. That's good to know.
Can we say the same for GC? Trying to restore from a datastore state copied just during a GC should work as expected? Or it is supposed to be consistent only after the GC stopped?

 

[fabian]

the following should hold at all times:
- new chunks are only added when creating a snapshot (either by doing a backup, or by running a sync/tape restore)
- chunks are only removed during a GC (iff they are no longer referenced by any snapshot)
- chunks are only renamed if detected as corrupt

the last one is orthogonal to your question

if you rsync a datastore while other operations might happen, some snapshots (on the target) might reference chunks that don't exist yet (for example, if you sync chunks first and then snapshot indices) or no longer exist (snapshot indices synced before chunks, but prune & GC removed snapshot and chunks in the meantime). in that case those snapshots won't be restorable for obvious reasons.

but any snapshots "in the middle" (neither newly created nor removed while the rsync is running) should be in the same state as in the datastore (so if no chunks are missing or corrupt on the source, the same should hold for the target).

but like @Matthias. said - the only good backup (strategy) is one you test regularly, including a restore