[SOLVED] Push/sync older snapshots than present on remote datastore

[Matrix9688]

Is it possible to force a sync (more specifically, a push sync) of a snapshot that is older than the currently present one on the remote datastore?


Background:
I f'd up a bit when configuring my onsite PBS instance - I didn't put in "keep latest x" in the prune job, and forgot to mark certain snapshots as protected. So the next time the datastore got prune'd and GC'd it only kept the latest snapshot.
This is a problem, since I wanted to use this particular host group in an unusual way: I used the same host ("archive") to create backups of old, large data exports of different origin - for example "old_fileserver" and "export_websiteX". The data was originally located on a windows machine, so I rsync'd it to a samba share of this temporary "archive" host, created a backup of one fileset via proxmox-backup-client, deleted the fileset on the "archive" host to get enough room for the next fileset, and repeated the process for the next fileset.

Fortunately, I had synced all snapshots to my offsite PBS instance before the "older" snapshots got pruned on the onsite PBS instance. So now I have all datasets only on my offsite PBS, but since I want all my data to be stored sufficiently redundantly, I would like to re-sync the older snapshots to my onsite PBS.

But when kicking off such a push sync job, the following happens:

2025-01-16T10:00:33+01:00: Starting datastore sync job '$PUSH_USER:$REMOTE_DATASTORE:$LOCAL_DATASTORE:$LOCAL_NAMESPACE:s-00ee9145-853c'
2025-01-16T10:00:33+01:00: sync datastore '$LOCAL_DATASTORE' to '$PUSH_USER/$REMOTE_DATASTORE'
2025-01-16T10:00:33+01:00: ----
2025-01-16T10:00:33+01:00: Syncing datastore '$LOCAL_DATASTORE', namespace '$LOCAL_NAMESPACE' into datastore '$REMOTE_DATASTORE', namespace '$REMOTE_NAMESPACE'
2025-01-16T10:00:33+01:00: Found 1 groups to sync (out of 54 total)
2025-01-16T10:00:33+01:00: skipped: 2 snapshot(s) (2024-08-22T12:14:01Z .. 2024-08-22T13:51:18Z) - older than the newest snapshot present on sync target
2025-01-16T10:00:33+01:00: Finished syncing namespace $LOCAL_NAMESPACE, current progress: 0 groups, 0 snapshots
2025-01-16T10:00:33+01:00: Summary: sync job found no new data to push
2025-01-16T10:00:33+01:00: sync job '$PUSH_USER:$REMOTE_DATASTORE:$LOCAL_DATASTORE:$LOCAL_NAMESPACE:s-00ee9145-853c' end
2025-01-16T10:00:33+01:00: queued notification (id=a131c2c1-53d1-465a-9459-40eeaead9c7e)
2025-01-16T10:00:33+01:00: TASK OK

So that's where I'm at.

 

[Chris]

Hi,

Is it possible to force a sync (more specifically, a push sync) of a snapshot that is older than the currently present one on the remote datastore?

no that is not possible for safety reasons.

If you do not want to prune and resync all the contents again, what you could do is create a new namespace, and push sync all the snapshots into that new namespace. Since deduplication is on the datastore level, that will only require additional space for the not already present snapshots.

 

[Matrix9688]

Hi Chris,

that's a good idea, thanks!
But I haven't even considered pruning the group completely and re-syncing it - that seems like a good, albeit inefficient alternative. I think I'll go with that route.

 

[Matrix9688]

As an aside, would it even be that inefficient though? If I were to just prune the snapshot and immediately do the sync before any garbage collect runs, shouldn't all chunks still be present and be reused without much down-/upload?

 

[Chris]

No, the push sync uses the backup API, and if it does not receive the list of already known chunks (which it only gets if there is a previous snapshot), it will re-upload the chunks again. So you will re-upload the chunks once, only for consecutive snapshot archives the push sync will optimize and avoid re-uploading known chunks.

But this will be true in both cases, for uploading after a complete prune and uploading to a new namespace.