Partial sync job ? (pre-prune)

[guerbywork]

Hi,

I want to set up a very fast secondary PBS server (NVME) and have it sync to our current existing primary PBS server (SSD). The NVME server will have less space than the SSD server and not enough space to have a copy of all backups of the primary, the idea is to keep only a limited number of backups on the secondary server but be able to restore really fast when needed.

Now my question is how to fill up initally the (empty) new secondary server with only a partial set of backups from the primary, I don't see any option in the web UI for sync job to filter/prune what to sync initially. I juste see a non web editable "backup groups: all" when I edit the sync job with the web UI.

In the manual I noticed --group-filter so I did proxmox-backup-manager sync-job update ID --group-filter group:vm/100 and in the web UI "backup groups" was updated, this allows to select some of the VM (nice) but does not do time based selection to copy/sync only a few weeks of backup.

Later I will likely set up another tertiary PBS with slow hdd for long term archiving (yearly, or manually specified) so I will have the same time based filtering need.

Did I miss something? Is such a feature planned? I don't see it in the roadmap https://pbs.proxmox.com/wiki/index.php/Roadmap

Thanks for your help

 

[fabian]

not yet possible, something like only syncing the latest snapshot (for "seeding" of future syncs, or if you only ever want to keep 1-2 snapshots on the target side anyway and do aggressive pruning), or pruning the list of eligible snapshots similar to the prune mechanism for stored snapshots would be doable if there is demand. for the prune thing some consideration what the exact semantics should be needs to be done of course.

you can 'cheat' by copying the backup group dir (e.g., vm/123 -> vm/123sync) and then pruning and syncing the copy only (via group-filter), and finally renaming the group back to the original name on the target once you are done. obviously this is rather manual and error-prone, but it still might help you to kickstart your remote. future syncs will always start with the last local one (per group), any older snapshots are ignored.

 

[guerbywork]

not yet possible, something like only syncing the latest snapshot (for "seeding" of future syncs, or if you only ever want to keep 1-2 snapshots on the target side anyway and do aggressive pruning), or pruning the list of eligible snapshots similar to the prune mechanism for stored snapshots would be doable if there is demand. for the prune thing some consideration what the exact semantics should be needs to be done of course.

you can 'cheat' by copying the backup group dir (e.g., vm/123 -> vm/123sync) and then pruning and syncing the copy only (via group-filter), and finally renaming the group back to the original name on the target once you are done. obviously this is rather manual and error-prone, but it still might help you to kickstart your remote. future syncs will always start with the last local one (per group), any older snapshots are ignored.

Thanks for the quick reply.

Additionnal issue is that the primary is still running PBS 1.x so no group pruning.

Looking at the PBS API docs I see that group-filter has a "regex" type but I couldn't find what regex syntax this is using, I assumed https://docs.rs/regex/latest/regex/ from pbs-api-types/src/job.rs and tried a simple regex (first line) to match one backup (second line):

vm.177.2021................
vm/177/2021-09-29T23:06:24Z

But it didn't work.

Do you have a working regex example as group-filter ?

 

[guerbywork]

Ah I think I understand the regex matches only the group name not the full backup ID.

Having a feature to match on the full backup ID would solve my issue.

 

[fabian]

well the last part is the snapshot time, the first two components are what makes up the 'group' so yeah, filtering for that would be a 'snapshot-filter' which would probably be implemented as prune setting and not a regular expression (regular expressions are kind of a poor fit for matching a timestamp after all ).

 

[guerbywork]

well the last part is the snapshot time, the first two components are what makes up the 'group' so yeah, filtering for that would be a 'snapshot-filter' which would probably be implemented as prune setting and not a regular expression (regular expressions are kind of a poor fit for matching a timestamp after all ).

May be a simple snapshot copy/import operation (API) based on a snapshot id would allow user to do what they want between a remote and PBS (at least on initial setup) with standard scripting tools?

 

[fabian]

yeah, something like pull-snapshot that re-uses the pull code but only pulls a single snapshot for a single group would work as well - that can be scripted around to bootstrap a new datastore with just the latest snapshot (or the N-last one) of certain or all groups.

 

[guerbywork]

yeah, something like pull-snapshot that re-uses the pull code but only pulls a single snapshot for a single group would work as well - that can be scripted around to bootstrap a new datastore with just the latest snapshot (or the N-last one) of certain or all groups.

Let me know if you want me to open an enhancement bugzilla.

Thanks!

 

[fabian]

https://bugzilla.proxmox.com/show_bug.cgi?id=3701