Delayed offsite sync

[Elbandi]

Hi,

I use pbs with following prune, backup running every night:
keep-daily 6
keep-weekly 6
keep-monthly 6

I want to sync to a offsite location, but the traffic is limited (bytes). So i dont want to sync every daily backup, just the weekly (+ monthly). How can i do that?
Maybe i need a new sync paramter same as transfer-last but opposite side: skip-first N.

 

[Dunuin]

You could create multiple namespaces and multiple backup jobs. For example do a daily backup job every monday to saturday (with keep-daily 6) to a "daily" namespace and a weekly backup job (with keep-weekly 6 + keep-monthly 6) every sunday to the "weekly" namespace. Then only sync the "weekly" namespace to the offsite PBS. This is how I solved it.

 

[Elbandi]

You could create multiple namespaces and multiple backup jobs. For example do a daily backup job every monday to saturday (with keep-daily 6) to a "daily" namespace and a weekly backup job (with keep-weekly 6 + keep-monthly 6) every sunday to the "weekly" namespace. Then only sync the "weekly" namespace to the offsite PBS. This is how I solved it.

Thx, thats a good idea!

 

[Elbandi]

You could create multiple namespaces and multiple backup jobs. For example do a daily backup job every monday to saturday (with keep-daily 6) to a "daily" namespace and a weekly backup job (with keep-weekly 6 + keep-monthly 6) every sunday to the "weekly" namespace. Then only sync the "weekly" namespace to the offsite PBS. This is how I solved it.

i tested, but not fully good: i lost the dirty-bitmap feature on sunday and monday too:
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared

 

[Dunuin]

Did you maybe restart one of the VMs, reboot the host or use a stop-mode backup job? As soon as the VM gets stopped the dirty-bitmap will be dropped.

 

[Elbandi]

Did you maybe restart one of the VMs, reboot the host or use a stop-mode backup job? As soon as the VM gets stopped the dirty-bitmap will be dropped.

All vm bitmap goes to invalid. there was no reboot. and backup mode: snapshot.

 

[fabian]

the bitmap will also be dropped if the backup target changes (since the last snapshot in the "new" target won't match the previous snapshot that the VM knows about).

note that there is no difference between the backups semantically, so you could just sync once a week with `--transfer-last 1` to only sync the latest snapshot in each run. another (more complicated) approach would be to always do the backups to a single namespace, and then use syncing locally to different namespaces with different prune settings to create different "views" on top, and then sync those views to your remote PBS.

 

[Elbandi]

the bitmap will also be dropped if the backup target changes (since the last snapshot in the "new" target won't match the previous snapshot that the VM knows about).

note that there is no difference between the backups semantically, so you could just sync once a week with `--transfer-last 1` to only sync the latest snapshot in each run. another (more complicated) approach would be to always do the backups to a single namespace, and then use syncing locally to different namespaces with different prune settings to create different "views" on top, and then sync those views to your remote PBS.

Thx. Can i do this "sync" with linux commands in crontab?
Something like this:

# create vm/XXX dirs
find daily  -maxdepth 2 -mindepth 2 -type d -printf "%P\n"|while read id; do mkdir -p weekly/$id ; done

# Copy last sunday dirs to weekly
LAST_SUNDAY=$(date -d'next sunday - 1 week' +%Y-%m-%d)
find daily -type d -name "${LAST_SUNDAY}*"  -printf "%P\n"|while read backup; do rsync -arv daily/$backup/ weekly/$backup/; done

 

[Dunuin]

I think fabian meant the built-in "sync" jobs of PBS.
So doing all backup to a "daily" namespace (monday to sunday) on the onsite PBS. Then creating a new "remote" with "localhost" as target IP so the onsite PBS is pointing to itself. On the onsite PBS you could then create a sync job that runs once a week (lets say sunday) that sync only the last backup snapshot (with "--transfer-last 1") from a "daily" to a "weekly" namespace. You can then use another sync job on your offsite PBS to only pull the backup snapshots from that "weekly" namespace.
So would be similar to what I initially wrote but as all backups are done to a single namespace the dirty-bitmaps won't be dropped.

 

[Elbandi]

I think fabian meant the built-in "sync" jobs of PBS.
So doing all backup to a "daily" namespace (monday to sunday) on the onsite PBS. Then creating a new "remote" with "localhost" as target IP so the onsite PBS is pointing to itself. On the onsite PBS you could then create a sync job that runs once a week (lets say sunday) that sync only the last backup snapshot (with "--transfer-last 1") from a "daily" to a "weekly" namespace. You can then use another sync job on your offsite PBS to only pull the backup snapshots from that "weekly" namespace.
So would be similar to what I initially wrote but as all backups are done to a single namespace the dirty-bitmaps won't be dropped.

I understood the buildin sync, but i dont like it: if for some reason a sync is missed, the weekly namespace is left behind, and need a manual run to catch up with yourself. With crontab job, i can sync every day, and skip if already copied.

 

[fabian]

no, that rsync command is not complete and doesn't handle chunks or in-progress snapshots.. please use the built in sync/pull mechanism - the pull variant can also be scheduled by you if you don't trust the built-in scheduling

 

[Elbandi]

no, that rsync command is not complete and doesn't handle chunks or in-progress snapshots.. please use the built in sync/pull mechanism - the pull variant can also be scheduled by you if you don't trust the built-in scheduling

how to handle if main pbs is offline (maintenance, etc) on sunday? will the build-in schedule run the skipped job when pbs is online again?

 

[fabian]

yes. it won't run it twice though if you skipped two whole iterations. so if your schedule is every sunday at 1pm, it's now monday, and the last run was 2,5 weeks ago, it will run once now, and then again next sunday. if is now sunday 11am, and the last run was on monday, it won't run until 1pm. if the last run was before last sunday 1pm, then it will run once now, and then again at 1pm.

 

[Dunuin]

And like already said...you could write your own script and let that check when the last successful/failed sync job finished and manually trigger sync jobs via the CLI or API.

 

[Elbandi]

Thx for the answers. the ultimate stable solution is to sync everything to a "view" namespace, prune hourly/daily snapshots, and sync the rest to offsite. the only issue is that this copy the hourly/daily snapshots (actually just the index blobs) too.

I wanted to skip this view namespace, so i added a little changes to pbs for me: there is a "weekly only" checkbox in syncjob properties, and if checked, only the weekly snapshots are synced. the snapshot is selected by same logic as keep-weekly prune: the latest of given week.

https://github.com/Elbandi/proxmox-backup/tree/weekly-only