How to push sync only a subset of backups?

[soupdiver]

I have a local PBS server to which I backup from different local Proxmox hosts. I keep backups like 12x hourly, 7x daily, 4 weekly ....

I want to push sync this PBS to an offsite. But I dont want to sync all of the local backups. Just a subset like 1x hourly, 1x daily ...
or just the last 1-2 of hourly/daily/weekly

is there a way to configure my sync job like this?

 

[tcabernoch]

Hey, that's a great question, but the answer is broad and goes in various directions.
You need to look at the Sync Job, and might separately consider Namespaces.

--------------------------------------------------------------------
Sync Job Advanced Setting

Put a check in the Advanced box at the bottom of your Sync Job.
You'll get a setting called "Transfer Last".
Adjust it from All to however many you want.




--------------------------------------------------------------------
Sync Job Group Filter Tab

You can either Include or Exclude, so if you can come up with a criteria, you can use it to filter.

Type (VM, Container, or Host)
Group (lousy name that means VMID)
Regex (and this can do anything at all, but take you weeks to figure out)




--------------------------------------------------------------------
Namespaces

You can create virtual directories that allow you to separate your backups and manage them.
They aren't really 'folders' in a file system, but they work pretty much the same way.
So if you wanted to break out the backups from these various hosts you mention into their own folders, this is how.

 

[atomique]

Thanks a lot for your question @soupdiver - I am also facing this issue. I just want to transfer my weekly backups to my offsite pbs, but it doesnt work.



@tcabernoch Can you go deeper into the namespace topic? How do I move the weeklys automatically to a namespace (for example to a namespace named "weekly") and after that how do I sync it to the offsite pbs?

Thanks a lot
atomique

 

[soupdiver]

I couldnt figure it out either... I just push the last 1 backup for now and do pruning on the receiving side. Not really what I wanted but it works for now

 

[Johannes S]

For namespaces you have several options to implement your backup scheme:

- First you would create several namespaces for your local backups like localdaily, localweekly, localmonthly and for your remote backups (meaning the backups you want to sync to your remote PBS) remotedaily, remoteweekly, remotemonthly etc backups. Depending on your usecase you don't need all of them, just do what you are comfortable with.
- Create a prunejob on your local namespaces (e.g. keep 24 hourly, 30 daily backups, 5 weekly and 12 monthly)
- Afterwards you have several options:

1. Configure additional backup jobs with a schedule of your wish. Data which is already in the datastore won't be added again
2. First create a sync job on your local PBS to sync from your local namespace/s to your remote namespace, e.G. localdaily to remotedaily. Afterwards create a prune job with your intended schedule for the remote backups for the namespace you want to sync to your remote pbs (e.g. keep 6 hourly, 14 daily, 2 weekly and 6 monthly). As final step create a sync job to push the remote namespace to your remote pbs.

I personally do it like this (I actually have different numbers, modified to make things easier):

• A local PBS has a namespace pve-cluster for the local backups, it's prune schedule is something like twenty-four hourly, thirty daily, six weekly and twelve monthly backups.
• I have a second namespace called remotesync, a local syncjob sync all backups from pve-cluster to remotesync every hour (e.G. 0:00, 1:00, 2:00 etc), a prunejob makes sure that I just keep the last seven daily, six weekly and twelve monthly backups.
• A pull-sync job on my remote PBS pulls everything new from the remotesync namespace on my local PBS. If I wanted I could also do this as a push-sync but I prefer pull-sync for security reasons (since no local server of mine need write-access to the remote PBS->Great for ransomware protection).

 

[soupdiver]

> First you would create several namespaces for your local backups like localdaily, localweekly, localmonthly and for your remote backups

Ah okay yea that kinda makes sense. I did not think about difference namespaces for the different intervals. I currently have one namespace for each of my machines I backup... I guess I would to multipy that then. I will see if this outweights the benefit

thanks!

 

[Johannes S]

> First you would create several namespaces for your local backups like localdaily, localweekly, localmonthly and for your remote backups

Ah okay yea that kinda makes sense. I did not think about difference namespaces for the different intervals. I currently have one namespace for each of my machines I backup... I guess I would to multipy that then. I will see if this outweights the benefit

Data which is already stored in the datastore (no matter in which namespace and from which vms/hosts) will not be saved again. Thus this shouldn't cause much overhead. Actually I think it's enough to have one namespace per cluster for your local backups and another one for the remote. This is (see above) how I do this in my homelab

Having dedicated namespaces per cluster is a good idea since this ensures that no vms/lxcs backups from several clusters and identical ids would get mixed up. Having dedicated namespaces per node inside one cluster shouldn't hurt but also shouldn't be needed at all.

 

[soupdiver]

Data which is already stored in the datastore (no matter in which namespace and from which vms/hosts) will not be saved again. Thus this shouldn't cause much overhead. Actually I think it's enough to have one namespace per cluster for your local backups and another one for the remote. This is (see above) how I do this in my homelab

Having dedicated namespaces per cluster is a good idea since this ensures that no vms/lxcs backups from several clusters and identical ids would get mixed up. Having dedicated namespaces per node inside one cluster shouldn't hurt but also shouldn't be needed at all.

I meant dealing with multiple namespaces with overhead, not storage wise.
But year doing it the way you described makes sense

 

[lco-sp]

For namespaces you have several options to implement your backup scheme:

- First you would create several namespaces for your local backups like localdaily, localweekly, localmonthly and for your remote backups (meaning the backups you want to sync to your remote PBS) remotedaily, remoteweekly, remotemonthly etc backups. Depending on your usecase you don't need all of them, just do what you are comfortable with.
- Create a prunejob on your local namespaces (e.g. keep 24 hourly, 30 daily backups, 5 weekly and 12 monthly)
- Afterwards you have several options:

1. Configure additional backup jobs with a schedule of your wish. Data which is already in the datastore won't be added again
2. First create a sync job on your local PBS to sync from your local namespace/s to your remote namespace, e.G. localdaily to remotedaily. Afterwards create a prune job with your intended schedule for the remote backups for the namespace you want to sync to your remote pbs (e.g. keep 6 hourly, 14 daily, 2 weekly and 6 monthly). As final step create a sync job to push the remote namespace to your remote pbs.

I personally do it like this (I actually have different numbers, modified to make things easier):

• A local PBS has a namespace pve-cluster for the local backups, it's prune schedule is something like twenty-four hourly, thirty daily, six weekly and twelve monthly backups.
• I have a second namespace called remotesync, a local syncjob sync all backups from pve-cluster to remotesync every hour (e.G. 0:00, 1:00, 2:00 etc), a prunejob makes sure that I just keep the last seven daily, six weekly and twelve monthly backups.
• A pull-sync job on my remote PBS pulls everything new from the remotesync namespace on my local PBS. If I wanted I could also do this as a push-sync but I prefer pull-sync for security reasons (since no local server of mine need write-access to the remote PBS->Great for ransomware protection).

Hi,

I've been trying to do this but I've run into a problem creating the "local syncjob" (I'm relatively new to working with PBS so I'm probably misunderstanding something or making a beginners mistake somewhere). I've set up a second namespace, but when attempting to create the syncjob (Datastore > Sync Jobs > Add > Add Pull Sync Job) PBS returns the error "source and target datastore can't be the same (400)"

Can you tell me what I'm missing?

 

[Johannes S]

As far I know a local sync is only possible between two different local datastores, not namespaces. At least this was the case when I did my setup back with PBS 3.4. I mitigated this by adding localhost as remote and use it in a remote sync. Obviouvsly this "remote sync" is actually on the local PBS, but that's just cosmetics. Otherwise it works for me.

 

[lco-sp]

As far I know a local sync is only possible between two different local datastores, not namespaces. At least this was the case when I did my setup back with PBS 3.4. I mitigated this by adding localhost as remote and use it in a remote sync. Obviouvsly this "remote sync" is actually on the local PBS, but that's just cosmetics. Otherwise it works for me.

Thanks (particularly for the quick response)! Works for me too

 

[lco-sp]

As far I know a local sync is only possible between two different local datastores, not namespaces. At least this was the case when I did my setup back with PBS 3.4. I mitigated this by adding localhost as remote and use it in a remote sync. Obviouvsly this "remote sync" is actually on the local PBS, but that's just cosmetics. Otherwise it works for me.

Okay, it is working (backups are getting synced fine), however the process is also throwing a number of errors (one per VM group) each time the syncjob runs. It spawns a reader task that's not ending properly, always finishing on "connection error: not connected". It looks to me as if the internal network connection is closed too aggressively, though I cannot explain just why that is. Shouldn't the server be behaving as if a "truly" remote system were connected to it?

 

[Johannes S]

Okay, it is working (backups are getting synced fine), however the process is also throwing a number of errors (one per VM group) each time the syncjob runs. It spawns a reader task that's not ending properly, always finishing on "connection error: not connected". It looks to me as if the internal network connection is closed too aggressively, though I cannot explain just why that is. Shouldn't the server be behaving as if a "truly" remote system were connected to it?

Yes it should, such things never happened to me. The only errors I got were permissions related due to wrongly set permissions or user.

Could you please post a complete error log from the task and your configuration of the task and remote?

 

[lco-sp]

Yes it should, such things never happened to me. The only errors I got were permissions related due to wrongly set permissions or user.

Could you please post a complete error log from the task and your configuration of the task and remote?

This is one of the reader tasks (one exists for each group, they are all the same except for the first one after a new backup was written; In those, all the chunks are downloaded first but it ends on the same error)

pve_user@pbs is the account used by the PVE to push backups to the PBS.

the permission setting for this datapool


the remote config

 

[SteveITS]

I dont want to sync all of the local backups. Just a subset

I know this post is old, but just to point it out, the advanced options of a sync job have a "transfer last" count. This will limit the sync to the last "n" copies of each backup. Which is not quite what OP asked but will avoid transferring a giant datastore with dozens of backups for each VM, one VM at a time.

 

[guerby]

Related bugzilla : https://bugzilla.proxmox.com/show_bug.cgi?id=7138