Offload copies for long term storage

[markf1301]

Hello

I think I noticed in version 3.3 (currently running 3.2-3) of Proxmox Backup that there was a way to 'push' copies of files to another datastore or location. As my 'long term' storage is getting full (backups kept for a long time like yearend or other requirement) and it still sits on my primary storage I was wondering if i can use one of my Synology servers to house/store that data. I've struggled to be able to copy files from the zfs datastore to my Synology in the past (can't seem to get the path/zfs send/etc to work) and wonder if I can use the Synology in this process. i know i can create a datastore easily on a Synology - but is it possible to replicate/push those older files there for long term storage? I'd like to keep any associated files (server #, cpu,network,etc) associated if i need to bring it back or mount it easily. Theoretically if i create another datastore on a Synology and push that data out there thru an automated task - then I can delete it from my primary storage, reclaim needed space there - and have a copy if I need to reload it in the future.

To sum this all up - this is what I'm trying to do.

replicate/pull from local to remote (DR site) as normal this process works.
replicate/push long term files from my DR site to another source so i can alleviate the storage constraints there and free up space.
due to the DR being 'remote' then it doesn't make sense to replicate that data from the production side, to the remote storage space. hoping i can replace/push/store a copy of the long-term files from the DR/remote side - to what is local there, the DR side.

Does this make sense?
What are others using for long-term storage of files if they don't have unlimited budgets or a jbod laying around?

suggestion are welcomed.

 

[weehooey-bh]

If your Synology is local to your DR PBS, you could present it to the PBS with either iSCSI or NFS. Once mounted on the DR PBS, you could put a PBS Datastore on it. Then, you would use a Sync Job to move the backups from the primary PBS Datastore to the Synology-based Datastore.

IMPORTANT: The performance of the Synology-backed Datastore will likely be poor terrible. However, it sounds like it would meet your budget objectives.

Instead of Synology, you could use an external USB drive or other removable media. Connect the drive, run the Sync Job and then remove the drive. It is more manual, but the archive copy will be an offline copy and possibly acceptable for the compliance backups you need.

 

[markf1301]

Hello. Let me clarify things a bit.
The remote side is where the DR side is and also where the current backups are 'pulled 'to. This is pretty normal. with storing all backups (long term) at the remote side I am trying to 'offload' some of those files to another repository that way i have more history kept. i don't want to create another 'repository' - just have the ability to offload some of the files there, but retain that info in case i need to load them back up in the future. for example, the last 2 years of 'yearly backups' from my current site - could be offloaded to the other location and then removed from the existing location. does this make sense? is there an easy way to mount/set up an usb-drive (multi tb space) that these files could be moved to?

 

[weehooey-bh]

just have the ability to offload some of the files there, but retain that info in case i need to load them back up in the future

Because of the way PBS handles the data, you will need to use another Datastore ("repository" is a misnomer) and a Sync Job to move the data.

is there an easy way to mount/set up an usb-drive (multi tb space) that these files could be moved to?

Yes, you can use Removable Datastores.

 

[markf1301]

Because of the way PBS handles the data, you will need to use another Datastore ("repository" is a misnomer) and a Sync Job to move the data.



Yes, you can use Removable Datastores.

Thanks - I'll look into the removable datastores as an option. I was hoping for a way to just ssh (copy) the data from it's current location to another 'dummy' device (synology box) and use that as my existing cheap long term storage. I'd of course want to retain the 'config' info with it (drive,cpu,mem,etc) in case i needed to restore (copy back and mount/load) it for reference.

 

[weehooey-bh]

I was hoping for a way to just ssh (copy) the data from it's current location to another 'dummy' device (synology box) and use that as my existing cheap long term storage

Mounting the NFS share from your Synology and using a Sync Job requires a couple of extra steps to set up, but once it is set up, it will be automatic and will not require you to SCP the files over manually. It would be slow, but the speed might be acceptable, given your use case.

 

[markf1301]

Mounting the NFS share from your Synology and using a Sync Job requires a couple of extra steps to set up, but once it is set up, it will be automatic and will not require you to SCP the files over manually. It would be slow, but the speed might be acceptable, given your use case.

Agreed - NFS and ISCSI can typically be slower depending on 'local' storage on the box or your 'local' shared device. The reasoning I haven't tried to do this before is that I wanted to do this step, from the remote site to the remote side and not replicate across a remote link which makes sense because the data is already 'local' to the remote side or by using the PBR which is remote to the Synology, which is also remote - on on the same network. Doing this initially (seeding the backups) is quite lengthy in time.

 

[weehooey-bh]

from the remote site to the remote side and not replicate across a remote link which makes sense because the data is already 'local' to the remote side or by using the PBR which is remote to the Synology, which is also remote - on on the same network

I may not have understood your topology correctly.

My understanding was you had a PBS local to your PVE (site #1). Then, you had a remote site (site #2) with a PBS and a Synology together.

From this last post, it sounds like you have a PBS local to your PVE (site #1), a remote site with another PBS (site #2) and another remote site with the Synology (site #3).

If you have three sites, I would look for a way to get a PBS on the third site, as a VM on the Synology or as a separate physical box. PBS-to-PBS is a much better way to move the backups.

 

[markf1301]

I may not have understood your topology correctly.

My understanding was you had a PBS local to your PVE (site #1). Then, you had a remote site (site #2) with a PBS and a Synology together.

From this last post, it sounds like you have a PBS local to your PVE (site #1), a remote site with another PBS (site #2) and another remote site with the Synology (site #3).

If you have three sites, I would look for a way to get a PBS on the third site, as a VM on the Synology or as a separate physical box. PBS-to-PBS is a much better way to move the backups.

Apologies for the confusion. Here is my current config -

Production site - PVE1 as main server also a PVE2/PBS (dual roles on it - not recommended but works for failover and has a good sized datastore).
DR/Remote site - single PVE3/PBS.
The production site has scheduled backups to the datastore on PVE2. The PBS side of PVE2 has scheduled prune jobs which keep the data for a shorter time and allows a failover and loading of 'archived/restored' data if needed - local to the production side.

DR/Remote side has a single server with dual roles - PVE3/PBS. Failover if needed and also the PBS side of it will pull from the production side the backups and store them for a longer time. For instance I might only keep 6 months of backups on the production side but the DR side has 6 years of backups.

Since the datastore on the DR side is getting close to full I want to archive off - say 4 years of backups from the datastore there (which would reduce the datastore size and free up more space) but have them for long term storage on some other 'media'. In this manner (crazy thoughts i know) I could have 10 years of (production) backups and be able to restore (copy them back to either the DR side or production) those vm's if needed.

Since the data is already 'semi-long term' at the DR side I just want to offload that data from the datastore there, but keep it for later. It's just a 3-2-1 method in that the production side has (shorter data) local on it - the DR Side (2) has data for a few years, and (1) has data for multiple years.

With the data (datastore) being at the DR side already (and the Synology box as well) I'm looking for a method or archiving/removing data from (option 2 in this case) over to option 1 for longer term storage.

Hopefully this makes sense. I think there might be 2 options here and am looking for advise to which is better: Removable Storage, or setting up another PBS at the Dr side. I was trying to reduce the need for another PBS at the DR side just for long term storage and utilize the Synology as the datastore (repository) for long term storage. That said, shouldn't I be able to create a NFS/ISCSI on the Synology box at the DR side and then create a 'Sync job' to push the data to it - for long term use? This might have been mentioned and I failed to 'listen' earlier.

 

[weehooey-bh]

Ah! Thanks for that overview @markf1301

Your architecture makes sense. Yes, having dedicated hardware for PBS is the ideal situation, but there are many use cases where that is not possible for technical reasons or resource constraints (e.g. money, time).

Based on what you have mentioned and my understanding at this point, my approach would be something like this:

DR PBS (PVE3/PBS) runs a Pull Sync Job from the main datastore on Prod PBS (PVE2/PBS), which is precisely what you have right now.

Then, I would mount shared storage (iSCSI or NFS) from the Synology on DR PBS and create a Datastore on it called something like "LongTerm".

On DR PBS, run a local sync job from the "main" datastore to "LongTerm". It would have its own Garbage Collection and Prune Jobs.

I would test NFS and iSCSI for performance and select the best one to back LongTerm.