Method to Copy PBS Backups Elsewhere?

[theprez1980]

I'm in the process of redoing my proxmox cluster setup and PBS server. I don't want to lose the VM backed up in PBS as I'll need all that to restore to the new instances of PVE.

Is it possible to connect a USB Drive or rclone or some other method to save what's in PBS' datastore and be able to place it back on a fresh install of PBS?

Said differently, I want to backup what's in PBS in order to reinstall PBS along with the previously backed up VMs so they can be restored to PVE.

Thanks

 

[meyergru]

When you look at how PBS works, you will understand that there are no individual backup "files" you can directly get out of it. Also, if you could, the sheer volume would probably kill you because of the deduplication factor.

You can, however, transfer a whole datastore as a data disk or the subdirectory / zfs where the PBS datastore resides. Essentially, what is in there is essentially the metadata for each backup and the chunks that make up the data. You can restore that data to another PBS instance and recreate the datastore in place.

You will have to use the same username that you did the backups with or change the owners of the backup groups.

 

[theprez1980]

When you look at how PBS works, you will understand that there are no individual backup "files" you can directly get out of it. Also, if you could, the sheer volume would probably kill you because of the deduplication factor.

You can, however, transfer a whole datastore as a data disk or the subdirectory / zfs where the PBS datastore resides. Essentially, what is in there is essentially the metadata for each backup and the chunks that make up the data. You can restore that data to another PBS instance and recreate the datastore in place.

You will have to use the same username that you did the backups with or change the owners of the backup groups.

Thanks for the detailed reply, that makes sense.

So no matter how I copy the /zfs directory either through rclone or some other method, and reuse the same User ID - I should be good to go then?

 

[andrew-transparent]

Adding to what meyergru has said, you could also attached a drive to your existing PBS server and then add it as a new repository and do a local sync from one datastore to the new one. It accomplishes the same thing but you'd be working in the GUI. When you setup the new PBS server you can add that datastore again. One of the benefits of doing the sync is you can choose how deep you want to sync (ie everything vs only the last x)

This post shows how that is done by editing the /etc/proxmox-backup/datastore.cfg
https://forum.proxmox.com/threads/datastore-recovery.72835/#post-325491


*note* be mindful of tabs/spaces in the datastore.cfg file. Its picky and you need to make sure you're using tabs and spaces in the right place or it will complain that it can't parse the file. This apply whether you are copying the files over manually or using PBS to sync them.

 

[meyergru]

I think so - I only did it by moving a whole disk containing a datastore, because my PBS is a VM.

By "user", I mean the PBS user. All Linux files seem to be owned by "backup:backup". The logical PBS user is kept in a file named "owner" in the backup group directory, I imagine you could also change its content.

And yes, you could as well mount your USB disk, create a new datastore and then sync it. That would also work via network if the new PBS instance can reach the old instance. In that case, you would not even need a transfer medium.

 

[cheiss]

Hi,

the easiest & most integrated method would be to use sync jobs - although that requires a secondary PBS instance.

Even more interesting to you would probably removable datastores, which are currently in the works.
These would allow you to add e.g. a removable disk conntected via USB as datastore and create a sync job to that.

 

[UdoB]

Even more interesting to you would probably removable datastores, which are currently in the works.

I would really like to see some of the work-in-progress mentioned on the roadmap https://pbs.proxmox.com/wiki/index.php/Roadmap and also https://pve.proxmox.com/wiki/Roadmap.

Both roadmaps should be a little bit more vivid. Remove the stroke through entries and nothing substantial is left...

 

[JensF]

Looking forward to see how you will do the implementation for that.
But I've to say, I'm happy with my own setup for our RDX drives already.

 

[Proksie_1]

I would really like to see some of the work-in-progress mentioned on the roadmap https://pbs.proxmox.com/wiki/index.php/Roadmap and also https://pve.proxmox.com/wiki/Roadmap.

Both roadmaps should be a little bit more vivid. Remove the stroke through entries and nothing substantial is left...

Nothing substantial left?
Then when is " Support (tape-like) syncing to S3/Object storage types " expected for Proxmox Backup Server? Usualy a roadmap comes with timeline...
Futher: I suppose that this would also make it possible to use a Truenas share as (offsite) storage? .... Which is now only possible using rsync (which is not recommended) or by the use of a second PBS instance, for example as a VM running under TrueNAS (Scale). Both is giving an extra layer of complexity in a simple 3-2-1 backup strategy for PBS.

 

[VictorSTS]

Even more interesting to you would probably removable datastores, which are currently in the works.
These would allow you to add e.g. a removable disk conntected via USB as datastore and create a sync job to that.

Please, add an option to removable datastores to automatically start a sync job + GC job + verify job when the disk is plugged in, so no need to add udev rules, write scripts or manual intervention just to click a few buttons. Also, an automatic disconnect of the drive once all the jobs are done would be wonderful

 

[Johannes S]

Thanks for the detailed reply, that makes sense.

So no matter how I copy the /zfs directory either through rclone or some other method, and reuse the same User ID - I should be good to go then?

rclone seems to break PBS datastores thus I wouldn't do this:
https://forum.proxmox.com/threads/datastore-synced-with-rclone-broken.154709/
https://forum.proxmox.com/threads/pbs-appears-not-to-write-to-disk.157751/

ZFS send/receive should work but I would throughly test that before trusting my backups with it. The issue is that PBS heavily relies on an attributed called atime which records the last access time of a file or directory. Depending how the data is transferred from one place to another this might get broken, resulting in damaged backups. rclone doesn't seem to handle it well (since rclone can't know about any PBS specifics) while PBS native sync job do.

I personally would use a removable datastore with your external disk in your case. For a offsite backup a cheap vserver or pbs cloud backup provider like tuxis.nl is propably your best bet. Or a mixed approach where you use PBS for local backups and Proxmox VEs native vzdump format on cloud storage for offsite backup. I do something similiar which I discussed here:
https://forum.proxmox.com/threads/backup-pbs-to-cloud-provider-with-incremental-chunks.157965/