Backing up ZFS Pool to another Pool?

[marchesmf]

Hi everyone, I am hoping someone can help out. First up, I am not super tech savvy, so I apologize if this is a dumb question, but I can't seem to find the answers I am looking for (or maybe I have and have not understood it properly).

I use Proxmox primarly to run a Plex LXC and as a server to access some personal media files throughout my household. Right now, I have a RAIDZ pool with all my media in it. Recently, I had two harddrives fail and lost all my data. So I am now rebuilding my media library and want to create a backup pool to protect against it happening again. I replaced the drives and remade the RAIDZ pool, and have an physical external enclosure of drives I want to use as a backup.

The idea is that the RAIDZ pool would backup the media files to this external JBOD, which I have made into its own pool. Ideally it would backup on a schedule, but I don't mind manually doing it if that is the best solution.

So, in short, I would have Pool_One (6 drives (60 TB), RAIDZ, installed in the PC) backing up the contents to Pool_Two (4 drives (48 TB), striped JBOD, enclosure connected via USB 3.0).

I think PBS would be the best way to do this at it seems mostly user-friendly with a pretty robust GUI. However, I can't seem to understand how to get this to work. I have two instances of PBS running and thought I could use Remote to get the first pool to dump into the second pool. However, the roadblock I have currently run into is that I cannot create a datastore for Pool_One as this pool already has content on it, so I get the error: unable to open existing chunk store path "/mnt/Pool_One" - permissions or owner not correct

What am I missing? What is the easiest way to go about backing up one ZFS pool into another?

Thanks for any help!

 

[tcabernoch]

Hi there. Interesting set of questions. Yes, its not to clear exactly what PBS is and does when you first encounter it.
In brief, I don't think PBS is the tool you are looking for.

PBS does use a Pool concept, but the only ZFS pools it knows about are its own storage.
https://pbs.proxmox.com/docs/storage.html
And you want something that can look at your storage and make copies, not something that needs to own the storage.

proxmox_backup_client can do file backups, but the target needs to be a datastore mounted in Proxmox, so either a PBS, NFS, local, or whatever.
https://pbs.proxmox.com/docs/backup-client.html
The output is a compressed file. I don't think that's what you want.

It sounds to me like you are looking for this info ...

zfs send:
-R: send everything under the given pool or dataset (recursive replication, needed all the time, includes -p). Also, when receiving, all deleted source snapshots are deleted on the destination.
-I: include all intermediate snapshots between the last replication snapshot and the current replication snapshot (needed only with incremental sends)

zfs recv:
-F: expand target pool, including deletion of existing datasets that are deleted on the source
-d: discard the name of the source pool and replace it with the destination pool name (the rest of the filesystem paths will be preserved, and if needed also created)
-u: do not mount filesystem on destination

And ... well ... are you sure this is the way you want to do this?
There's all kinds of apps to do this.
Or if you want to get fancy, rsync is a better tool than ZFS imho.

Good luck.

 

[marchesmf]

Hi there. Interesting set of questions. Yes, its not to clear exactly what PBS is and does when you first encounter it.
In brief, I don't think PBS is the tool you are looking for.

PBS does use a Pool concept, but the only ZFS pools it knows about are its own storage.
https://pbs.proxmox.com/docs/storage.html
And you want something that can look at your storage and make copies, not something that needs to own the storage.

proxmox_backup_client can do file backups, but the target needs to be a datastore mounted in Proxmox, so either a PBS, NFS, local, or whatever.
https://pbs.proxmox.com/docs/backup-client.html
The output is a compressed file. I don't think that's what you want.

It sounds to me like you are looking for this info ...

zfs send:
-R: send everything under the given pool or dataset (recursive replication, needed all the time, includes -p). Also, when receiving, all deleted source snapshots are deleted on the destination.
-I: include all intermediate snapshots between the last replication snapshot and the current replication snapshot (needed only with incremental sends)

zfs recv:
-F: expand target pool, including deletion of existing datasets that are deleted on the source
-d: discard the name of the source pool and replace it with the destination pool name (the rest of the filesystem paths will be preserved, and if needed also created)
-u: do not mount filesystem on destination

And ... well ... are you sure this is the way you want to do this?
There's all kinds of apps to do this.
Or if you want to get fancy, rsync is a better tool than ZFS imho.

Good luck.

Thanks so much for the information!

I'm definitely open to other apps that can do this. I have tried using Open Media Vault but an into some issues with it being able to recognize the mounted pools. But PBS is not the end all be all for me if something better exists

I recently used this method for making a backup: https://linuxconfig.org/how-to-create-a-backup-with-proxmox-backup-client.
However, since I have over 1TB of media already...it took a long time, around 7 hours. So I don't think it's sustainable to use as I continue to expand my media.

 

[tcabernoch]

My friend, if you are first examining the idea of backups and resiliency, then you need to consider external storage for your backup.

I too have a large media library.
For me, the source of truth for that library is the external drive that I keep it on.
If I've been collecting new media, every couple months I plug the drive in and back up the media.

There are various apps that can do the backup for you, but I like it simple.
I right click on my media folder and copy. Then I right click on my media backup folder and paste.
Windows will ask you if you want to overwrite, and just say No.
It will then just paste in the new files.
Yay. All backed up.

 

[marchesmf]

My friend, if you are first examining the idea of backups and resiliency, then you need to consider external storage for your backup.

I too have a large media library.
For me, the source of truth for that library is the external drive that I keep it on.
If I've been collecting new media, every couple months I plug the drive in and back up the media.

There are various apps that can do the backup for you, but I like it simple.
I right click on my media folder and copy. Then I right click on my media backup folder and paste.
Windows will ask you if you want to overwrite, and just say No.
It will then just paste in the new files.
Yay. All backed up.

Yes, this is exactly what I am used to doing, but I don't use Windows, I run everything in Proxmox via Linux. So it's a learning curve to figure out the best way to dump my media from one drive pool to another drive pool.

 

[Johannes S]

Yes, this is exactly what I am used to doing, but I don't use Windows, I run everything in Proxmox via Linux. So it's a learning curve to figure out the best way to dump my media from one drive pool to another drive pool.

If both have ZFS I would go with ZFS send/receive since then you can also transfer the snapshots which together with zfs-auto-snapshot will give you a history of your files.
If you don't have ZFS on both sides I would go with a dedicated tool, but which one depends on your usecase:

• If you want to do actual backups (including a history aka preserving old versions of the files) I would go with a deduplicating backup tool like restic/borgbackup/duplicati
• If you just want to sync your media files to another local storage drive (but one not formated with ZFS) rsync or one of it's fronends (like rsnapshot) would be my way to go
• If you want to sync the files to a cloud storage (NOT for backup, for this borg or restic are more suited!) rclone would be a good choice.

Please note that I'm only talking about bulk data here. For VM and LXC backups ouf of ProxmoxVE nothing beats ProxmoxBackup Server. But PBS and manual syncing of it'S datastores doesn't play nice together (we have several threads here in the forum where people broke their backups by using something like rclone for syncing their PBS datastore to a cloud provider so please don't do this. Syncing raw data or the native vzdump/vma Backups is fine though)