Using Unraid as a Datastore to then push to tape?
- Thread starter SC8198
- Start date
while it's possible to use a remote fileshare as a datastore, we generally recommend against that for performance reasons. Every bit of latency and slowing down to the files of the datastore increase the time to verify/garbage collect, etc.
Gotcha. Well my delima is that the machine running the tape storage only has 12 TB of disk. Unraid server of around 60tb to back up. Is there a way to use pbs to where I move data to datastore then move to tape. Then remove from datastore and then add more storage to then move to tape? If not I can explore elsewhere.
i did not say you cannot use the unraid share as a datastore, just that we recommend against it for perfomance reasons
if you recreate and delete the datastore each time, you won't gain anything versus just keeping the datastore around
probably i'd just try it and see how it performs
I think I'm trying to do a very similar thing as SC8198 and have a couple of questions about it. I currently have about 100TB of data that I want to back up to tape (LTO-8). I want to use PBS to write the data to tape in 12TB batches, because I don't have the budget to build a performant PBS server with a 100TB+ size data store, and I don't even need to keep most of the data in hot storage anymore (e.g. old video projects). I just want to keep it as an archive on tape (two copies) and want to use PBS to write those tapes because I am planning to build a PBS backup server anyway for hot backups of VMs.
I'm thinking of putting in 12TB of SSD storage for a datastore which simply functions as a buffer to push data onto, then it's written to tape, then deleted again from that datastore when the tape backup job is done. That way I would be able to archive the 100TB in about nine batches. And then in future when I have new data that I want to archive, I push it to the SSD datastore again, write to tape, and then wipe it off the SSDs again.
If I delete the data from the datastore, will PBS somehow delete or overwrite it on the tapes as well, or can it stay on the tapes inside the attached library? And is it possible to delete the data that has been written to tapes from the datastore while keeping the catalog for the tapes that were written in PBS so that if I did want to restore a file from tape, I can easily find and restore it to the datastore?
I'm thinking of putting in 12TB of SSD storage for a datastore which simply functions as a buffer to push data onto, then it's written to tape, then deleted again from that datastore when the tape backup job is done. That way I would be able to archive the 100TB in about nine batches. And then in future when I have new data that I want to archive, I push it to the SSD datastore again, write to tape, and then wipe it off the SSDs again.
If I delete the data from the datastore, will PBS somehow delete or overwrite it on the tapes as well, or can it stay on the tapes inside the attached library? And is it possible to delete the data that has been written to tapes from the datastore while keeping the catalog for the tapes that were written in PBS so that if I did want to restore a file from tape, I can easily find and restore it to the datastore?
no, the data on the tapes is not touched in that case
yes, as i said the data on the tapes is not touched then.. just note that only 'snapshots' are backed up to tape, not individual files, so if you want to restore a single file from the backup, you have to first restore the snapshot from tape to a datastore, and then
you can restore the file from the backup
Thanks for the fast response, that's good to hear.
So if I understand correctly, even if the contents of the snapshots are deleted from the datastore after backing up to tape, the index/catalog of the snapshots remains on PBS so it can be searched for easy restore from tape, without needing to read the catalog of each tape?
So if I understand correctly, even if the contents of the snapshots are deleted from the datastore after backing up to tape, the index/catalog of the snapshots remains on PBS so it can be searched for easy restore from tape, without needing to read the catalog of each tape?
I have a couple of questions. If I were to do a similar thing to what pibber is doing. How do I get PBS to backup the files from a datastore? When I added my NAS as a datastore none of the files showed up? I guess I am confused on how proxmox sees files on the datastore?
Would the correct way be have a datastore (12TB of disks) that I would then push a host backup to? Then PBS would push that backup to tape? Then as pibber said could delete that data from the data store and push a new host backup of fresh data?
Sorry this is a little confusing for me.
Would the correct way be have a datastore (12TB of disks) that I would then push a host backup to? Then PBS would push that backup to tape? Then as pibber said could delete that data from the data store and push a new host backup of fresh data?
Sorry this is a little confusing for me.
not super clear what you mean here, but i try to explain some basiscs, maybe it will clear itself up with that
a pbs datastore is a place for incoming backups with the proxmox-backup-client, and only these 'snapshots' can be managed with pbs (and backed up to tape)
pbs is not a general file-to-tape backup solution, you always have to go through proxmox-backup-client -> datastore first (which will take space + time of course)
does that make it clearer?
I believe yes it does. So say I wanted to backup a folder from my Unraid machine. I would use the client to do a host backup to my 12tb datastore. Back that up to tape. Then as pibber suggested delete the data store backup and do it all over again?
Sorry for being confusing I really appreciate the help.
Sorry for being confusing I really appreciate the help.
related question: is there a specific reasion why you want to backup the files with pbs ? wouldn't it make more sense to directly backup it to tape with e.g. tar or other solutions that are made for direct file backup?
it just seems like a strange workaround in pbs to achieve this, when there are probably other easier solutions...
it just seems like a strange workaround in pbs to achieve this, when there are probably other easier solutions...
I'll be honest I really like the Gui proxmox has. I will be able to easily search backed up files to tape. I have spent the last few weeks trying to find an easier way to backup my data to tape. PBS is way more intuitive than other tape backup solutions. (that are free and open source) Didn't really wanna pay for a solution and also didn't want to mess with a ton of configuration files and cli based configuration.
Oh also I am wanting to move all my servers to Proxmox VE nodes and a cluster. (possibly ceph as well I'm the future).
Oh also I am wanting to move all my servers to Proxmox VE nodes and a cluster. (possibly ceph as well I'm the future).
Last edited: