PBS - Recovery of Tape Archives from failed PBS (Host Disk Failure)

[jw6677]

I have a PBS install associated with a PVE cluster. To make a long story short, I am reinstalling PBS due to some pretty catastrophic data loss which impacted both PVE and PBS (ceph failure).

I have a tape library which has all of my archives, but need to reinstall PBS, and do not have access to the original PBS disk anymore.

Disaster recovery in full swing, before I dive too deeply into a fresh PBS install, and re-adding the tape drive, I wanted to post here to be sure I am not going to accidentally clobber the tape backups along the process.

As I do not have access anymore, I wanted to confirm:

When re-adding the tape library, do I need to be cautious of the names given to the drive, changer, etc?
Where in the process do I need to stop, to ensure recovery of the past pool, instead of creation of a new pool? (Or do I create a new pool and expect the media to recover itself?)

I am not seeing in the documentation a tape library disaster recovery, type section. Does that exist?

 

[dcsapak]

When re-adding the tape library, do I need to be cautious of the names given to the drive, changer, etc?

not really, this information is not saved anywhere on the tapes themself

if you use a changer you can simply use the 'inventory' function to restore the infos about the tapes/media-sets and if your packages are new enough (make sure to update after reinstallation)
you can check 'restore catalogs' which will also try to restore the information about the backups on tape

otherwise you'd have to restore the catalogs for each tape seperately, see also: https://pbs.proxmox.com/docs/tape-backup.html#restore-catalog

you're right that there is no explicit documentation how to recover with tapes from a disaster, i opened a bug for that: https://bugzilla.proxmox.com/show_bug.cgi?id=4408

 

[jw6677]

Thank you. For anyone in future finding this thread, while in a particularly vulnerable position, I came across whatever error is being explored over here:
https://forum.proxmox.com/threads/tape-job-failing-with-scsi-transport-error.115529/

Not able to restore anything :'( Hopefully it is worked through.

 

[jw6677]

Just adding to this saga now that things are slowly inching towards recovery.

I managed to restore a huge chunk of my backups from the tape library back to a datastore, which was exciting, but I immediately ran into the issue of missing the encryption key required to restore the backups.

PBS tape encryption keys could not be recovered from the drive, this was very disheartening for a week or so while I mourned the loss of my data.

Not giving up, I later discovered that the encryption key that I needed wasn't actually on the tapes, but was held on the PBS client, my PVE nodes.

Per: https://pve.proxmox.com/wiki/Storage:_Proxmox_Backup_Server
At /etc/pve/priv/storage/ the previous encryption key file (*.enc) needed to be copied to the new storage ID, and then /etc/pve/storage.cfg needed to be updated to include encryption-key 1.

It turns out that there is more than one type/place for encryption keys? Not 100% sure, but all of this should really be documented in the disaster recovery docs when they are prepared.

I hope this helps someone down the road!

 

[dcsapak]

PBS tape encryption keys could not be recovered from the drive, this was very disheartening for a week or so while I mourned the loss of my data.

why couldn't they be recovered?

Per: https://pve.proxmox.com/wiki/Storage:_Proxmox_Backup_Server
At /etc/pve/priv/storage/ the previous encryption key file (*.enc) needed to be copied to the new storage ID, and then /etc/pve/storage.cfg needed to be updated to include encryption-key 1.

the gui also provides an upload button for the encryption key when creating a pbs storage

It turns out that there is more than one type/place for encryption keys? Not 100% sure, but all of this should really be documented in the disaster recovery docs when they are prepared.

what do you mean exactly? there is this in the documentation:

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#storage_pbs_encryption

 

[guywhotypeslow]

Just adding to this saga now that things are slowly inching towards recovery.

I managed to restore a huge chunk of my backups from the tape library back to a datastore, which was exciting, but I immediately ran into the issue of missing the encryption key required to restore the backups.

PBS tape encryption keys could not be recovered from the drive, this was very disheartening for a week or so while I mourned the loss of my data.

Not giving up, I later discovered that the encryption key that I needed wasn't actually on the tapes, but was held on the PBS client, my PVE nodes.

Per: https://pve.proxmox.com/wiki/Storage:_Proxmox_Backup_Server
At /etc/pve/priv/storage/ the previous encryption key file (*.enc) needed to be copied to the new storage ID, and then /etc/pve/storage.cfg needed to be updated to include encryption-key 1.

It turns out that there is more than one type/place for encryption keys? Not 100% sure, but all of this should really be documented in the disaster recovery docs when they are prepared.

I hope this helps someone down the road!

Hey, I just wanted to check in and see how this wound up for you. I am currently in the process of implementing PBS with LTO6 tape. Having seen how reliant on the existing PBS system these backups seem to be I am trying to research and test what a total loss of a datacenter would look like because at first blush it kind of seems like losing your PBS means you are SOL even if you have tapes.

Whether or not you were able to recover your data, do you have any advice for someone who is just starting with PBS and tape to ensure they don't suffer the same headaches you did?

As for the encryption key being in two places and the other commenter seeming to think that didn't make since. Is it that you were using hardware encryption on the tape drive in addition to software encryption from PBS?

 

[dcsapak]

Hey, I just wanted to check in and see how this wound up for you. I am currently in the process of implementing PBS with LTO6 tape. Having seen how reliant on the existing PBS system these backups seem to be I am trying to research and test what a total loss of a datacenter would look like because at first blush it kind of seems like losing your PBS means you are SOL even if you have tapes.

no, you just have to install a new pbs instance, restore the encryption keys (if any), setup the drives/changer and restore the media-sets/snapshots to a datastore

if you use the encryption, you have to make sure you make backups of the key, otherwise it's not restorable (but that's the same for every data storage solution with encryption )

 

[guywhotypeslow]

no, you just have to install a new pbs instance, restore the encryption keys (if any), setup the drives/changer and restore the media-sets/snapshots to a datastore

if you use the encryption, you have to make sure you make backups of the key, otherwise it's not restorable (but that's the same for every data storage solution with encryption )

It all sounds nice and easy but I'm having difficulty finding information on how one would go about doing any of that. Care to point me in the right direction?

I have a two test PBS servers. One I created a datastore on and a second one I have moved the datastore into. How do I add that datastore into the second PBS server without wiping the data from it?

 

[dcsapak]

It all sounds nice and easy but I'm having difficulty finding information on how one would go about doing any of that. Care to point me in the right direction?

it's the same procedure as setting up a new pbs, no special handling there... the key can be imported with 'proxmox-tape key restore' (check 'man proxmox-tape' and https://pbs.proxmox.com/docs/tape-backup.html#encryption-key-management for more info)

I have a two test PBS servers. One I created a datastore on and a second one I have moved the datastore into. How do I add that datastore into the second PBS server without wiping the data from it?

what do you mean 'moved the datastore into' ? i thought it's about tape backups?

 

[guywhotypeslow]

what do you mean 'moved the datastore into' ? i thought it's about tape backups?

Ultimately yes I will be attempting the same thing with tapes but I am taking this one step at a time. So I am attempting a practice recovery where I only have the storage medium the backup was on and not the PBS or PVE from which it came.

 

[dcsapak]

for datastores it's different. if you just have the disk, you have to mount it (however it was mounted before) and manually edit the /etc/proxmox-backup/datastore.cfg to point a datastore to the path
with tapes, it's a bit easier to go from zero, since you just have to re-inventory the tapes and restore to a new datastore (from which you can then restore your vms/ct/etc.)

 

[guywhotypeslow]

for datastores it's different. if you just have the disk, you have to mount it (however it was mounted before) and manually edit the /etc/proxmox-backup/datastore.cfg to point a datastore to the path
with tapes, it's a bit easier to go from zero, since you just have to re-inventory the tapes and restore to a new datastore (from which you can then restore your vms/ct/etc.)

Good to know that tapes will be easier. Can you point me to resources that will explain how to do this with datastores? Even reading the manual it isn't all that clear what I am supposed to do. I should also mention that when creating the datastore originally I chose ZFS.

 

[dcsapak]

there is no real procedure documented for now, though we plan to implement an 'import' function (though not timeframe for that yet) so for now
you'd just have to edit the datastore config manually, best is to simply create a datastore open the file with an editor. the format should be simple enough to be understand
ofc if you have more concrete questions to your setup, feel free to ask them

 

[guywhotypeslow]

there is no real procedure documented for now, though we plan to implement an 'import' function (though not timeframe for that yet) so for now
you'd just have to edit the datastore config manually, best is to simply create a datastore open the file with an editor. the format should be simple enough to be understand
ofc if you have more concrete questions to your setup, feel free to ask them

An import function would be awesome. Glad to know you guys are already thinking of that. I did figure it out now.

For anyone who happens to stumble upon this:

Since my datastore is on ZFS I first had to import the zpool:

zpool import -f pool-name

I happened to know the zpool name I created. If I hadn't known presumably there is away to find it but I don't know how.


Find the directory the zpool is mounted to.

zfs list

Edit the config file at /etc/proxmox-backup/datastore.cfg

Add the datastore with following format:

datastore: datastore-name
 path /path/to/datastore

Note the single space before "path" is necessary.