PBS Reinstall + Tape Disaster Recovery

[ahorner]

I have dome a search of the forum and also in the documentation and haven't found anything that specifically refers to recovery from tape in the case of a complete PBS failure (such as a fire or other disaster) which destroys both the PBS server and the physical datastores.

To somewhat simulate this, I created a test container on a PVE host, backed it up to my PBS's datastore, backed it up to tape, and then deleted the container on PVE, deleted the backup from my PBS's datastore, deleted the media pool and the encryption key. I noticed during this that I was unable to fully remove the tape from both the Content and Inventory tabs.

I successfully restored the encryption key from the tape. I cannot however see a way to recreate the Media Pool. I therefore cannot recover from the tape because there is no Media Pool to select.

How would I recover from tape in the scenario that a brand new PBS instance must be deployed with a brand new datastore?

 

[sterzy]

Hi,

have you run proxmox-tape catalog after putting a tape you want to restore into the drive? This will re-import your tape catalog and then you can restore your backups from there. Also see manual for more information [1].

[1]: https://pbs.proxmox.com/docs-2/tape-backup.html#restore-catalog

 

[ahorner]

Hi @sterzy,

I did try this yes, it didn't appear to have any affect (presumably because Content and Inventory already contained the required catalogue and tape data) and no Media Pool was created.

 

[sterzy]

I did try this yes, it didn't appear to have any affect (presumably because Content and Inventory already contained the required catalogue and tape data) and no Media Pool was created.

Alright, maybe there is a slight misunderstanding here. You don't need the media pool to be able to restore from tape, but rather a media set. At least that's my understanding at the moment. What do you see when you run proxmox-tape media content?

 

[ahorner]

Hi @sterzy,

Apologies for the confusion. I have done a little more testing since my last message and I think I can explain my situation better now:

I backed up ct/100 from PVE to my datastore on PBS. I then backed up ct/100 from the datastore to tape SN8300L5.

I deleted ct/100 from both PVE and my datastore on PBS.

I entirely removed the tape SN8300L5 from PBS with proxmox-tape media destroy SN8300L5 --force true so now the backup of ct/100 is completely removed from everywhere in PBS, and is only present on tape SN8300L5 which PBS no longer knows anything about.

I inserted the tape SN8300L5 and tried both clicking the Catalog button (Tape Backup -> my drive -> Catalogue) which resulted in:

TASK ERROR: media catalog exists (please use --force to overwrite)

And also running proxmox-tape catalog --drive external which resulted in:

TASK ERROR: media catalog exists (please use --force to overwrite)
Error: task failed (status media catalog exists (please use --force to overwrite))

As you can see below, tape SN8300L5 is no longer present on PBS and I do not wish to overwrite my entire PBS catalog, because I have other tapes already in the catalog which I need to keep:


Thanks!

 

[sterzy]

Ok, but this isn't comparable to actually having your PBS be destroyed, because in that case it obviously would have no knowledge of your media set or catalog for that matter. So I am not sure how comparable this really is.

If I understand you correctly, you basically just want to re-import that one tape into your already existing catalog?

 

[ahorner]

The other tapes in the screenshots I sent are from other media pools. Unless there is a specific technical reasoning it is not, I believe this scenario would be comparable to importing multiple pool/media-sets one after another to a new PBS

 

[dzidek23]

@ahorner did you get any further with this?
@sterzy we are in the exact same scenario, could you explain how can one recover PBS from a tape after a complete loss of the system?

Just to add some background...

We've happily created LTO tape backups for quite some time without actually thinking how to recover PBS from tape.
Having a spare server, decided to test a complete recovery from tape, and we found ourselves stuck at the recover PBS stage.

What we've done to date:
1. Install clean PBS to recover the tape (worked)
2. List available backups and recover the necessary files (done)

Here, one CAN'T use proxmox-backup-client to recover root of running server (or at least we couldn't figure out how to do it).
Booted up a Live Debian, installed proxmox-backup-client and that's fine, even mounted the LVM to get to PBS's drive (this is where the tape recovery resides).

And we're stuck... no repo to reference, no root.pxar to mount (the offline datastore holds root.pxar.didx only).

One possible option is to take a step back and download root.pxar from running PBS (one that has accessed the tape). Boot live distro and mount the archive with proxmox-backup-client mount, copy/paste files from archive to the root.

edit:
The above was a great plan, however it isn't possible to download root.pxar from the user interface?

If there's a way to easily recover PBS to its pre-disaster state, it would be good to include this in proxmox-backup-client documentation.

 

[ahorner]

@dzidek23 After being ignored over on this https://forum.proxmox.com/threads/encrpytion-light-on-tape-drive-lit.143935/#post-647252 post about what I would say is a critical tape security issue, I have lost all confidence in PBS and I am unsure what to do now regarding my backups to tape... I don't know anymore if my tapes are even encrypted properly, or possibly even corrupted.

 

[dzidek23]

@dzidek23 After being ignored over on this https://forum.proxmox.com/threads/encrpytion-light-on-tape-drive-lit.143935/#post-647252 post about what I would say is a critical tape security issue, I have lost all confidence in PBS and I am unsure what to do now regarding my backups to tape... I don't know anymore if my tapes are even encrypted properly, or possibly even corrupted.

I guess the only option is to test. And I would say do it sooner than later ... don't lose faith that quickly, Proxmox, both PVE and PBS, look good and work reliably for many of us and answers in the forum shouldn't be a gauge for a software.

 

[ahorner]

I guess the only option is to test. And I would say do it sooner than later ... don't lose faith that quickly, Proxmox, both PVE and PBS, look good and work reliably for many of us and answers in the forum shouldn't be a gauge for a software.

I don't know of a good way to test that. I can assure you, this hasn't been quick. PBS has many limitations. We don't even have a reliable linux client, which is yet another factor in my loss of confidence.

The fact the software provides no easy means to verify tapes, and the fact the tapes weren't encrypted by hardware before and suddenly are (hardwar enecryption was enabled as displayed by the LED on the drive, which to my knowledge should be impossible unless you want to lose previous data not encrypted by hardware) without any kind of warning, prompt, anything whatsoever from anyone, and that there is seemingly no answer as to whether the key is therefore backed up using the password onto the tape, as it used to be... Yea, I cannot trust it right now.

As for forum answers, I was replied to by staff, and then no further replies. That isn't a good show.

I need some answers before I can trust PBS again. I'm greatly disappointed in the lack of communication and lack of documentation regarding this. You shouldn't have to guess why a system is behaving the way it is, especially for something critical like backups. It should be predictable, it is not.

 

[ahorner]

@dzidek23 Did you manage to get any further?

 

[dzidek23]

@ahorner
Hi,

Well, we had some other, more serious issues recently, so the PBS recovery testing got paused. We've managed to recover PBS from a tape backup, but some things still need to be ironed out.

Roughly what we've done (this is from memory, so might not be 100% accurate):

1. Install PBS in version equal to what you were running
2. Add tape drive, catalogue tape and download what you need from the GUI in tar.zst format. Copy that to either separate partition or portable media.
3. Boot your server with PBS Install media, choose Advanced and go to the live distro (Ctrl+D if I remember correctly)
4. Mount your physical partition
5. Recover from the tar.zst
6. Reboot

At this stage we had a working server with the entire configuration recovered.

Issues...
We've learned that backing up just the /etc isn't enough. And that's what we are currently doing.
Yes, you get everything Proxmox related back, but that's not everything. Our problems are also caused by poor LVM configuration we chose at the time of initial installation. One large LVM partition holding root and datastore.

The plan is to reinstall the PBS with slightly smarter LVM partitioning and backup disks rater than files. Recovery should be much easier.