no catalog written to tape

[BriBriBri]

Hello All,

I'm trying to be diligent in that I am currently simulating a "total loss" and trying to restore from our pbs tapes with only the tapes, our tape library, and a blanked server.

I have installed pbs to the blanked server. I have connected the tape library and imported the tape encryption key, and thirteen of our fourteen tapes were inventoried fine since catalogs were detected on them. Only one tape had no catalog. This tape is now about six hours into being "cataloged" manually since no catalog was found on it.

I note that this tape also happens to be our most recently written tape. Each of our weekly backups to tape writes to two tapes. And this is the second of those two tapes, written just this last weekend.

So, my question is: Why, of all our tapes, does this one, latest tape not have a catalog on it?

I have a theory: Our media pool is set to an allocation policy of "always" which creates a new media set with each backup job. So, last weekend's backup job created the media set and then was essentially "holding it open" until next weekend's job starts? At which point, next week, it would have "closed" the existing media set, written the catalog to the tape, and created a new media set. So, in the future, if I want to have the catalog written to our latest tape earlier, I should schedule the creation of a new media shortly after I expect the tape backup job to finish rather than leaving it to "always" create a new one when the next backup job runs. Is this correct?

Thanks!

--Brian

 

[dcsapak]

mhmm... normally the catalog is always written at the end of a backup job, the only thing i could imagine was either the backup job was interrupted (e.g. stopped or powerloss or similar), or it did not fit on the tape anymore and another one was requested but not found (which essentially is the same as the interrupted case, but a bit more complex)

 

[BriBriBri]

Odd. I can confirm that the backup job finished normally. I received the email from PBS stating the job was successfull and which tapes were used. I'm looking at it now (just to double-confirm).

So, I guess it's a mystery and hopefully just a one-off fluke. I might be able to try to reproduce this behavior after this weekend's writes.

 

[dcsapak]

So, I guess it's a mystery and hopefully just a one-off fluke. I might be able to try to reproduce this behavior after this weekend's writes.

ok, hope so too, if you can reproduce it though, please open a bug report here: https://bugzilla.proxmox.com with as much details as possible so that we can try to reproduce it locally (otherwise fixing it is probably not easy)

 

[BriBriBri]

Will do! In positive news, the manual re-cataloging succeeded overnight. It's an LTO8 tape so it took 11 hours.

I'll follow up here as well if there's any further information to share (as well as bug report).

 

[BriBriBri]

One note: We had an hba that proved untrustworthy early on when we first imported the encryption key and did the intitial inventory. With the info provided above (thanks!) I now find myself focusing on this (which I had forgotten about). I now suspect things got into an odd state (the content area of the pbs "tape backup" gui was throwing errors for this pool/media date, etc.). When I replaced the hba with one we know the tape drive/library likes, I then selected the tape that hadn't cataloged correctly and just chose "catalog" from the GUI. . . if this didn't search the tape anew for a catalog already on tape, and instead just began the manual cataloging process, perhaps that's our answer. I might have found one if I hadn't jumped to the "catalog" option.

Though, it did seem to at least try to find it. . . (this is from the post-hba-swap successful manual re-cataloging):

2024-01-22T17:37:28-08:00: searching for catalog at EOT (moving to EOT)
2024-01-22T17:38:46-08:00: no catalog found
2024-01-22T17:38:46-08:00: scanning entire media to reconstruct catalog

 

[dcsapak]

One note: We had an hba that proved untrustworthy early on when we first imported the encryption key and did the intitial inventory. With the info provided above (thanks!) I now find myself focusing on this (which I had forgotten about). I now suspect things got into an odd state (the content area of the pbs "tape backup" gui was throwing errors for this pool/media date, etc.). When I replaced the hba with one we know the tape drive/library likes, I then selected the tape that hadn't cataloged correctly and just chose "catalog" from the GUI. . . if this didn't search the tape anew for a catalog already on tape, and instead just began the manual cataloging process, perhaps that's our answer. I might have found one if I hadn't jumped to the "catalog" option.

Though, it did seem to at least try to find it. . . (this is from the post-hba-swap successful manual re-cataloging):

2024-01-22T17:37:28-08:00: searching for catalog at EOT (moving to EOT)
2024-01-22T17:38:46-08:00: no catalog found
2024-01-22T17:38:46-08:00: scanning entire media to reconstruct catalog

so that hba was only there for the inventory not the backup? then these message are probably legti and there is not catalog (for whatever reason) on the tape...

if you have the time, you could try to do a 'proxmox-tape catalog --drive <drive> --scan --force --verbose'

this will force a reread of the tape (like it did the first time) but with the verbose flag it will print more details, maybe something is visible/logged there that gives a hint why there's (seemingly) no catalog

 

[BriBriBri]

Well, I think we can call this a fluke or a test that went awry (perhaps due to the HBA issue that threw my processes (etc.) askew). I repeated my test scenario just now, using the last three weekends of written tapes as well as the original tape from the weekend where the catalog wasn't found (still in its original state, not overwritten).

I wiped my testing pbs server just to be sure to have a clean slate. . . and I was able to Inventory (with "restore catalog" checked) each tape successfully. No missing catalogs. Was ready to start restoring in about 20 minutes.

Sorry for the false alarm!