What is the data format on tapes written by PBS?

[pibber]

What format/file system is the data stored on tapes in? From reading other threads I guess it's NOT LTFS, and it can't be BRU due to that proprietary nature. Is it tar? Or some other format?

I ask for two reasons:
1) I'm trying to understand what is needed in order to restore the tapes in case of a complete failure of the backup server. Is it enough to just have the tapes (and potential encryption keys)? Does the catalog or anything else need to be backed up separately, or is it all stored on the tapes (as with LTFS)?

2) I wonder if PBS append new data to an existing tape without risking the loss of previous data due to being overwritten? Or asked in a different way, does PBS manage and know what tapes have space and where and decides on the best location to write new data based on the size of data to be written?

 

[dcsapak]

What format/file system is the data stored on tapes in? From reading other threads I guess it's NOT LTFS, and it can't be BRU due to that proprietary nature. Is it tar? Or some other format?

it's a custom format, sadly we did not have time yet to properly document it (it can be found here in the code: https://git.proxmox.com/?p=proxmox-...6cc869eea87148f06adf7149;hb=refs/heads/master )

in short, we make use of ltos file marks to be able to "quickly" navigate around the tape if needed (and have some sanitiy checks) . we generally write 'chunk archives' 'snapshot archives' and 'catalog archives' on the tape
if you want more details, you can dive into the code, or ask me here

1) I'm trying to understand what is needed in order to restore the tapes in case of a complete failure of the backup server. Is it enough to just have the tapes (and potential encryption keys)? Does the catalog or anything else need to be backed up separately, or is it all stored on the tapes (as with LTFS)?

you need a working (and compatible) backup server, as we save a custom format on the tape, the catalogs don't have to be seperately archived, they are stored on the tape after each backup run (alternatively one read through the tape can reconstruct the catalog)
but you can install a fresh one if needed, and re-inventorize the tapes and start from there

2) I wonder if PBS append new data to an existing tape without risking the loss of previous data due to being overwritten? Or asked in a different way, does PBS manage and know what tapes have space and where and decides on the best location to write new data based on the size of data to be written?

this is best explained in the docs: https://pbs.proxmox.com/docs/tape-backup.html

but for short:
each media-pool has an allocation policy, and this determines if a new media-set should be allocated, and which tapes can be used. if a media-set is continued, we write the existing tapes in it until they are near full, finish the current archive (see above) write the current catalog and then go over to the next tape
if a new media-set is allocated, a "free" tape is selected (which are available depends on the retention policy of the media pool)

 

[pibber]

Thank you Dominik for the detailed and helpful response! I have another question regarding tape but it fits well with another thread, so I'll post there.