Ladder schedules for tape backups

[giovvv]

Before PBS, I managed my tape backups manually using a ladder schedule, for example:

• Do a daily backup
• Keep the last 7 days. Every 7 days, move the backup to the "weekly" category
• Keep the last 4 weeklies. Every 4 weeks, move the backup to the "monthly" category
• Keep the monthlies indefinitely.

If I understand the docs correctly, the Media Set Allocation Policy of a pool determines how media sets leave the pool: never ("continue"), after each backup ("always"), or after a calendar event.

I think that to follow a ladder scheme like the one above, one should set the allocation policy to "continue" and then force a media set out of a pool manually when the scheme requires so. However, I haven't found a way to do it (i.e. how to manually remove a certain media set from a media pool). Is it possible?

Thanks

 

[dcsapak]

i think you cannot do it *exactly* in this way, but how long do you want to keep which backups/which tapes can fail?

the backups are deduplicated on the media sets, the allocation policy determines when a new media-set is created (and the last one 'finished') and the retention policy determines how long to keep media-sets

one possibility would be e.g. having a pool with retention policy 7 days, allocation policiy every day, with a daily backup (this will give you 7 media-sets / week) and then export a media-set once a week (phyiscally remove from the changer for example)
those tapes/media-sets will not be requested (as long as there are tapes to write) and insert the oldest one, and once month, take such an offline media-set to your permanent location and

i hope i understand your desired tape scheme correcltly..

 

[giovvv]

@dcsapak I don't need to follow that scheme exactly; what I would like to achieve, somehow, is simply to have frequent backups for recent times, and then more spaced backups for less recent times. For example, daily backups for the last 7 days, then weekly backups for the last 4 weeks, then monthly backups for the last 6 months. The reason is to retain some long-term fallback in case some data is found to be missing or damaged after a certain time.

I suppose another way to obtain this could be:

• have three separate media pools (for daily, weekly, monthly)
• have appropriate retention policies: 1 week for daily, 1 month for weekly, 6 months for monthly)
• schedule backups to avoid overlaps (for example, not more than one backup per day, using a sequence such as: mddddddwddddddwdddddddwddddddd)

Could this work?

Also, I have two more questions:

1. Since tapes cannot be overwritten indefinitely, how can one retire a tape (and/or a media-set)?
2. Suppose an asteroid falls destroying both the data server and the backup server, but a media-set has been stored offsite. After replacing the hardware and reinstalling the OSes, is it possible to restore from that media-set, or it is necessary also to have other critical information, for example some configuration files from the backup server?

Thanks a lot!

 

[dcsapak]

I suppose another way to obtain this could be:

• have three separate media pools (for daily, weekly, monthly)
• have appropriate retention policies: 1 week for daily, 1 month for weekly, 6 months for monthly)
• schedule backups to avoid overlaps (for example, not more than one backup per day, using a sequence such as: mddddddwddddddwdddddddwddddddd)

Could this work?

seems workable as long as the tape backup jobs do not interfere with each other

1. Since tapes cannot be overwritten indefinitely, how can one retire a tape (and/or a media-set)?

in the gui under tape -> inventory, you can set a status for a tape, e.g. damaged or retired

Suppose an asteroid falls destroying both the data server and the backup server, but a media-set has been stored offsite. After replacing the hardware and reinstalling the OSes, is it possible to restore from that media-set, or it is necessary also to have other critical information, for example some configuration files from the backup server?

just insert the tapes into the drive/change and do a 'catalog' of each tape (either via gui or 'proxmox-tape catalog')
we save a catalog of what is contained on a tape at the end of it
as soon as you have all tapes cataloged, you can begin the restore from them (you need all or at least the first X tapes of a media-set, since the chunks are deduplicated, and such only a complete media-set is a valid point to restore from)

 

[giovvv]

@dcsapak: I understand that the retention policy specifies when a media-set is retired. Currently, it appears it is only possible to specify a time interval (e.g. retire after two weeks). This is appropriate for scheduled backups, but it seems to me that it does not work well for manual backups, because in that case it should be possible to manually retire a media-set: either because one wants to keep only a certain number of backups, or because of some other cherry-picking reason.

I think there should be a way to manually retire a media-set. Unless I've missed something, it appears there is no way to do so. Or the way to do it is to manually retire the single tapes that belong to that media-set?

Thanks
Giovanni

 

[dcsapak]

note that there is a difference between 'expired' and 'retired'. you can retire tapes but this will lead to them not being picked up as targets anymore

in case of manual backup i'd also manually manage the media sets, you can force a new media set on backup, and set the retention period (and retire status of tapes) before you make a backup

 

[giovvv]

Thus to manually delete an old media-set and reuse its tapes, the correct way is just to select the tapes that belong to it and click FORMAT?

(The reason is: since I backup manually without a fixed schedule, I am forced to set the retention policy to forever, but I don't really want to keep them until the end of the universe, I just want to decide myself when a certain media-set can be finally recycled, for instance after a certain number of media-sets have been produced).

[note: deleted a former draft of this message, posted by mistake]

 

[dcsapak]

the correct way is just to select the tapes that belong to it and click FORMAT?

thats one possibility

another would be to set the rentention of the pool so that the tapes you want get reused (and if you have older ones, mark them retired before)

as i said, the tape backup is really intended to be used in an automated fashion

also, since the backups on the tapes are deduplicated, you can have much more backups on tapes so you don't have to remove old backups that often