proxmox-tape : will setting --latest-only on a job destroy earlier snapshots? What about retention?

[wbk]

Hi all,

I'm on PBS 2.4.x. There is an external tape drive connected to the system, but it is not running 24/7 as I need change the tapes manually anyway. The result is that tape backups occur less frequently than planned in the job.

So there is a bit of a backlog of un-taped backup snapshots from CTs and VMs.

Before upgrading to PBS 3.0, I'd like to run a tape backup. While checking the job options (the external tape drive was upgraded to a newer generation, LTO3 to LTO4), I noticed some options I am not sure what their impact is.

Last snapshot only

There is a checkbox "Latest only". The help page says:

"The backup includes all snapshots from a backup group by default. You canset the latest-only flag to include only the latest snapshots:

# proxmox-tape backup-job update job2 --latest-only

"

Does this erase data from the previous snapshot before writing the new snapshot to tape? Or will it first deduplicate over the delta between the earlier and the latest snapshot, and add the delta from the latest snapshot only?

In practical terms: there are two or three snapshots between the last tape backup and the currently latest snapshot. The snapshots grow over time, they are between 1.5 TB and 2 TB in size. Mostly stale data. Deduplicated I expect each new snapshot to be 100-300 GB in size when taping each snapshot, and the delta from last taped snapshot to the current last backup snapshot 500-800 GB.

Retention

I had set the retention to 6 months, but the last tape job ran over half a year ago. Will it wipe the stale backup before writing a new to tape (either the last or all snapshots within <6 months, depending on `--latest-only`)? Or will it deduplicate in PBS to keep the current bits and add the new bits in the delta?

General

I know tape is a linear medium (even though LTO5 and up present it otherwise; that is not relevant for my LTO4 drive and LTO3/4 tapes). So writing a delta will only contain additions, no deletions from previous snapshots.
Deduplication is mentioned a few times in the help, it is not clear to me whether this is over concurrent snapshots of multiple CTs/VMs in the same pool, or across time.

Thanks for reading my question, I am looking forward to replies!

 

[dcsapak]

Hi

Last snapshot only

There is a checkbox "Latest only". The help page says:

"The backup includes all snapshots from a backup group by default. You canset the latest-only flag to include only the latest snapshots:

# proxmox-tape backup-job update job2 --latest-only

"

Does this erase data from the previous snapshot before writing the new snapshot to tape? Or will it first deduplicate over the delta between the earlier and the latest snapshot, and add the delta from the latest snapshot only?

In practical terms: there are two or three snapshots between the last tape backup and the currently latest snapshot. The snapshots grow over time, they are between 1.5 TB and 2 TB in size. Mostly stale data. Deduplicated I expect each new snapshot to be 100-300 GB in size when taping each snapshot, and the delta from last taped snapshot to the current last backup snapshot 500-800 GB.

'latest only' just controls which snapshots will be backed up to tape. in general if you continue to backup to a media-set (depends on the allocation/retention policy) no old snapshot on that media-set will be removed and the new necessary chunks + snapshots will be backed up
we have a catalog which chunks are already on the media-set so we can just backup the missing ones (so the deduplication of the datastores transfer to a media-set)

Retention

I had set the retention to 6 months, but the last tape job ran over half a year ago. Will it wipe the stale backup before writing a new to tape (either the last or all snapshots within <6 months, depending on `--latest-only`)? Or will it deduplicate in PBS to keep the current bits and add the new bits in the delta?

it depends more on your 'allocation' policy when a new media-set is created, the retention policy should only come into effect for "finished' media-sets (so when there is a newer one)

General

I know tape is a linear medium (even though LTO5 and up present it otherwise; that is not relevant for my LTO4 drive and LTO3/4 tapes). So writing a delta will only contain additions, no deletions from previous snapshots.
Deduplication is mentioned a few times in the help, it is not clear to me whether this is over concurrent snapshots of multiple CTs/VMs in the same pool, or across time.

i don't completely get what you mean, but the deduplication works by chopping up the content of the backups and use a hash of the content as identifier. these chunks will then be deduplicated across all backups in a datastore
and all snapshots in a media-set for tape backups

 

[wbk]

Hi dcsapak,

Thank you for your quick reply!

I think it asnwered all my questions. Would you mind reading my rehash of your explanation, to see whether I understood correctly?

Latest only

'latest only' just controls which snapshots will be backed up to tape.

Ok, so as long as I play according to the rules and if not playing with retention or manual deletion of data on tape: existing data will stay htere, no matter the options for new backups.

If I got a chronological snapshot series of A, B, C, D of a container, and A is already on this tape:
regularly, without `--latest-only`, the following backup will append B, C, D to this tape;
alternatively, with `--latest-only`, the following backup will append will skip B, C, and just D will be appended

In either case, chunks that have already been written to tape as part of A and have not changed, will not be contained in (B, C and) D. This is assuming I continue the same media set, else this backup would be a new set (with tis own isolated deduplication).

Retention

the retention policy should only come into effect for "finished' media-sets (so when there is a newer one)

I could not grok this when reading it on the help page, but you gave just a little nudge in the right direction I think. My 'current' media set will never expire; it is only after creating a new media set for a particular backup (-group), that the counter for retention of the previous media set kicks in.

If I got many tapes, ample space to store them and no adverse psychological effects of unused stuff laying around , there is no reason to let them expire (hoarding FTW!).

If there is any reason to not let the pile of tapes with stale backups grow indefinitely, setting a retention period allows PBS to help me reuse tapes in an orderly fashion. Because of the ID-chip in the tape it will recognize if I insert a tape that is part of a not yet expired media set, thus preventing me from containing it in a new media set (and inadvertently overwrite it).

General

(...) chunks will (..) be deduplicated across all backups in a datastore and all snapshots in a media-set for tape backups

You already answered my not well formed question earlier. The thing I wondered about was whether a new backup would expel stale chunks that have become unused in the current snapshot. This is not the case: what is on tape, will stay on tape. I can always revert to any one of the earlier backup points.

I think I learned a lot, thanks once again!

 

[dcsapak]

yeah it's 99% correct, one small detail is not completely correct though:

Because of the ID-chip in the tape it will recognize if I insert a tape that is part of a not yet expired media set, thus preventing me from containing it in a new media set (and inadvertently overwrite it).

it's not actually the chip, but we write a small media-set and tape header at the beginning of the tape that is read. this tells us which tape it is, and which media-set it belongs to

in an autoloader/changer there is the additional information with the barcode (which must match with the label written to the beginning)