backup remote files directly on tape

[quantum_nt]

Hello,
I have a remote server on which I have a proxmox-backup-client. I want to know if is possible to make the backup directly on a proxmox-backup-server that has attached tape storage on it. I had configured the tape storage but from what I see I can only backup the local datastore on tape. The problem is that I do not have so much space on the local datastore to store backup there and after that to transfer on tape.
So is there a solution in this case?

 

[fabian]

no, PBS tape support is only for storing extra copies of backups already stored on PBS in a regular datastore.

 

[justin.anderson]

Is this still the case? We will need to backup over 200TB of data weekly to be taken off site. We have a LTO-8 Media changer and more than enough tapes. However, if we need to first backup the data to a datastore, then the project becomes a bit unwieldly and we may need to find another solution.

 

[fabian]

yes, this is still the case.

 

[dot.]

I'm digging out this old thread...

PBS tape support is only for storing extra copies of backups already stored on PBS in a regular datastore

Does a S3-backend combined with local cache qualify as "local datastore" (a local, on-prem S3 without any network bottlenecks, not AWS, backblaze or such) ? In other words, If PBS has already successfully backed up data from PVE to S3, can it then save these data from S3 to tape?

 

[fabian]

no, direct transfers from an S3-backed datastore and tapes are not supported yet.

 

[dot.]

Thanks for clearing the air, Fabian!

...are not supported yet.

...does that mean it's on the roadmap? If so, IMHO this could be a real game changer for many users by making huge local HDD storage unnecessary.

 

[fabian]

I don't think there is a final decision yet.

Currently there are technical blockers that require refactoring of some code for it to be on the table in the first place..

but there's a few considerations that apply here on top of that:
- tape works well if you can feed it data, which is at odds with S3's usual high(er) latency
- tape's main use case requires periodically starting over (to have independent media sets, e.g. for offsite storage/rotation), which means you need to re-transfer all needed chunks from S3 unless they are already cached (unlikely, because that would require a huge cache)

these two combined mean that for many practical deployments, combining S3 with tape will be both very slow/inefficient and very expensive (requiring lots of S3 requests and transfers).

 

[dot.]

That's very interesting!

While technically the fact that "tape works well if you can feed it data, which is at odds with S3's usual high(er) latency" could be addressed with local cache,
this statement here:

tape's main use case requires periodically starting over (to have independent media sets, e.g. for offsite storage/rotation), which means you need to re-transfer all needed chunks from S3 unless they are already cached

...is much more relevant. Indeed, this usecase may simply be way off the point for most scenarios and may imply enormous running&recovery costs for the users.

One usecase I could think of is if you already have a solid, distributed S3-compatible storage infrastructure on-prem (CEPH Object Gateway for instance). There are drawbacks to this design too (e.g. in a catastrophic disaster where your storage is unavailable), but if you have other backups as well (and you should, according to the 3-2-1 strategy), it is viable.

I'm curious what the final decision will be! Until then, stay tuned!

 

[fabian]

One usecase I could think of is if you already have a solid, distributed S3-compatible storage infrastructure on-prem (CEPH Object Gateway for instance).

that is probably the single case where it can make sense to do this, though in many cases you would also get away with pointing PBS at that distributed storage without S3 as layer in the middle (e.g., cephfs or a block device backed by that storage mounted locally). performance likely won't be great either though

 

[dot.]

Absolutely. I couldn't agree more!