external datastore

[michabbs]

Has anyone tried to set up datastore located somewhere at online storage provider? Like this or this or this.
Will it work? What protocol would work best - smb, sftp? (And what about s3fs or s3ql?)

I suppose it should work, but I am afraid datastore verification would need to read the whole cloud disk (right?), so it might be very slow and bandwidth consuming.
On the other hand - this would easily provide simple off site backup location, which generally is a good idea.

 

[Dunuin]

You shouldn't use HDDs or local SMB/NFS because of the additional latency as PBS needs IOPS performance. Now add several ms of internet latency and performance should be terrible. Lets say you got 1TB of backups so 500,000x to 1,000,000x 1-2MB chunk files. A GC needs to read and write all of those. So this would add 1-2 million times the additional internet latency. Similar problem with verify jobs.

You usually want a remote PBS with local storage. Then the internet latency won't matter when doing GC or verify jobs as these can be done locally. And backup jobs aren't that bad because of the incremental backups. And it would be even better to backup/restore from a onsite PBS and then use a sync job so the offsite PBS pulls backup snapshots from the onsite PBS where bandwidth/performance doesn't matter that much.

 

[michabbs]

I was afraid of this... :-(

Any chance to periodically push local datastore from onsite to a "non-pbs" cloud location? (Rsync? Rclone?)
The problem is that any offsite "pbs with local hdd" is order of magnitude more expensive that "just cloud storage". Moreover - it's really hard to find vps with large hdd (counted in TB's), and dedicated server is another order of magnitude more expensive. We are talking about price like ~$5 per cloud storage TB compared to $100 per private server TB. It's a huge difference, especially considering offsite backup is something that most likely will never be used. So I would expect to reduce cost of it. (On the other hand - offsite backup is something I want to have in case of some terrible local disaster....)

 

[Dunuin]

Any chance to periodically push local datastore from onsite to a "non-pbs" cloud location? (Rsync? Rclone?)

Some people here do that. The biggest problem would be that you have to download the WHOLE datastore before you could start restoring your guests.
So extreme downtime in case of a disaster with the loss of your local PBS.

The problem is that any offsite "pbs with local hdd" is order of magnitude more expensive that "just cloud storage". Moreover - it's really hard to find vps with large hdd (counted in TB's), and dedicated server is another order of magnitude more expensive.

And something like a Debian VPS at Hetzner where you install the PBS packages + some cheap storage (storagebox) in the same datacenter to keep the latency low?

 

[michabbs]

Some people here do that. The biggest problem would be that you have to download the WHOLE datastore before you could start restoring your guests.

Maybe I could then mount remote datastore r/o and thus restore faster? This might work if I don't do GC nor veryfication.

So extreme downtime in case of a disaster with the loss of your local PBS.

Well... Yes. But I can see only two possibilities when offsite copy would be the last surviving one: fire or war. In both cases I will most likely have plenty of other things to do than to restore whatever immedialety.

 

[rdwild]

I would like to announce that A-Z Computer Solutions with a grant from the Sia Foundation will be building a cloud backup module for the PBS Server with plugin architecture. First plugin will support S3 backups. You can see updates here; https://forum.sia.tech/t/proxmox-backup-server-cloud-extension/397/14

 

[michabbs]

I would like to announce that A-Z Computer Solutions with a grant from the Sia Foundation will be building a cloud backup module for the PBS Server with plugin architecture. First plugin will support S3 backups. You can see updates here; https://forum.sia.tech/t/proxmox-backup-server-cloud-extension/397/14

But... it's already on the PBS roadmap. So why independent project? :-O

 

[itNGO]

I would like to announce that A-Z Computer Solutions with a grant from the Sia Foundation will be building a cloud backup module for the PBS Server with plugin architecture. First plugin will support S3 backups. You can see updates here; https://forum.sia.tech/t/proxmox-backup-server-cloud-extension/397/14

And why don't you just contribute your upcoming solution/code to proxmox repository direct?

 

[rdwild]

And why don't you just contribute your upcoming solution/code to proxmox repository direct?

The main module and first plugin will be open source and we have plans to contribute the code to the project but there is "no guarantee that it will be accepted".

 

[rdwild]

But... it's already on the PBS roadmap. So why independent project? :-O

Because it's been on the road map for a while along with a number of other things and our hope is to just speed things along on some things that I would like to see sooner than later.

 

[fabian]

it would definitely help your chances of acceptance if you discuss the architecture/design of your plans with us first (preferably on the pbs-devel mailing list). please also read https://pve.proxmox.com/wiki/Developer_Documentation , as most of it also applies to PBS.

 

[rdwild]

it would definitely help your chances of acceptance if you discuss the architecture/design of your plans with us first (preferably on the pbs-devel mailing list). please also read https://pve.proxmox.com/wiki/Developer_Documentation , as most of it also applies to PBS.

We are already on the list and we have posted there along with where we are posting updates. we will post more as we go along. The supporting docs are large so I'm not sure if posting them to the list is a good idea ... unless it is.

 

[fabian]

I know - what I meant is, before starting an implementation, please post and get feedback on design specs, else you risk spending a lot of time implementing something that has no chance of being merged. sending a patch first without much discussion works for small bug fixes, because there the loss of having to redo it again in a different manner is not as big, but for bigger features it definitely is with it to first discuss the concepts/approach before spending the hours churning out code

 

[rdwild]

I know - what I meant is, before starting an implementation, please post and get feedback on design specs, else you risk spending a lot of time implementing something that has no chance of being merged. sending a patch first without much discussion works for small bug fixes, because there the loss of having to redo it again in a different manner is not as big, but for bigger features it definitely is with it to first discuss the concepts/approach before spending the hours churning out code

reposted as per your suggestion.

pbs-devel Digest, Vol 42, Issue 5

I would like to note when I posted it the first time around someone from proxmox did go in and read it as they downloaded the extended documentation.

 

[michabbs]

And something like a Debian VPS at Hetzner where you install the PBS packages + some cheap storage (storagebox) in the same datacenter to keep the latency low?

I did it and here are my notices:

• PBS in Debian VPS at Hetzner + datastore via SMB at Hetzner Storagebox
• It works!
• Backup/upload works well and there are no problems at all. In my case the limiting factor is my upload bandwidth (60mbps), so I really cannot expect anything better.
• Restore/download works. It's quite slow, but usable. The process was unfortunately not able to fully fill up my download bandwidth (600mbps). Restoring "small" vm's is not a problem. Restoring bigger ones of course takes time. (On the other hand - this is expected with remote backup.)
• I tested live restore and it works! (Of course very slow and I think there is no point in really doing it...)
• Very slow verification.
• Extremely slow garbage collection.
• Very low cost.

Conclusions:

• Obviously this cannot be main backup solution.
• As a secondary (or better - tertiary) backup solution it might do the job. It's rather kind of "cold" backup, but still usable in case of total disaster at a main site.
• Slow GC is a problem, but... anyway it's done in background. Just set up and forget.
• Considering really low cost - it's something really interesting. It might (or not?) still be faster than extracting copy from AWS Glacier Storage.