S3 Backup support

T

tizbac

New Member
Dec 4, 2023
8
20
3
Hi, i've already announced it on reddit, now many fixes have been done since then
I've developed with the help also of some contributors a proxy that will expose PBS api using S3 as backing storage

https://github.com/tizbac/pmoxs3backuproxy

Advantages:

- Much more reliable, no s3fs fuse etc , each request is proxied to an s3 request mostly
- Live restore supported
- Incremental backup
- Garbage collection
- With a WIP patch on PBS side it will be possible in the future to push-sync to S3
- Working file restore
- Speed is almost as native PBS unlike solutions using s3 with some mount helper
- Dynamic index ( proxmox backup client ) support , incremental too
TODO:

Dynamic index support ( file backup )
 
Last edited:
  • Like
Reactions: galaober, ucholak, tcabernoch and 1 other person
Hmm. That's cool. Still pretty raw thought, huh? "extreme caution".

It's not clear to me from reading your github, is this thing supposed to be a backup target? So it runs backups like a PBS?

Or is it a Remote Sync Job target? You pair with it and send your backups there?
 
tcabernoch said:
Hmm. That's cool. Still pretty raw thought, huh? "extreme caution".

It's not clear to me from reading your github, is this thing supposed to be a backup target? So it runs backups like a PBS?

Or is it a Remote Sync Job target? You pair with it and send your backups there?
Click to expand...
It is a proxy that behaves like a PBS so you add it as proxmox backup server on PVE, but it uses S3 as backing storage
Much faster and reliable than using fuse s3 or stuff like that with regular PBS
The warnings are just because it is still not widely tested altough functioning , with time if no bug reports of any kind of corruption arise they will be gone :)
(Maybe i'm a bit out of schemes being that cautios given last Crowdstrike fiascos etc ahah )
 
Last edited:
  • Like
Reactions: tcabernoch
danielt said:
Hi tizbac,
Thanks for you tool.
can you maybe upload some step-by-step guide? I didn`t succeed
Click to expand...
Main blocking points when using it first times may be
1. You need to start the proxy with required parameters as per docs
2. When you add it on PVE , use s3keyid@pbs as username , key secret as password, and datastore name as bucket name
3. Use it just like PBS
 
tizbac said:
Main blocking points when using it first times may be
1. You need to start the proxy with required parameters as per docs
2. When you add it on PVE , use s3keyid@pbs as username , key secret as password, and datastore name as bucket name
3. Use it just like PBS
Click to expand...
i didn`t understood at all your guide. i installed the docker I don't know where and which parameters I need to add my AWS keys and the S3 bucket.
I will be happy to test and use your tool
 
Hi,

much appreciated, great work!
I've tested 0.0.3-alpha with idrive e2 and worked very well (vm, ct and proxmox-backup-client host backup).
Very good performance both on backup and restore.
 
  • Like
Reactions: tizbac
I would love to give this a try, but I am a little lost on how to set up the minio side. Would someone be able to point to a guide for the minio setup? I've been skimming https://min.io/docs/minio/linux/operations/installation.html but it seems it's expecting dedicated disk(s) locally? I'm unclear how to configure minio to act as an S3 proxy.

My use case would be to use pmoxs3backuproxy as a PBS replication target which is backed by a Backblaze B2 bucket (using their S3 compatible API of course).
 
You must log in or register to reply here.
Top Bottom
Back