S3 Backup support

[tizbac]

Hi, i've already announced it on reddit, now many fixes have been done since then
I've developed with the help also of some contributors a proxy that will expose PBS api using S3 as backing storage

https://github.com/tizbac/pmoxs3backuproxy

Advantages:

- Much more reliable, no s3fs fuse etc , each request is proxied to an s3 request mostly
- Live restore supported
- Incremental backup
- Garbage collection
- With a WIP patch on PBS side it will be possible in the future to push-sync to S3
- Working file restore
- Speed is almost as native PBS unlike solutions using s3 with some mount helper
- Dynamic index ( proxmox backup client ) support , incremental too
TODO:

Dynamic index support ( file backup )

 

[Kingneutron]

What is ' mc ' here? I doubt it would work for me and others, Midnight Commander has been around for decades and I have it installed everywhere:

> mc alias set 'myminio' 'http://127.0.0.1:9000' 'minioadmin' 'minioadmin'

 

[tcabernoch]

Hmm. That's cool. Still pretty raw thought, huh? "extreme caution".

It's not clear to me from reading your github, is this thing supposed to be a backup target? So it runs backups like a PBS?

Or is it a Remote Sync Job target? You pair with it and send your backups there?

 

[danielt]

Hi tizbac,
Thanks for you tool.
can you maybe upload some step-by-step guide? I didn`t succeed

 

[tizbac]

Hmm. That's cool. Still pretty raw thought, huh? "extreme caution".

It's not clear to me from reading your github, is this thing supposed to be a backup target? So it runs backups like a PBS?

Or is it a Remote Sync Job target? You pair with it and send your backups there?

It is a proxy that behaves like a PBS so you add it as proxmox backup server on PVE, but it uses S3 as backing storage
Much faster and reliable than using fuse s3 or stuff like that with regular PBS
The warnings are just because it is still not widely tested altough functioning , with time if no bug reports of any kind of corruption arise they will be gone
(Maybe i'm a bit out of schemes being that cautios given last Crowdstrike fiascos etc ahah )

 

[tizbac]

Hi tizbac,
Thanks for you tool.
can you maybe upload some step-by-step guide? I didn`t succeed

Main blocking points when using it first times may be
1. You need to start the proxy with required parameters as per docs
2. When you add it on PVE , use s3keyid@pbs as username , key secret as password, and datastore name as bucket name
3. Use it just like PBS

 

[danielt]

Main blocking points when using it first times may be
1. You need to start the proxy with required parameters as per docs
2. When you add it on PVE , use s3keyid@pbs as username , key secret as password, and datastore name as bucket name
3. Use it just like PBS

i didn`t understood at all your guide. i installed the docker I don't know where and which parameters I need to add my AWS keys and the S3 bucket.
I will be happy to test and use your tool

 

[tenyosama]

Hi,

much appreciated, great work!
I've tested 0.0.3-alpha with idrive e2 and worked very well (vm, ct and proxmox-backup-client host backup).
Very good performance both on backup and restore.