VZDump and S3 don't mix well

[Ataraxxia]

Hey,

Straight to the point, the "move" operation at the end of VZDump kills it if the destination is S3 backend (for which move is actually copy&delete). Currently I'm using rclone to mount S3 bucket.

Would it be possible to add a switch that would write backup directly to the destination file, avoiding the move operation?

 

[LnxBil]

There is a (still open) feature request about S3 integration in PBS.

 

[Ataraxxia]

I'm fully aware of that and the fact that at this point it's almost 3 years old.

The suggested fix is fairly simple and wouldn't break any compatibility. While small backup jobs can get away with this, in practice backing up anything larger than 600GiB is impossible.

I also DON'T want to back up to local storage and then push files to S3, this method duplicates times and writes unnecessarily.

 

[leesteken]

I'm fully aware of that and the fact that at this point it's almost 3 years old.

The Proxmox team tracks bugs and changes via bugzilla, therefore it's probably better to add your comments there. Feel free to discuss it here as much as you want but bugzilla is where the actual process is made.

The suggested fix is fairly simple and wouldn't break any compatibility. While small backup jobs can get away with this, in practice backing up anything larger than 600GiB is impossible.

If you can provide that (beginning of a) fix on bugzilla, it's more likely to be picked up.

 

[LnxBil]

I also DON'T want to back up to local storage and then push files to S3, this method duplicates times and writes unnecessarily.

I'm unfamiliar with S3 and don't see it advantages compared to juse a simple file server, but is there not any kind of S3 filesystem support in Linux, e.g. with FUSE so that you can just mount the S3 stuff and directly write to it? Sounds like the way to go to me. It just adds the storage on Linux / OS level and PVE just uses it. No idea if this works performance-wise, yet it technically would work.