S3 Upload Fails with Large VMs (Cache Issue?)

Please post the full verify task log (as attachment) and try the steps (adapted to your chunk) as suggested in the link above.
 
additional question for my understanding of the S3 process.
Does the verification of the backups lead to lots of download traffic from the S3 storage? If that's the case it would be questionable business-wise for my case.
 
Str1atum said:
here's the error log
Click to expand...
Seems you run into a similar issue as https://bugzilla.proxmox.com/show_bug.cgi?id=6858, added this as additional information there.

Str1atum said:
Does the verification of the backups lead to lots of download traffic from the S3 storage? If that's the case it would be questionable business-wise for my case.
Click to expand...
Yes, currently the client fetches the objects for verification, a more lightweight verification process is planned but not implemented yet, see https://bugzilla.proxmox.com/show_bug.cgi?id=4594
 
  • Like
Reactions: Johannes S
Str1atum said:
Does the verification of the backups lead to lots of download traffic from the S3 storage? If that's the case it would be questionable business-wise for my case.
Click to expand...
Depending on your storage provider cost plans and available vserver offerings an option could be to spin up a virtual PBS instance in their cloud and use it for verifying the backups from time to time. E.G: AWS or Hetzner offer S3 and virtual servers, you need to pay for data transfer out of their datacenter. But inside the same datacenter the transfer are free so you could use a vserver for doing the verification from time to time and remove it afterwards.

I got this idea from an older discussion from restics (Linux generic backup tool who also does deduplication like PBS) forum on verifying backups on a cloud storage:
cdhowie Apr 2019
Just be careful when using this in a cloud context as you have to pay the necessary transfer fees to basically download your entire repository. (If you are using S3, it may be cheaper to spin up a short-lived EC2 in the same region and perform the check there. An m5.large is $0.096 per hour, which is just a hair more than the cost to transfer 1GB of data out to the Internet. You could probably check TBs of data on an m5.large in an hour.)

https://forum.restic.net/t/read-data-read-data-subset-use-cases/1542/5
Click to expand...
 
At the moment I already have a virtual PBS on Hetzner Cloud with an attached storage box (5 TB) and do a simple sync to my local PBS server.
Wanted to try out if S3 has any advantages but this far not really.
Will the egress be significantly lower with the lightweight verification process?
Or replacing Hetzner Storage Box with Hetzner S3?
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom