GarageHQ S3 (like MinIO) won't work

[fma965]

Hey, i can't seem to get GarageHQ to work with PBS S3.

I tried both VHost and Path style with no difference.

When i try to add the datastore it fails at listing the buckets


Other services, Discourse, DBBackup, Wordpress, Kopia etc are working fine with this S3 solution.

Is there some way to force specify a bucket, i'm assuming this is failing due to not being able to list buckets, but if i could get past this stage it would probably be fine.

EDIT: Garage logs are showing
Response: error 400 Bad Request, Bad request: signed header `content-length` is not present

EDIT2: PBS also reports
root@PBS:~# proxmox-backup-manager s3 endpoint list-buckets f9casa
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>Bad request: signed header `content-length` is not present</Message><Resource>/</Resource><Region>eu-west-2</Region></Error>
Error: failed to list buckets

Caused by:
unexpected status code 400 Bad Request

 

[Chris]

EDIT: Garage logs are showing
Response: error 400 Bad Request, Bad request: signed header `content-length` is not present

EDIT2: PBS also reports
root@PBS:~# proxmox-backup-manager s3 endpoint list-buckets f9casa
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>Bad request: signed header `content-length` is not present</Message><Resource>/</Resource><Region>eu-west-2</Region></Error>
Error: failed to list buckets

Thanks for the report, will have a closer look.

Is there some way to force specify a bucket, i'm assuming this is failing due to not being able to list buckets, but if i could get past this stage it would probably be fine.

You can create an s3 backed datastore also via the cli, not requiring to list the buckets, just run proxmox-backup-manager datastore create <your-store-name> <absolute-path-for-store-cache> --backend type=s3,client=<endpoint-id>,bucket=<your-bucket-name> make sure to set the provider quirks in case the If-None-Match header makes issues.

 

[fma965]

Thanks for the report, will have a closer look.


You can create an s3 backed datastore also via the cli, not requiring to list the buckets, just run proxmox-backup-manager datastore create <your-store-name> <absolute-path-for-store-cache> --backend type=s3,client=<endpoint-id>,bucket=<your-bucket-name> make sure to set the provider quirks in case the If-None-Match header makes issues.

Thanks, same thing when i use that command though unfortunately

 

[fma965]

any update on this?

 

[Chris]

Hi, unfortunately I did not have the time to tackle this issue due to other more pressing issues. A first glance indicates that the issue might be on how GarageHQ handles the content length header, which the PBS S3 client sends as part of the signed headers of the aws v4 sign scheme.

Please open an issue for this on htttps://bugzilla.proxmox.com and reference this thread, so this is not lost and you will get status notifications on updates. Thanks.

 

[karla.tomasz]

I'm also observing this. Minio started some strange moves with hiding most functionality in GUI for free users, so I started using garage, but this error is stopping me from using it for backups.

 

[Chris]

There was a recent bugfix [0] which most likely fixes this issue as well. The bugfix has been packaged with proxmox-backup-server 4.0.17-1, which should be available on the pbs-test repo soon, if no blocker is encountered during internal testing.

Once available, please test if that version solves the issue, thanks!

[0] https://git.proxmox.com/?p=proxmox.git;a=commit;h=3f242b0d973a77496b55d5a56499790fc05943a4