S3 backend lowercases Access Key ID during SigV4 signing → SignatureDoesNotMatch (MinIO/RADOS)

[dmaahr]

Reporting what appears to be a bug in the native S3 backend's SigV4 signing. Happy to provide additional logs or test a patch. Details below.

Version: proxmox-backup-server 4.2.5-1 (also reproduced on 4.2.4-1)

S3 provider: StorageFactory.io (MinIO/RADOS-based), via Curanet

Summary: PBS transmits the S3 Access Key ID in lowercase when generating the AWS Signature Version 4 Authorization header, even though the key is stored correctly (uppercase) in /etc/proxmox-backup/s3.cfg. Because SigV4 signing is case-sensitive on the Access Key ID, the server-side signature never matches and every request fails with SignatureDoesNotMatch.

Stored config is correct (cat /etc/proxmox-backup/s3.cfg, owner root:backup):

access-key AKIA[REDACTED]339BCA

But the server receives it lowercased (from the endpoint's own error response, s3 endpoint list-buckets):

<AWSAccessKeyId>akia[REDACTED]339bca</AWSAccessKeyId>

Ruled out experimentally:

• Config file stores uppercase, byte-verified
• Full daemon restart (proxmox-backup-proxy.service proxmox-backup.service) — no stale cache
• File ownership correct (root:backup)
• Region tested: us-east-1, dk6, default — no change
• path-style true, port 443 explicit
• Access key is plain alphanumeric (no shell-special chars); set via proxmox-backup-manager CLI, not env vars
• Same credentials work perfectly with MinIO Client (mc) against the identical endpoint — HEAD /obj4535/ returns 200 OK, proving bucket, region, path-style and credentials are all valid. Only PBS's signing path fails.

Conclusion: The lowercasing occurs inside PBS's SigV4 implementation, not in config, cache, or shell parsing. This makes the native S3 backend unusable against MinIO/RADOS-based S3 providers that (correctly) preserve Access Key ID case. Is there a known fix in the pipeline, or a config workaround to preserve Access Key ID case? Since the provider auto-generates keys with an AKIA prefix, requesting an all-lowercase key is not an option on my side.

 

[Chris]

Conclusion: The lowercasing occurs inside PBS's SigV4 implementation, not in config, cache, or shell parsing. This makes the native S3 backend unusable against MinIO/RADOS-based S3 providers that (correctly) preserve Access Key ID case. Is there a known fix in the pipeline, or a config workaround to preserve Access Key ID case? Since the provider auto-generates keys with an AKIA prefix, requesting an all-lowercase key is not an option on my side.

The PBS S3 client does NOT lowercase neither the access key, neither does it interfere with the secret key (implementation details at [0]). This is not the case for any of the mentioned PBS versions, the implementation of the signing already in place and tested with MinIO and Ceph. E.g. my local Ceph RADOS Gateway works perfectly fine with my all upper case access key.

Signature issues are most commonly encountered when the region is not configured correctly, can you verify with your provider?

[0] https://git.proxmox.com/?p=proxmox....4e26a8034637958515a36a5282a6bcf56;hb=HEAD#l23