S3 backend lowercases Access Key ID during SigV4 signing → SignatureDoesNotMatch (MinIO/RADOS)

[dmaahr]

Reporting what appears to be a bug in the native S3 backend's SigV4 signing. Happy to provide additional logs or test a patch. Details below.

Version: proxmox-backup-server 4.2.5-1 (also reproduced on 4.2.4-1)

S3 provider: StorageFactory.io (MinIO/RADOS-based), via Curanet

Summary: PBS transmits the S3 Access Key ID in lowercase when generating the AWS Signature Version 4 Authorization header, even though the key is stored correctly (uppercase) in /etc/proxmox-backup/s3.cfg. Because SigV4 signing is case-sensitive on the Access Key ID, the server-side signature never matches and every request fails with SignatureDoesNotMatch.

Stored config is correct (cat /etc/proxmox-backup/s3.cfg, owner root:backup):

access-key AKIA[REDACTED]339BCA

But the server receives it lowercased (from the endpoint's own error response, s3 endpoint list-buckets):

<AWSAccessKeyId>akia[REDACTED]339bca</AWSAccessKeyId>

Ruled out experimentally:

• Config file stores uppercase, byte-verified
• Full daemon restart (proxmox-backup-proxy.service proxmox-backup.service) — no stale cache
• File ownership correct (root:backup)
• Region tested: us-east-1, dk6, default — no change
• path-style true, port 443 explicit
• Access key is plain alphanumeric (no shell-special chars); set via proxmox-backup-manager CLI, not env vars
• Same credentials work perfectly with MinIO Client (mc) against the identical endpoint — HEAD /obj4535/ returns 200 OK, proving bucket, region, path-style and credentials are all valid. Only PBS's signing path fails.

Conclusion: The lowercasing occurs inside PBS's SigV4 implementation, not in config, cache, or shell parsing. This makes the native S3 backend unusable against MinIO/RADOS-based S3 providers that (correctly) preserve Access Key ID case. Is there a known fix in the pipeline, or a config workaround to preserve Access Key ID case? Since the provider auto-generates keys with an AKIA prefix, requesting an all-lowercase key is not an option on my side.

 

[Chris]

Conclusion: The lowercasing occurs inside PBS's SigV4 implementation, not in config, cache, or shell parsing. This makes the native S3 backend unusable against MinIO/RADOS-based S3 providers that (correctly) preserve Access Key ID case. Is there a known fix in the pipeline, or a config workaround to preserve Access Key ID case? Since the provider auto-generates keys with an AKIA prefix, requesting an all-lowercase key is not an option on my side.

The PBS S3 client does NOT lowercase neither the access key, neither does it interfere with the secret key (implementation details at [0]). This is not the case for any of the mentioned PBS versions, the implementation of the signing already in place and tested with MinIO and Ceph. E.g. my local Ceph RADOS Gateway works perfectly fine with my all upper case access key.

Signature issues are most commonly encountered when the region is not configured correctly, can you verify with your provider?

[0] https://git.proxmox.com/?p=proxmox....4e26a8034637958515a36a5282a6bcf56;hb=HEAD#l23

 

[dmaahr]

The PBS S3 client does NOT lowercase neither the access key, neither does it interfere with the secret key (implementation details at [0]). This is not the case for any of the mentioned PBS versions, the implementation of the signing already in place and tested with MinIO and Ceph. E.g. my local Ceph RADOS Gateway works perfectly fine with my all upper case access key.

Signature issues are most commonly encountered when the region is not configured correctly, can you verify with your provider?

[0] https://git.proxmox.com/?p=proxmox....4e26a8034637958515a36a5282a6bcf56;hb=HEAD#l23

Root cause identified, with confirmation from the storage provider.

The provider (Curanet) uses Dell ECS / ObjectScale as the S3 backend. Their
systems architect confirmed the symptom matches AWS's default-checksum change:
newer AWS SDKs send CRC32 checksums by default (x-amz-checksum-crc32 /
x-amz-sdk-checksum-algorithm) on requests, which older/non-AWS S3
implementations (Dell ECS, Backblaze B2, Cloudflare R2) reject — manifesting
as SignatureDoesNotMatch because the checksum trailer is part of the signed
payload the server doesn't expect.

The standard client-side fixes are the env vars:
AWS_REQUEST_CHECKSUM_CALCULATION=when_required
AWS_RESPONSE_CHECKSUM_VALIDATION=when_required

These have no effect on PBS (tested via systemd Environment=), since PBS's S3
client is a custom Rust implementation, not the AWS SDK — so it doesn't honor
these variables.

This explains why the CanonicalRequest/StringToSign/secret are all
byte-identical to a working mc request, yet PBS fails: mc defaults to
"when_required" checksum behavior, PBS apparently always sends the checksum.

Request: could PBS's S3 backend add a "when_required" checksum mode, or a
provider-quirk to suppress the default request checksum? This would restore
compatibility with Dell ECS/ObjectScale, Backblaze B2, and R2. Same class of
issue tracked publicly for R2 (Cloudflare community) and B2.

References: the AWS SDK change is documented; ECS/B2/R2 all needed the
same when_required workaround.

 

[Chris]

This is again partially incorrect information! Backblaze B2 and Cloudflare R2 have been tested internally already during initial development and are already in use by users without the issues you describe.

I suspect the output you are posting here is AI generated? If so please do not do that, at least not without verifying what you are posting.

Request: could PBS's S3 backend add a "when_required" checksum mode, or a
provider-quirk to suppress the default request checksum?

It is true that requests contain payload integrity checksum headers, which are signed as part of the request as otherwise they would be useless, since they could be tampered with in transit. Also, not calculating these has possible severe side effects, since the S3 API server cannot verify that the contents are actually the ones expected/send by the client, which is not really acceptable for PBS. Therefore I'm rather reluctant in adding such a flag/quirk.

The provider (Curanet) uses Dell ECS / ObjectScale as the S3 backend. Their
systems architect confirmed the symptom matches AWS's default-checksum change:
newer AWS SDKs send CRC32 checksums by default (x-amz-checksum-crc32 /
x-amz-sdk-checksum-algorithm) on requests

Was this verified to be the root cause or is this just the assumed error cause? Because PBS S3 client does not use crc32 as checksum algorithm for integrity checks, but rather sha256 and we send this as header, not as payload trailer.