[SOLVED] All backups failed with "unable to update manifest blob"

[SerhiiT]

Hello.

PBS 4.0.11
proxmox-backup-client on the same host

S3 endpoint using Hitachi Content Platform, LAN connection.

Any backup (from 2GB to 121GB, doesn't matter) fails while trying to upload manifest blob:

2025-10-19T22:08:04+03:00: Upload size: 926010 (100%)
2025-10-19T22:08:04+03:00: Duplicates: 0+1 (100%)
2025-10-19T22:08:04+03:00: Compression: 41%
2025-10-19T22:08:04+03:00: successfully closed dynamic index 1
2025-10-19T22:08:04+03:00: POST /blob
2025-10-19T22:08:04+03:00: Uploaded blob to object store: .cnt/host/pbs/2025-10-19T19:07:54Z/index.json.blob
2025-10-19T22:08:04+03:00: add blob "/mnt/hcpstore/host/pbs/2025-10-19T19:07:54Z/index.json.blob" (312 bytes, comp: 312)
2025-10-19T22:08:04+03:00: POST /finish
2025-10-19T22:08:04+03:00: POST /finish: 400 Bad Request: unable to update manifest blob - failed to update manifest on s3 backend
2025-10-19T22:08:04+03:00: backup failed: connection error
2025-10-19T22:08:04+03:00: removing failed backup
2025-10-19T22:08:05+03:00: removing backup snapshot "/mnt/hcpstore/host/pbs/2025-10-19T19:07:54Z"
2025-10-19T22:08:05+03:00: TASK ERROR: connection error: not connected

Chunks seems to be successfully stored on S3 storage, but no signs if any index.json.blob files.

Any advise or suggestion how to find root cause of the issue would be appreciated.

UPDATE:
Versioning should be enabled for bucket (namespace) on HCP side.

 

[Chris]

Hello.

PBS 4.0.11
proxmox-backup-client on the same host

S3 endpoint using Hitachi Content Platform, LAN connection.

Any backup (from 2GB to 121GB, doesn't matter) fails while trying to upload manifest blob:

2025-10-19T22:08:04+03:00: Upload size: 926010 (100%)
2025-10-19T22:08:04+03:00: Duplicates: 0+1 (100%)
2025-10-19T22:08:04+03:00: Compression: 41%
2025-10-19T22:08:04+03:00: successfully closed dynamic index 1
2025-10-19T22:08:04+03:00: POST /blob
2025-10-19T22:08:04+03:00: Uploaded blob to object store: .cnt/host/pbs/2025-10-19T19:07:54Z/index.json.blob
2025-10-19T22:08:04+03:00: add blob "/mnt/hcpstore/host/pbs/2025-10-19T19:07:54Z/index.json.blob" (312 bytes, comp: 312)
2025-10-19T22:08:04+03:00: POST /finish
2025-10-19T22:08:04+03:00: POST /finish: 400 Bad Request: unable to update manifest blob - failed to update manifest on s3 backend
2025-10-19T22:08:04+03:00: backup failed: connection error
2025-10-19T22:08:04+03:00: removing failed backup
2025-10-19T22:08:05+03:00: removing backup snapshot "/mnt/hcpstore/host/pbs/2025-10-19T19:07:54Z"
2025-10-19T22:08:05+03:00: TASK ERROR: connection error: not connected

Chunks seems to be successfully stored on S3 storage, but no signs if any index.json.blob files.

Any advise or suggestion how to find root cause of the issue would be appreciated.

Hi,
did you ever perform a valid backup with PBS to the S3 backend provided by Hitachi Connect? Could you please verify uploading an object with the same object key (.cnt/host/pbs/2025-10-19T19:07:54Z/index.json.blob) works by some other tooling? Might be that the object key uri encoding makes issues here, similar to https://forum.proxmox.com/threads/pbs-corrupted-chunk-s3-issue.172690/post-803961

 

[SerhiiT]

Hi,
did you ever perform a valid backup with PBS to the S3 backend provided by Hitachi Connect?

No, this is a first attempt to store backup on HCP ( Hitachi Content Platform ).

Could you please verify uploading an object with the same object key (.cnt/host/pbs/2025-10-19T19:07:54Z/index.json.blob) works by some other tooling?

curl uploads index.json.blob to /.cnt/host/pbs/2025-10-19T19:07:54Z/ without any issues, for example:

...skip...

* Connected to xxxxxxxxxxxxx (111.222.333.44) port 443
* using HTTP/1.x
> PUT //hcp3data/.cnt/host/pbs/2025-10-19T19:13:19Z/index.json.blob HTTP/1.1
> Host: xxxxxxxxxxxxxxxxx
> User-Agent: curl/8.14.1
> Accept: */*
> Content-Type: application/octet-stream
> Date: Tue, 21 Oct 2025 16:02:31 +0300
> Authorization: AWS someauth:signature
> Content-Length: 53
>
* upload completely sent off: 53 bytes
< HTTP/1.1 200 OK
< Date: Tue, 21 Oct 2025 13:02:18 GMT
< Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
< Cache-Control: no-cache,no-store,must-revalidate
< X-Download-Options: noopen
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< X-Frame-Options: SAMEORIGIN
< Pragma: no-cache
< Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
< X-XSS-Protection: 1; mode=block
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-DNS-Prefetch-Control: off
< X-Content-Type-Options: nosniff
< ETag: "5a0e1635f438a54b2b0c823e5a6e860d"
< Content-Length: 0
<
* Connection #0 to host xxxxxxxxxxxxxxxxxx left intact

Might be that the object key uri encoding makes issues here, similar to https://forum.proxmox.com/threads/pbs-corrupted-chunk-s3-issue.172690/post-803961

Oops.

 

[Chris]

Do you get an error in the systemd journal on the PBS when the error occurs?

 

[SerhiiT]

Hi.

There are no errors in systemd for proxmox-backup at the time of backup:

Oct 19 22:04:49 pbs systemd[1]: Stopping proxmox-backup.service - Proxmox Backup API Server...
Oct 19 22:04:49 pbs systemd[1]: proxmox-backup.service: Deactivated successfully.
Oct 19 22:04:49 pbs systemd[1]: Stopped proxmox-backup.service - Proxmox Backup API Server.
Oct 19 22:04:49 pbs systemd[1]: proxmox-backup.service: Consumed 1.921s CPU time, 25.4M memory peak.
Oct 19 22:04:49 pbs systemd[1]: Starting proxmox-backup.service - Proxmox Backup API Server...
Oct 19 22:04:50 pbs proxmox-backup-api[2433557]: catching shutdown signal
Oct 19 22:04:50 pbs proxmox-backup-api[2433557]: catching reload signal
Oct 19 22:04:50 pbs systemd[1]: Started proxmox-backup.service - Proxmox Backup API Server.
Oct 20 00:00:51 pbs proxmox-backup-api[2433557]: notified via target `mail-to-root`

And task error for proxmox-backup-proxy:

Oct 19 22:04:55 pbs systemd[1]: Stopping proxmox-backup-proxy.service - Proxmox Backup API Proxy Server...
Oct 19 22:04:55 pbs systemd[1]: proxmox-backup-proxy.service: Deactivated successfully.
Oct 19 22:04:55 pbs systemd[1]: Stopped proxmox-backup-proxy.service - Proxmox Backup API Proxy Server.
Oct 19 22:04:55 pbs systemd[1]: proxmox-backup-proxy.service: Consumed 1h 8min 18.086s CPU time, 121.9G memory peak.
Oct 19 22:04:55 pbs systemd[1]: Starting proxmox-backup-proxy.service - Proxmox Backup API Proxy Server...
Oct 19 22:04:55 pbs proxmox-backup-proxy[2433612]: catching shutdown signal
Oct 19 22:04:55 pbs proxmox-backup-proxy[2433612]: catching reload signal
Oct 19 22:04:55 pbs systemd[1]: Started proxmox-backup-proxy.service - Proxmox Backup API Proxy Server.
Oct 19 22:04:55 pbs proxmox-backup-proxy[2433612]: applied rrd journal (2946 entries in 0.008 seconds)
Oct 19 22:04:55 pbs proxmox-backup-proxy[2433612]: rrd journal successfully committed (25 files in 0.007 seconds)
Oct 19 22:05:00 pbs proxmox-backup-proxy[2433612]: Using datastore cache with capacity 14166382 for store hcp3data
Oct 19 22:08:05 pbs proxmox-backup-proxy[2433612]: TASK ERROR: connection error: not connected

 

[Chris]

So no error response body being logged which would help to further debug. Can you verify that you have versioning enabled on the bucket? A quick online search points towards issues with conflicting uploads if not enabled. If that does not help and the issue persists, please open an issue at https://bugzilla.proxmox.com, we will have to see if we can get hold of a testing account for further debugging.

 

[SerhiiT]

So no error response body being logged which would help to further debug. Can you verify that you have versioning enabled on the bucket? A quick online search points towards issues with conflicting uploads if not enabled.

Bingo, you are absolutely correct, versioning was not enabled for bucket (namespace in HCP terminology).

Hitachi documentation says that HTTP error 409 should be returned in case of object exists and versioning is not enabled, but it was hard to narrow root cause because this error from HCP was not logged by proxmox backup server and there are no mentions in S3 backend configuration documentation that S3 endpoint should have versioning enabled.

Backups are now created without errors, i can see several index.json.blob versions in object store.

Thank you very much for assistance with this issue.