Garbage collection fails with Google Cloud Bucket

R

rmundel

Renowned Member
May 9, 2015
35
7
73
Brazil
Hey guys, I started to using the new S3 Endpoint for sync our local backups.

The backup process went fine with one 800ish GB snapshot, but the GC gave me the following error in the task log:

Code: 
2025-10-03T00:00:00-04:00: starting garbage collection on store google-cloud
2025-10-03T00:00:00-04:00: task triggered by schedule 'daily'
2025-10-03T00:00:01-04:00: Access time update check successful, proceeding with GC.
2025-10-03T00:00:01-04:00: Using access time cutoff 1d 5m, minimum access time is 2025-10-02T03:55:00Z
2025-10-03T00:00:01-04:00: Start GC phase1 (mark used chunks)
2025-10-03T00:00:02-04:00: marked 50% (1 of 2 index files)
2025-10-03T00:00:02-04:00: marked 100% (2 of 2 index files)
2025-10-03T00:00:02-04:00: Start GC phase2 (sweep unused chunks)
2025-10-03T00:00:02-04:00: queued notification (id=1c695637-0675-443f-8476-ba2e305c242b)
2025-10-03T00:00:02-04:00: TASK ERROR: failed to list chunk in s3 object store: failed to parse response body: custom: missing field `StorageClass`

I'm using a Nearline class bucket.

Any thoughts on how to fix the GC?
 
Just chiming in here with the same issue. Using the S3 endpoint for GCP on my setup results in the same missing field task error. I have tried recreating the endpoint, and clearing and recreating my backup configuration, but it is the same. I have been running various verification and pruning jobs to try to see if something will get me past this, but so far it's consistently happening.
 
desmo.nz said:
I also run into the missing StortageClasss field when garbage collecting Google Cloud S3 Storage.

It seems it's optional if storage class is STANDARD:

x-amz-storage-class
Provides storage class information of the object. Amazon S3 returns this header for all objects except for S3 Standard storage class objects.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_ResponseSyntax
Click to expand...
This error does not stem from the header and not related to the getObject call. It is rather related to the response body parsing for the listObjectsv2 call, where the storage class is optional as well, but was not treated as such. A patch regarding this has already been send to the developers mailing list, see https://lore.proxmox.com/pbs-devel/20251020091559.106203-1-c.ebner@proxmox.com/T/
 
I just tried BackupServer 4.0.22 to garbage collect the google s3.

I'm using Path Style with endpoint storage.googleapis.com

Is garbage collection supported yet with s3?

cheers

Code: 
025-11-25T12:42:41+13:00: marked 97% (889 of 916 index files)
2025-11-25T12:42:44+13:00: marked 98% (898 of 916 index files)
2025-11-25T12:42:44+13:00: marked 99% (907 of 916 index files)
2025-11-25T12:45:05+13:00: marked 100% (916 of 916 index files)
2025-11-25T12:45:05+13:00: Start GC phase2 (sweep unused chunks)
2025-11-25T12:45:11+13:00: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NotImplemented</Code><Message>A header or query you provided requested a function that is not implemented.</Message><Details>POST ?delete is not implemented for objects.</Details></Error>
2025-11-25T12:45:11+13:00: queued notification (id=24ffb2ff-2921-4962-a636-984c59107738)
2025-11-25T12:45:11+13:00: TASK ERROR: invalid request
 
desmo.nz said:
I just tried BackupServer 4.0.22 to garbage collect the google s3.

I'm using Path Style with endpoint storage.googleapis.com

Is garbage collection supported yet with s3?
Click to expand...
Yes, garbage collection does work for s3 compatible object stores which implement the listObjectsV2 and deleteObjects API endpoints. This issue seems to be Google cloud storage specific. Please open an issue for this at bugzilla.proxmox.com and reference this post with some initial details, so we can look into this and you get status notifications on updates.

desmo.nz said:
Code: 
025-11-25T12:42:41+13:00: marked 97% (889 of 916 index files)
2025-11-25T12:42:44+13:00: marked 98% (898 of 916 index files)
2025-11-25T12:42:44+13:00: marked 99% (907 of 916 index files)
2025-11-25T12:45:05+13:00: marked 100% (916 of 916 index files)
2025-11-25T12:45:05+13:00: Start GC phase2 (sweep unused chunks)
2025-11-25T12:45:11+13:00: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NotImplemented</Code><Message>A header or query you provided requested a function that is not implemented.</Message><Details>POST ?delete is not implemented for objects.</Details></Error>
2025-11-25T12:45:11+13:00: queued notification (id=24ffb2ff-2921-4962-a636-984c59107738)
2025-11-25T12:45:11+13:00: TASK ERROR: invalid request
Click to expand...
Thanks for the logs, it seems that the Google API does not implement the deleteObjects endpoint as expected, it reports a POST ?delete is not implemented for objects. This should however be implemented as https://docs.aws.amazon.com/AmazonS...eObjects.html#API_DeleteObjects_RequestSyntax

Unfortunately the Google API docs seem to be incorrect on this one, as they seem to expect a list of object to delete on the deleteObject with a DELETE API call while expecting a DELETE with query paramerters for the deleteObjects. To quote the docs [0]:
DeleteObject (versionId optional)

This operation lets you delete multiple objects from a single bucket with one request. The request can contain a list of up to 1000 keys that you want to delete.

Request example:

DELETE /{fully_qualified_bucket_name}/{object_name}?versionId={VersionId} HTTP/1.1
Host: host
X-Amz-Date: date
X-Amz-Content-SHA256: authorization string
Content-type: text/plain
<Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Object>
<Key>object_one.jpg</Key>
<VersionId>1234568892</VersionId>
</Object>
<Object>
<Key>object_two.jpg</Key>
</Object>
</Delete>

Response example (success):

HTTP/1.1 200 OK
x-amz-id-2: 5h4FxSNCUS7wP5z92eGCWDshNpMnRuXvETa4HH3LvvH6VAIr0jU7tH9kM7X+njXx
x-amz-request-id: A437B3B641629AEE
Date: Fri, 02 Oct 2023 01:53:42 GMT
Content-Type: application/xml
Content-Length: 251
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Deleted>
<Key>object_one.jpg</Key>
</Deleted>
<Error>
<Key>object_two.jpg</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>

DeleteObjects (versionId optional)

Removes an object from a bucket. The behavior depends on the bucket's versioning state: - If bucket versioning is not enabled, the operation permanently deletes the object. - If bucket versioning is enabled, the operation inserts a delete marker, which becomes the current version of the object. To permanently delete an object in a versioned bucket, you must include the object's versionId in the request.

Request example:

DELETE /{fully_qualified_bucket_name}/{object_name}?versionId={VersionId} HTTP/1.1
Host: host
X-Amz-Date: date
X-Amz-Content-SHA256: authorization string
Content-type: text/plain

Response example (success):

HTTP/1.1 204 NoContent
x-amz-id-2: iVmcB7OXXJRkRH1FiVq1151/T24gRfpwpuZrEG11Bb9ImOMAAe98oxSpXlknabA0LTvBYJpSIX
x-amz-request-id: B34E94CACB2CEF6C
Date: Mon, 17 Oct 2022 22:47:09 GMT
Click to expand...

[0] https://cloud.google.com/distribute...vice/storage/storage-s3-rest-api#DeleteObject
 
You must log in or register to reply here.
Top Bottom